Nov 02 17:32:35.972375 ip-10-0-142-133 systemd[1]: Starting Kubernetes Kubelet... Nov 02 17:32:36.382939 ip-10-0-142-133 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:32:36.382939 ip-10-0-142-133 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 17:32:36.382939 ip-10-0-142-133 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:32:36.382939 ip-10-0-142-133 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:32:36.387212 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387130 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:32:36.387212 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387211 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:32:36.387212 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387215 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387218 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387221 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387224 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387226 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387229 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387232 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387234 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387237 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387239 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387242 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387245 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387248 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387251 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387253 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387256 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387260 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387263 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387265 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387268 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:32:36.387306 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387271 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387274 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387277 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387279 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387282 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387285 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387294 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387297 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387299 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387302 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387305 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387308 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387311 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387314 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387317 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387320 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387323 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387326 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387329 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387331 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:32:36.387771 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387334 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387336 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387339 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387342 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387345 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387347 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387350 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387352 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387355 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387357 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387360 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387362 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387364 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387368 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387371 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387374 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387376 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387379 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387382 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387386 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:32:36.388259 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387388 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:32:36.388814 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387391 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:32:36.388814 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387393 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:32:36.388814 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387396 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:32:36.388814 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387398 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:32:36.388814 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387404 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:32:36.388814 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387409 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:32:36.388814 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387412 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:32:36.388814 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387415 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:32:36.388814 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387418 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:32:36.388814 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387421 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:32:36.388814 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387423 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:32:36.388814 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387426 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:32:36.388814 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387429 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:32:36.388814 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387431 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:32:36.388814 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387434 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:32:36.388814 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387437 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:32:36.388814 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387439 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:32:36.388814 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387442 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387446 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387450 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387453 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387456 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387459 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387462 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387466 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387468 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387471 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387475 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387477 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387480 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387483 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387486 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387488 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387491 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387494 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387496 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:32:36.389271 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387498 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387501 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387504 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387506 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387510 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387513 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.387515 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388890 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388899 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388902 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388906 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388908 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388911 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388914 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388917 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388919 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388922 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388925 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388927 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:32:36.389718 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388930 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388933 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388936 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388939 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388942 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388945 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388947 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388950 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388953 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388957 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388960 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388962 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388965 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388967 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388970 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388972 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388975 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388978 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388980 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:32:36.390193 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388983 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388986 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388988 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388990 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388993 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388995 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.388998 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389000 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389003 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389005 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389008 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389011 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389027 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389030 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389034 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389036 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389041 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389045 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389049 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:32:36.390642 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389052 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389054 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389057 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389060 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389063 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389066 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389069 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389072 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389075 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389078 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389080 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389083 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389085 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389088 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389091 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389096 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389099 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389103 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389106 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:32:36.391122 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389109 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389111 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389114 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389117 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389119 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389122 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389124 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389128 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389132 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389134 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389137 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389139 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389142 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389145 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389148 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389150 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389153 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389155 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389158 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:32:36.391568 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389162 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389164 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389167 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389169 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389172 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389174 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389177 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389179 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389182 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389185 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389188 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389191 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389194 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389196 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389199 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389202 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389205 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389208 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389714 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389720 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:32:36.392007 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389723 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389726 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389729 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389732 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389735 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389737 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389740 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389743 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389745 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389748 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389750 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389754 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389756 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389759 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389762 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389764 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389767 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389770 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389772 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389775 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:32:36.392514 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389777 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389780 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389782 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389785 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389788 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389791 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389793 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389796 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389799 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389801 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389804 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389806 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389808 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389812 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389814 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389817 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389819 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389822 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389824 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389827 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:32:36.392993 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389829 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389832 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389834 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389837 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389842 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389846 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389849 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389852 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389855 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389858 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389860 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389863 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389865 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389868 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389871 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389873 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389876 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389878 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389881 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:32:36.393481 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389883 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389886 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389888 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389892 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389894 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389896 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389899 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389902 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389904 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389907 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389909 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389912 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389914 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389917 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389920 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389922 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389925 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389928 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389931 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389933 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:32:36.393977 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389938 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389941 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389944 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389946 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389949 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389951 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389954 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389956 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389959 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389961 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389964 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389967 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389971 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389975 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389979 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389982 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389985 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389989 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389992 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:32:36.394479 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389996 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.389999 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.390001 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.390004 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.390007 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.390009 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390522 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390532 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390540 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390544 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390549 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390553 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390557 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390564 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390567 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390570 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390574 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390577 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390580 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390583 2578 flags.go:64] FLAG: --cgroup-root="" Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390586 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390589 2578 flags.go:64] FLAG: --client-ca-file="" Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390592 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 02 17:32:36.394981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390595 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390601 2578 flags.go:64] FLAG: --cluster-domain="" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390604 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390607 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390610 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390613 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390617 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390621 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390624 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390627 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390630 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390633 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390636 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390639 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390642 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390647 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390651 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390654 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390657 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390660 2578 flags.go:64] FLAG: --enable-server="true" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390663 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390668 2578 flags.go:64] FLAG: --event-burst="100" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390672 2578 flags.go:64] FLAG: --event-qps="50" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390676 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 17:32:36.395532 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390679 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390682 2578 flags.go:64] FLAG: --eviction-hard="" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390686 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390689 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390692 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390695 2578 flags.go:64] FLAG: --eviction-soft="" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390698 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390701 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390705 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390708 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390711 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390714 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390717 2578 flags.go:64] FLAG: --feature-gates="" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390721 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390724 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390727 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390731 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390734 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390736 2578 flags.go:64] FLAG: --help="false" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390740 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390743 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390746 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390748 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 17:32:36.396132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390752 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390755 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390758 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390761 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390764 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390766 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390769 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390772 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390776 2578 flags.go:64] FLAG: --kube-reserved="" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390779 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390782 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390785 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390788 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390790 2578 flags.go:64] FLAG: --lock-file="" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390793 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390796 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390800 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390805 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390808 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390812 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390815 2578 flags.go:64] FLAG: --logging-format="text" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390817 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390821 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390824 2578 flags.go:64] FLAG: --manifest-url="" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390827 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 02 17:32:36.396691 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390831 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390834 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390838 2578 flags.go:64] FLAG: --max-pods="110" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390841 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390844 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390864 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390868 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390871 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390874 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390878 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390885 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390888 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390892 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390895 2578 flags.go:64] FLAG: --pod-cidr="" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390898 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390900 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390904 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390907 2578 flags.go:64] FLAG: --port="10250" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390910 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390913 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0589b06646a566b08" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390924 2578 flags.go:64] FLAG: --qos-reserved="" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390928 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390931 2578 flags.go:64] FLAG: --register-node="true" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390934 2578 flags.go:64] FLAG: --register-with-taints="" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390943 2578 flags.go:64] FLAG: --registry-burst="10" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390946 2578 flags.go:64] FLAG: --registry-qps="5" Nov 02 17:32:36.397385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390949 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390952 2578 flags.go:64] FLAG: --reserved-memory="" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390956 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390959 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390962 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390965 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390968 2578 flags.go:64] FLAG: --runonce="false" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390971 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390974 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390977 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390980 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390982 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390985 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390989 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390992 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390995 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.390997 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391000 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391003 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391007 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391009 2578 flags.go:64] FLAG: --system-cgroups="" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391012 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391030 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391034 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391037 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 17:32:36.398468 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391042 2578 flags.go:64] FLAG: --tls-min-version="" Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391045 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391047 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391050 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391054 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391057 2578 flags.go:64] FLAG: --v="2" Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391061 2578 flags.go:64] FLAG: --version="false" Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391065 2578 flags.go:64] FLAG: --vmodule="" Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391069 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391073 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391211 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391215 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391218 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391220 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391223 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391226 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391228 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391231 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391233 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391236 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391238 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:32:36.399289 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391241 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391244 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391247 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391249 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391252 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391257 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391260 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391263 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391266 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391268 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391272 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391275 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391278 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391280 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391283 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391290 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391293 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391296 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391298 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:32:36.399793 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391301 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391303 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391306 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391309 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391312 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391315 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391317 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391320 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391323 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391326 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391330 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391333 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391336 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391339 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391342 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391344 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391347 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391350 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391354 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391356 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:32:36.400268 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391359 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391362 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391365 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391367 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391372 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391376 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391379 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391381 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391386 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391388 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391391 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391394 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391397 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391399 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391402 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391404 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391407 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391409 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391412 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:32:36.400738 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391414 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391417 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391420 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391422 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391425 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391427 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391430 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391432 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391435 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391438 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391440 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391444 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391447 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391450 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391453 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391455 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391458 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391461 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391464 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391467 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:32:36.401208 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391469 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:32:36.401682 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391473 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:32:36.401682 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391476 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:32:36.401682 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391478 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:32:36.401682 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391481 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:32:36.401682 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391484 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:32:36.401682 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391486 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:32:36.401682 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391489 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:32:36.401682 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391492 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:32:36.401682 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391495 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:32:36.401682 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391497 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:32:36.401682 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391500 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:32:36.401682 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391502 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:32:36.401682 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391505 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:32:36.401682 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391508 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:32:36.401682 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391511 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:32:36.401682 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391513 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:32:36.402078 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.391540 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:32:36.402078 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391575 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:32:36.402078 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391580 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:32:36.402078 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391583 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:32:36.402078 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391585 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:32:36.402078 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391588 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:32:36.402078 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391590 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:32:36.402078 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391594 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:32:36.402078 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391597 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:32:36.402078 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391600 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:32:36.402078 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391602 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:32:36.402078 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391605 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:32:36.402078 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391608 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:32:36.402078 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391611 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:32:36.402078 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391614 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:32:36.402078 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391617 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391620 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391623 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391626 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391628 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391631 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391634 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391636 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391639 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391642 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391644 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391647 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391650 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391652 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391655 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391657 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391660 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391663 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391665 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:32:36.402466 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391668 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391670 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391673 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391676 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391678 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391682 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391685 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391687 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391690 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391693 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391696 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391699 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391701 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391705 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391709 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391711 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391714 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391716 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391719 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:32:36.402959 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391722 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391724 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391727 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391730 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391732 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391735 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391738 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391740 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391746 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391749 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391753 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391757 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391760 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391763 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391765 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391768 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391770 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391773 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391778 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391780 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:32:36.403443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391783 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391786 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391789 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391791 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391795 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391799 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391803 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391807 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391810 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391812 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391815 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391817 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391820 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391823 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391825 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391828 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391830 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391833 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391835 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:32:36.403929 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391838 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391840 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391843 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391846 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391848 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391851 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391854 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391856 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391859 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391862 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391864 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391868 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391871 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391874 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.391877 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.399847 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.399866 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.399881 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.399887 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.399956 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:32:36.404398 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.399961 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.399965 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.399968 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.399971 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.399973 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.399976 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.399979 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.399981 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.399984 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.399987 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.399989 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.399992 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.399995 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.399998 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400001 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400004 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400007 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400010 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400012 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400030 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:32:36.404896 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400035 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400039 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400043 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400047 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400052 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400055 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400058 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400061 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400064 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400066 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400069 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400072 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400075 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400078 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400081 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400083 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400086 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400088 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400092 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:32:36.405474 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400094 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400097 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400099 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400102 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400105 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400109 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400115 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400118 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400122 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400127 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400130 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400133 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400136 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400139 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400142 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400145 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400147 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400150 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400153 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:32:36.405951 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400156 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400159 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400162 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400165 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400168 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400171 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400174 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400176 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400179 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400182 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400184 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400187 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400190 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400192 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400195 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400198 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400200 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400203 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400205 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400208 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:32:36.406403 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400211 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400214 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400217 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400219 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400222 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400225 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400228 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400231 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400233 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400236 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400238 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400241 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400244 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400246 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400249 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400252 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400254 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400257 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400259 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400262 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:32:36.406882 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400264 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:32:36.407430 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400267 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:32:36.407430 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400270 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:32:36.407430 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400272 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:32:36.407430 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400275 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:32:36.407430 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400278 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:32:36.407430 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400281 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:32:36.407430 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.400309 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:32:36.407430 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400465 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:32:36.407430 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400470 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:32:36.407430 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400473 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:32:36.407430 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400476 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:32:36.407430 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400480 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:32:36.407430 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400485 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:32:36.407430 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400488 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:32:36.407430 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400491 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:32:36.407430 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400495 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400498 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400501 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400503 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400506 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400509 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400511 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400514 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400517 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400519 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400522 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400525 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400527 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400530 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400533 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400535 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400538 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400541 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400543 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400546 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:32:36.407805 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400548 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400551 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400554 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400556 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400559 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400561 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400564 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400567 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400569 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400572 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400575 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400578 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400581 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400583 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400586 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400588 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400591 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400593 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400596 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:32:36.408340 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400598 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400601 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400603 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400606 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400609 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400612 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400614 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400617 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400620 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400622 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400625 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400627 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400630 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400633 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400635 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400638 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400640 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400643 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400647 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:32:36.408828 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400651 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400654 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400656 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400659 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400661 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400665 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400667 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400670 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400672 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400675 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400678 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400680 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400683 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400685 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400688 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400691 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400694 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400696 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400699 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:32:36.409341 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400701 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400704 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400706 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400709 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400712 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400715 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400717 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400720 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400723 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400725 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400727 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400730 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400732 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400735 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400737 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400740 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400743 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400745 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400748 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400752 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:32:36.409780 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400754 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:32:36.410315 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.400776 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:32:36.410315 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400830 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:32:36.410315 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400836 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:32:36.410315 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400839 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:32:36.410315 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400843 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:32:36.410315 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400847 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:32:36.410315 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400851 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:32:36.410315 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400854 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:32:36.410315 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400857 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:32:36.410315 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400860 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:32:36.410315 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400862 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:32:36.410315 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400865 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:32:36.410315 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400867 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:32:36.410315 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400870 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:32:36.410315 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400873 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400875 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400878 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400880 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400883 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400886 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400888 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400891 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400893 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400896 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400899 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400901 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400904 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400906 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400909 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400911 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400914 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400917 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400919 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400922 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:32:36.410676 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400925 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400927 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400930 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400932 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400935 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400937 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400940 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400942 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400945 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400947 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400950 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400952 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400955 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400957 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400960 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400964 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400967 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400970 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400973 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400975 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:32:36.411175 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400978 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400980 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400983 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400985 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400988 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400991 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400994 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400996 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.400999 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401001 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401004 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401007 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401010 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401012 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401026 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401029 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401032 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401034 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401037 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401040 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:32:36.411665 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401042 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401045 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401048 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401050 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401053 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401055 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401058 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401061 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401064 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401066 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401069 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401071 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401074 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401077 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401079 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401082 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401084 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401087 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401090 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:32:36.412150 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401093 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:32:36.412588 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401096 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:32:36.412588 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401098 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:32:36.412588 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401101 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:32:36.412588 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401104 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:32:36.412588 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401107 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:32:36.412588 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401109 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:32:36.412588 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401112 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:32:36.412588 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401114 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:32:36.412588 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401117 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:32:36.412588 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401119 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:32:36.412588 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401122 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:32:36.412588 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401124 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:32:36.412588 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:36.401127 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:32:36.412588 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.401188 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 17:32:36.412588 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.402704 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 17:32:36.412588 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.403687 2578 server.go:1014] "Starting client certificate rotation" Nov 02 17:32:36.412588 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.403785 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:32:36.412588 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.403822 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:32:36.413254 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.405806 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:32:36.413254 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.407417 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:32:36.421670 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.421652 2578 log.go:25] "Validated CRI v1 runtime API" Nov 02 17:32:36.424981 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.424960 2578 log.go:25] "Validated CRI v1 image API" Nov 02 17:32:36.425788 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.425766 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 17:32:36.431106 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.431076 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c861d13b-7785-48cd-be99-049f18fba5b5:/dev/nvme0n1p4 cce73ded-b912-458e-b0ac-911a50df920c:/dev/nvme0n1p3] Nov 02 17:32:36.431179 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.431104 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 17:32:36.433517 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.433500 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 17:32:36.436978 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.436856 2578 manager.go:217] Machine: {Timestamp:2026-11-02 17:32:36.435204237 +0000 UTC m=+0.346372056 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098871 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec268d136db8825b1663df7af2e7ecac SystemUUID:ec268d13-6db8-825b-1663-df7af2e7ecac BootID:21454693-700e-486d-ac79-41437dc05655 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cf:d2:70:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cf:d2:70:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:f5:46:68:9f:79 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 17:32:36.437473 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.437462 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 17:32:36.437578 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.437566 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 17:32:36.439151 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.439127 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 17:32:36.439283 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.439154 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-133.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 17:32:36.439330 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.439285 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 17:32:36.439330 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.439292 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 17:32:36.439330 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.439314 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:32:36.439330 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.439327 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:32:36.440096 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.440086 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 17:32:36.444184 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.444168 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 17:32:36.444322 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.444311 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 17:32:36.444387 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.444377 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 02 17:32:36.444433 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.444397 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 17:32:36.444433 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.444412 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 17:32:36.444433 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.444427 2578 kubelet.go:405] "Adding apiserver pod source" Nov 02 17:32:36.444545 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.444440 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 17:32:36.444796 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.444782 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:32:36.447154 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.447137 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 17:32:36.451829 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.451809 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l2v5w" Nov 02 17:32:36.453405 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.453393 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:32:36.455238 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.455223 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 17:32:36.455305 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.455254 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 17:32:36.455409 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.455399 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 17:32:36.455445 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.455415 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 17:32:36.455445 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.455422 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 17:32:36.455445 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.455428 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 17:32:36.455445 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.455434 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 17:32:36.455445 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.455440 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 17:32:36.457224 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.457209 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 17:32:36.457276 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.457225 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 17:32:36.457276 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.457246 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 17:32:36.457276 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.457256 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 17:32:36.457389 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.457277 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 17:32:36.457389 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.457292 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 17:32:36.458202 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.458190 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 17:32:36.458202 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.458200 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 17:32:36.459175 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.459159 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l2v5w" Nov 02 17:32:36.462384 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.462362 2578 server.go:1263] "Started kubelet" Nov 02 17:32:36.462559 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.462534 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 17:32:36.462559 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.462518 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 17:32:36.462686 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.462587 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 17:32:36.463150 ip-10-0-142-133 systemd[1]: Started Kubernetes Kubelet. Nov 02 17:32:36.464348 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.464328 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 02 17:32:36.465063 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.465046 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 17:32:36.469191 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:36.469169 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 17:32:36.471048 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.471032 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:32:36.471668 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.471654 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 17:32:36.471723 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.471677 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 17:32:36.472645 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:36.472622 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-133.us-east-2.compute.internal\" not found" Nov 02 17:32:36.473142 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.473123 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 17:32:36.473142 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.473143 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 17:32:36.473261 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.473240 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 17:32:36.473355 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.473341 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 02 17:32:36.473355 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.473352 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 02 17:32:36.473613 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.473596 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:32:36.474546 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.474505 2578 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-142-133.us-east-2.compute.internal" not found Nov 02 17:32:36.474953 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.474934 2578 factory.go:55] Registering systemd factory Nov 02 17:32:36.475032 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.474957 2578 factory.go:223] Registration of the systemd container factory successfully Nov 02 17:32:36.475197 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.475185 2578 factory.go:153] Registering CRI-O factory Nov 02 17:32:36.475250 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.475199 2578 factory.go:223] Registration of the crio container factory successfully Nov 02 17:32:36.475250 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.475241 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 17:32:36.475330 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.475255 2578 factory.go:103] Registering Raw factory Nov 02 17:32:36.475330 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.475272 2578 manager.go:1199] Started watching for new ooms in manager Nov 02 17:32:36.476246 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.476227 2578 manager.go:319] Starting recovery of all containers Nov 02 17:32:36.477682 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.477664 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:32:36.479196 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:36.478217 2578 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-133.us-east-2.compute.internal\" not found" node="ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.485976 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.485958 2578 manager.go:324] Recovery completed Nov 02 17:32:36.487582 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.487565 2578 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-142-133.us-east-2.compute.internal" not found Nov 02 17:32:36.489979 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.489967 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:32:36.493107 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.493091 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:32:36.493172 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.493119 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:32:36.493172 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.493129 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:32:36.494090 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.494079 2578 cpu_manager.go:226] "Starting" policy="none" Nov 02 17:32:36.494090 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.494089 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 17:32:36.494177 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.494081 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:32:36.494177 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.494108 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 17:32:36.494872 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.494857 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:32:36.494923 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.494893 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:32:36.494923 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.494908 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:32:36.496342 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.496331 2578 policy_none.go:50] "Start" Nov 02 17:32:36.496385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.496347 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 17:32:36.496385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.496357 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 17:32:36.498379 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.498363 2578 policy_none.go:44] "Start" Nov 02 17:32:36.555305 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.537718 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 17:32:36.555305 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.540861 2578 manager.go:341] "Starting Device Plugin manager" Nov 02 17:32:36.555305 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:36.541757 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 17:32:36.555305 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.541776 2578 server.go:87] "Starting device plugin registration server" Nov 02 17:32:36.555305 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.542029 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 17:32:36.555305 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.542045 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 17:32:36.555305 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.542123 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 17:32:36.555305 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.542194 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 17:32:36.555305 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.542202 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 17:32:36.555305 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:36.542722 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 17:32:36.555305 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:36.542773 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-133.us-east-2.compute.internal\" not found" Nov 02 17:32:36.555305 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.543722 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 17:32:36.555305 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.544299 2578 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-142-133.us-east-2.compute.internal" not found Nov 02 17:32:36.555305 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.545123 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 17:32:36.555305 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.545143 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 17:32:36.555305 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.545161 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 17:32:36.555305 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:36.545191 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 17:32:36.555305 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.548962 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:32:36.642828 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.642740 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:32:36.644648 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.644632 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:32:36.644746 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.644664 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:32:36.644746 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.644679 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:32:36.644746 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.644708 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.645636 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.645617 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal"] Nov 02 17:32:36.645688 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.645670 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:32:36.646447 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.646433 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:32:36.646519 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.646457 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:32:36.646519 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.646467 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:32:36.648523 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.648511 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:32:36.648675 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.648659 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.648740 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.648697 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:32:36.649237 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.649224 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:32:36.649308 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.649238 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:32:36.649308 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.649264 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:32:36.649308 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.649273 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:32:36.649308 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.649244 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:32:36.649438 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.649322 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:32:36.651319 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.651302 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.651382 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.651330 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:32:36.651968 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.651951 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:32:36.652033 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.651973 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:32:36.652033 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.651985 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:32:36.654565 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.654416 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.669517 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.669499 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.672908 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.672895 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.675620 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.675608 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.680320 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.680308 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:32:36.686100 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.686087 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:32:36.689398 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.689385 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:32:36.689440 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:36.689416 2578 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.689440 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.689426 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.702398 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.702381 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:32:36.702459 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:36.702412 2578 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.774447 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.774423 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebe0e6ace491eef2c5b9302a33c08d4d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal\" (UID: \"ebe0e6ace491eef2c5b9302a33c08d4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.774567 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.774451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7fad33c464d219dfb7d5fa21dbe7ad22-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal\" (UID: \"7fad33c464d219dfb7d5fa21dbe7ad22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.774567 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.774469 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fad33c464d219dfb7d5fa21dbe7ad22-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal\" (UID: \"7fad33c464d219dfb7d5fa21dbe7ad22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.874979 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.874945 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebe0e6ace491eef2c5b9302a33c08d4d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal\" (UID: \"ebe0e6ace491eef2c5b9302a33c08d4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.874979 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.874978 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7fad33c464d219dfb7d5fa21dbe7ad22-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal\" (UID: \"7fad33c464d219dfb7d5fa21dbe7ad22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.875197 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.875002 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fad33c464d219dfb7d5fa21dbe7ad22-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal\" (UID: \"7fad33c464d219dfb7d5fa21dbe7ad22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.875197 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.875078 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fad33c464d219dfb7d5fa21dbe7ad22-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal\" (UID: \"7fad33c464d219dfb7d5fa21dbe7ad22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.875197 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.875080 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7fad33c464d219dfb7d5fa21dbe7ad22-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal\" (UID: \"7fad33c464d219dfb7d5fa21dbe7ad22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.875197 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.875080 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebe0e6ace491eef2c5b9302a33c08d4d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal\" (UID: \"ebe0e6ace491eef2c5b9302a33c08d4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.985649 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.985576 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:36.990410 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:36.990082 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:32:38.747897 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.747869 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 17:32:38.748441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.748023 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 17:32:38.748441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.748022 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 17:32:38.748441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.748030 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 17:32:38.789302 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.789270 2578 apiserver.go:52] "Watching apiserver" Nov 02 17:32:38.796882 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.796857 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 17:32:38.797166 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.797150 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-nx2vb","openshift-ovn-kubernetes/ovnkube-node-956tn","kube-system/global-pull-secret-syncer-drvjl","openshift-cluster-node-tuning-operator/tuned-m78lg","openshift-dns/node-resolver-278vv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal","openshift-multus/multus-jw64s","openshift-multus/network-metrics-daemon-zcdw6","kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal","kube-system/konnectivity-agent-8s6rw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8","openshift-image-registry/node-ca-dxmdj","openshift-multus/multus-additional-cni-plugins-z4ptm","openshift-network-diagnostics/network-check-target-lnbcn"] Nov 02 17:32:38.800154 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.800136 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nx2vb" Nov 02 17:32:38.802238 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.802218 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.804467 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.804443 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:38.804586 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.804513 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:32:38.804586 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:38.804514 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:32:38.804586 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.804509 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 17:32:38.804586 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.804578 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k78hz\"" Nov 02 17:32:38.804769 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.804625 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 17:32:38.805491 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.805475 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 17:32:38.805783 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.805766 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 17:32:38.805874 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.805785 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 17:32:38.805874 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.805797 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 17:32:38.805874 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.805800 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 17:32:38.805874 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.805768 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sbqnj\"" Nov 02 17:32:38.806039 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.805908 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 17:32:38.806542 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.806530 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.808935 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.808900 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:27:37 +0000 UTC" deadline="2028-05-30 03:25:35.944772372 +0000 UTC" Nov 02 17:32:38.808935 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.808934 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13785h52m57.135840173s" Nov 02 17:32:38.809312 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.809298 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-278vv" Nov 02 17:32:38.811623 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.811605 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.814068 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.814047 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:38.814172 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:38.814146 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:32:38.815847 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.815828 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 17:32:38.817600 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.817579 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8s6rw" Nov 02 17:32:38.820836 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.820815 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-var-lib-openvswitch\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.820945 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.820851 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-node-log\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.820945 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.820877 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9w5w\" (UniqueName: \"kubernetes.io/projected/2b6949d0-0006-4488-89ad-296077fcf045-kube-api-access-h9w5w\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.820945 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.820900 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-sysconfig\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.821102 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-device-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.821102 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821043 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/491761ca-d114-411c-9196-3bd0065ae73a-host-slash\") pod \"iptables-alerter-nx2vb\" (UID: \"491761ca-d114-411c-9196-3bd0065ae73a\") " pod="openshift-network-operator/iptables-alerter-nx2vb" Nov 02 17:32:38.821207 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821111 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-kubelet\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.821207 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b6949d0-0006-4488-89ad-296077fcf045-ovnkube-config\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.821301 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821203 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-sysctl-conf\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.821301 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821236 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-systemd\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.821301 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3d709e2-f49d-4825-8beb-056b4582f358-tmp\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.821301 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821292 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62013dcf-af15-4041-bdc2-f5437c000082-kubelet-config\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:38.821461 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4a3f746-8b39-4e1e-8f18-8694b16929e1-konnectivity-ca\") pod \"konnectivity-agent-8s6rw\" (UID: \"e4a3f746-8b39-4e1e-8f18-8694b16929e1\") " pod="kube-system/konnectivity-agent-8s6rw" Nov 02 17:32:38.821461 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821360 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-cni-dir\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.821461 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821403 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-run-k8s-cni-cncf-io\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.821461 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821449 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-registration-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.821637 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821481 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-cnibin\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.821637 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821500 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-var-lib-cni-multus\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.821637 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821564 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-run-openvswitch\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.821637 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821598 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-host\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.821779 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821673 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-modprobe-d\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.821779 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821723 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-run\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.821779 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821769 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7t6q\" (UniqueName: \"kubernetes.io/projected/491761ca-d114-411c-9196-3bd0065ae73a-kube-api-access-k7t6q\") pod \"iptables-alerter-nx2vb\" (UID: \"491761ca-d114-411c-9196-3bd0065ae73a\") " pod="openshift-network-operator/iptables-alerter-nx2vb" Nov 02 17:32:38.821891 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-slash\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.821891 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-run-netns\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.821891 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821868 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-system-cni-dir\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.822373 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821892 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-var-lib-cni-bin\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.822373 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.821941 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-daemon-config\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.822373 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822002 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:38.822373 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822046 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-sys-fs\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.822373 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-etc-openvswitch\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.822373 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822097 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-run-ovn\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.822373 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822110 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b6949d0-0006-4488-89ad-296077fcf045-env-overrides\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.822373 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822144 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b6949d0-0006-4488-89ad-296077fcf045-ovn-node-metrics-cert\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.822373 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822195 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-sys\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.822373 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-cni-bin\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.822373 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822341 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-run-netns\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.822373 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822376 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-run-multus-certs\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.822838 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822440 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfmgm\" (UniqueName: \"kubernetes.io/projected/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-kube-api-access-cfmgm\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.822838 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822461 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b6949d0-0006-4488-89ad-296077fcf045-ovnkube-script-lib\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.822838 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs\") pod \"network-metrics-daemon-zcdw6\" (UID: \"a73fde2c-c65f-4374-adef-ee60a09c0b4a\") " pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:38.822838 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822522 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.822838 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822551 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-kubernetes\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.822838 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822596 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-sysctl-d\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.822838 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9g7j\" (UniqueName: \"kubernetes.io/projected/e354a328-13db-422e-b0e7-0cc0a5af5a90-kube-api-access-c9g7j\") pod \"node-resolver-278vv\" (UID: \"e354a328-13db-422e-b0e7-0cc0a5af5a90\") " pod="openshift-dns/node-resolver-278vv" Nov 02 17:32:38.822838 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822680 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-etc-kubernetes\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.822838 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822709 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62013dcf-af15-4041-bdc2-f5437c000082-dbus\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:38.822838 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822744 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-cni-netd\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.822838 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822773 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e354a328-13db-422e-b0e7-0cc0a5af5a90-tmp-dir\") pod \"node-resolver-278vv\" (UID: \"e354a328-13db-422e-b0e7-0cc0a5af5a90\") " pod="openshift-dns/node-resolver-278vv" Nov 02 17:32:38.822838 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822795 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-cni-binary-copy\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.822838 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4a3f746-8b39-4e1e-8f18-8694b16929e1-agent-certs\") pod \"konnectivity-agent-8s6rw\" (UID: \"e4a3f746-8b39-4e1e-8f18-8694b16929e1\") " pod="kube-system/konnectivity-agent-8s6rw" Nov 02 17:32:38.823274 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822860 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-log-socket\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.823274 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822899 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3d709e2-f49d-4825-8beb-056b4582f358-etc-tuned\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.823274 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822931 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e354a328-13db-422e-b0e7-0cc0a5af5a90-hosts-file\") pod \"node-resolver-278vv\" (UID: \"e354a328-13db-422e-b0e7-0cc0a5af5a90\") " pod="openshift-dns/node-resolver-278vv" Nov 02 17:32:38.823274 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.822973 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-conf-dir\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.823274 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.823007 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.823274 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.823024 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-lib-modules\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.823274 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.823038 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-var-lib-kubelet\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.823274 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.823064 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-run-ovn-kubernetes\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.823274 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.823085 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjhkj\" (UniqueName: \"kubernetes.io/projected/d3d709e2-f49d-4825-8beb-056b4582f358-kube-api-access-xjhkj\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.823274 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.823099 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8kcn\" (UniqueName: \"kubernetes.io/projected/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-kube-api-access-h8kcn\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.823274 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.823134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnhzb\" (UniqueName: \"kubernetes.io/projected/a73fde2c-c65f-4374-adef-ee60a09c0b4a-kube-api-access-nnhzb\") pod \"network-metrics-daemon-zcdw6\" (UID: \"a73fde2c-c65f-4374-adef-ee60a09c0b4a\") " pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:38.823274 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.823164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-run-systemd\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.823274 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.823195 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.823274 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.823268 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-socket-dir-parent\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.823730 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.823298 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-var-lib-kubelet\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.823730 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.823321 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-hostroot\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.823730 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.823332 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.823730 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.823351 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-socket-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.823730 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.823376 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/491761ca-d114-411c-9196-3bd0065ae73a-iptables-alerter-script\") pod \"iptables-alerter-nx2vb\" (UID: \"491761ca-d114-411c-9196-3bd0065ae73a\") " pod="openshift-network-operator/iptables-alerter-nx2vb" Nov 02 17:32:38.823730 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.823391 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-systemd-units\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.826149 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.826132 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dxmdj" Nov 02 17:32:38.827825 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.827807 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 17:32:38.828717 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.828701 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:38.831506 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.831490 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:38.831573 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:38.831539 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:32:38.848071 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.848053 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zvt25" Nov 02 17:32:38.856885 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.856867 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zvt25" Nov 02 17:32:38.905044 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.905021 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zg2g5\"" Nov 02 17:32:38.905156 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.905054 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 17:32:38.905156 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.905084 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ttfxd\"" Nov 02 17:32:38.905156 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.905107 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 17:32:38.905495 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.905479 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 17:32:38.905553 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.905511 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 17:32:38.905553 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.905531 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jl4vs\"" Nov 02 17:32:38.905628 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.905566 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:32:38.905676 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.905638 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 17:32:38.905676 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.905666 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wjf7q\"" Nov 02 17:32:38.905739 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.905699 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 17:32:38.906155 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.906141 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 17:32:38.906204 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.906148 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 17:32:38.906204 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.906169 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 17:32:38.906204 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.906173 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 17:32:38.906335 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.906318 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 17:32:38.906385 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.906348 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lkfrw\"" Nov 02 17:32:38.907533 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.906712 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 17:32:38.907533 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.906772 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 17:32:38.907533 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.906780 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 17:32:38.907533 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.906819 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 17:32:38.907533 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.907005 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 17:32:38.907533 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.907008 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xjkc9\"" Nov 02 17:32:38.907533 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.907024 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 17:32:38.907533 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.907101 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dtld9\"" Nov 02 17:32:38.918526 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.918506 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 17:32:38.924122 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.923671 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9w5w\" (UniqueName: \"kubernetes.io/projected/2b6949d0-0006-4488-89ad-296077fcf045-kube-api-access-h9w5w\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.924122 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.923712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-sysconfig\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.924122 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.923747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:38.924122 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.923777 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-device-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.924122 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.923826 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/491761ca-d114-411c-9196-3bd0065ae73a-host-slash\") pod \"iptables-alerter-nx2vb\" (UID: \"491761ca-d114-411c-9196-3bd0065ae73a\") " pod="openshift-network-operator/iptables-alerter-nx2vb" Nov 02 17:32:38.924122 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.923838 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-sysconfig\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.924122 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.923872 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-kubelet\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.924122 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.923888 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/491761ca-d114-411c-9196-3bd0065ae73a-host-slash\") pod \"iptables-alerter-nx2vb\" (UID: \"491761ca-d114-411c-9196-3bd0065ae73a\") " pod="openshift-network-operator/iptables-alerter-nx2vb" Nov 02 17:32:38.924122 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.923923 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b6949d0-0006-4488-89ad-296077fcf045-ovnkube-config\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.924122 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.923932 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-kubelet\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.924122 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.923962 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-sysctl-conf\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.924122 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.923996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-systemd\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.924122 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924020 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3d709e2-f49d-4825-8beb-056b4582f358-tmp\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.924122 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924041 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-device-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.924938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924157 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-sysctl-conf\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.924938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924196 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62013dcf-af15-4041-bdc2-f5437c000082-kubelet-config\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:38.924938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924286 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4a3f746-8b39-4e1e-8f18-8694b16929e1-konnectivity-ca\") pod \"konnectivity-agent-8s6rw\" (UID: \"e4a3f746-8b39-4e1e-8f18-8694b16929e1\") " pod="kube-system/konnectivity-agent-8s6rw" Nov 02 17:32:38.924938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924301 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 17:32:38.924938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-cni-dir\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.924938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924352 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-run-k8s-cni-cncf-io\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.924938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-registration-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.924938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924405 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-cnibin\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.924938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924469 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b6949d0-0006-4488-89ad-296077fcf045-ovnkube-config\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.924938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-var-lib-cni-multus\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.924938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-run-openvswitch\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.924938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924543 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-host\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.924938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924567 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-registration-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.924938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-system-cni-dir\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:38.924938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924578 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-cnibin\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.924938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924605 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62013dcf-af15-4041-bdc2-f5437c000082-kubelet-config\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:38.924938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924635 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-run-openvswitch\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.924938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924712 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-systemd\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.925814 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924707 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-run-k8s-cni-cncf-io\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.925814 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924758 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-var-lib-cni-multus\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.925814 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924756 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-cni-dir\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.925814 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924791 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:38.925814 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924825 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-host\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.925814 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924872 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:38.925814 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-modprobe-d\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.925814 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924966 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-run\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.925814 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.924997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:38.925814 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925029 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f-host\") pod \"node-ca-dxmdj\" (UID: \"ffdc716d-d67b-41da-aa12-c3ede5dd6b1f\") " pod="openshift-image-registry/node-ca-dxmdj" Nov 02 17:32:38.925814 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925051 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smnk7\" (UniqueName: \"kubernetes.io/projected/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f-kube-api-access-smnk7\") pod \"node-ca-dxmdj\" (UID: \"ffdc716d-d67b-41da-aa12-c3ede5dd6b1f\") " pod="openshift-image-registry/node-ca-dxmdj" Nov 02 17:32:38.925814 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925070 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7t6q\" (UniqueName: \"kubernetes.io/projected/491761ca-d114-411c-9196-3bd0065ae73a-kube-api-access-k7t6q\") pod \"iptables-alerter-nx2vb\" (UID: \"491761ca-d114-411c-9196-3bd0065ae73a\") " pod="openshift-network-operator/iptables-alerter-nx2vb" Nov 02 17:32:38.925814 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925085 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-slash\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.925814 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925125 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-run-netns\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.925814 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925144 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-system-cni-dir\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.925814 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925158 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-var-lib-cni-bin\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.925814 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925175 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-run\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.926441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925184 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4a3f746-8b39-4e1e-8f18-8694b16929e1-konnectivity-ca\") pod \"konnectivity-agent-8s6rw\" (UID: \"e4a3f746-8b39-4e1e-8f18-8694b16929e1\") " pod="kube-system/konnectivity-agent-8s6rw" Nov 02 17:32:38.926441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925192 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-daemon-config\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.926441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925205 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-slash\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.926441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925211 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tszcx\" (UniqueName: \"kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx\") pod \"network-check-target-lnbcn\" (UID: \"558e70e5-ac1c-4205-af45-7ef40a464aed\") " pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:38.926441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925255 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-system-cni-dir\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.926441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925281 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-modprobe-d\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.926441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925299 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-var-lib-cni-bin\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.926441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925328 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-run-netns\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.926441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:38.926441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-sys-fs\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.926441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925375 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-etc-openvswitch\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.926441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925478 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-run-ovn\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.926441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925483 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-etc-openvswitch\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.926441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925506 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b6949d0-0006-4488-89ad-296077fcf045-env-overrides\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.926441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925534 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-sys-fs\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.926441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925571 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b6949d0-0006-4488-89ad-296077fcf045-ovn-node-metrics-cert\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.926441 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925595 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-sys\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.926940 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925622 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-run-ovn\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.926940 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:38.925698 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:38.926940 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925713 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-daemon-config\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.926940 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:38.925744 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret podName:62013dcf-af15-4041-bdc2-f5437c000082 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:39.425729775 +0000 UTC m=+1.992962358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret") pod "global-pull-secret-syncer-drvjl" (UID: "62013dcf-af15-4041-bdc2-f5437c000082") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:38.926940 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925770 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-sys\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.926940 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-cni-bin\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.926940 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-run-netns\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.926940 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925829 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-run-multus-certs\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.926940 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925856 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfmgm\" (UniqueName: \"kubernetes.io/projected/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-kube-api-access-cfmgm\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.926940 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925871 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b6949d0-0006-4488-89ad-296077fcf045-ovnkube-script-lib\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.926940 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925888 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs\") pod \"network-metrics-daemon-zcdw6\" (UID: \"a73fde2c-c65f-4374-adef-ee60a09c0b4a\") " pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:38.926940 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925897 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-cni-bin\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.926940 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925943 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-cnibin\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:38.926940 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925971 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b6949d0-0006-4488-89ad-296077fcf045-env-overrides\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.926940 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.925976 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.926940 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926010 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.926940 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-kubernetes\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.927780 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-sysctl-d\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.927780 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:38.926065 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:38.927780 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926099 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-run-multus-certs\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.927780 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:38.926123 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs podName:a73fde2c-c65f-4374-adef-ee60a09c0b4a nodeName:}" failed. No retries permitted until 2026-11-02 17:32:39.42610594 +0000 UTC m=+1.993338523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs") pod "network-metrics-daemon-zcdw6" (UID: "a73fde2c-c65f-4374-adef-ee60a09c0b4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:38.927780 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926122 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-run-netns\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.927780 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926132 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-sysctl-d\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.927780 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926174 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-kubernetes\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.927780 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926212 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9g7j\" (UniqueName: \"kubernetes.io/projected/e354a328-13db-422e-b0e7-0cc0a5af5a90-kube-api-access-c9g7j\") pod \"node-resolver-278vv\" (UID: \"e354a328-13db-422e-b0e7-0cc0a5af5a90\") " pod="openshift-dns/node-resolver-278vv" Nov 02 17:32:38.927780 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-etc-kubernetes\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.927780 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62013dcf-af15-4041-bdc2-f5437c000082-dbus\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:38.927780 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926318 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-cni-netd\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.927780 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e354a328-13db-422e-b0e7-0cc0a5af5a90-tmp-dir\") pod \"node-resolver-278vv\" (UID: \"e354a328-13db-422e-b0e7-0cc0a5af5a90\") " pod="openshift-dns/node-resolver-278vv" Nov 02 17:32:38.927780 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926412 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-etc-kubernetes\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.927780 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926421 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-cni-binary-copy\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.927780 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926452 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62013dcf-af15-4041-bdc2-f5437c000082-dbus\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:38.927780 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926526 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f-serviceca\") pod \"node-ca-dxmdj\" (UID: \"ffdc716d-d67b-41da-aa12-c3ede5dd6b1f\") " pod="openshift-image-registry/node-ca-dxmdj" Nov 02 17:32:38.927780 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926557 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4a3f746-8b39-4e1e-8f18-8694b16929e1-agent-certs\") pod \"konnectivity-agent-8s6rw\" (UID: \"e4a3f746-8b39-4e1e-8f18-8694b16929e1\") " pod="kube-system/konnectivity-agent-8s6rw" Nov 02 17:32:38.927780 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-log-socket\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.928674 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926609 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3d709e2-f49d-4825-8beb-056b4582f358-etc-tuned\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.928674 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926692 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e354a328-13db-422e-b0e7-0cc0a5af5a90-hosts-file\") pod \"node-resolver-278vv\" (UID: \"e354a328-13db-422e-b0e7-0cc0a5af5a90\") " pod="openshift-dns/node-resolver-278vv" Nov 02 17:32:38.928674 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926694 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b6949d0-0006-4488-89ad-296077fcf045-ovnkube-script-lib\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.928674 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926718 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-conf-dir\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.928674 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926758 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.928674 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926782 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-lib-modules\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.928674 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926803 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-var-lib-kubelet\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.928674 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926840 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-run-ovn-kubernetes\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.928674 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926905 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjhkj\" (UniqueName: \"kubernetes.io/projected/d3d709e2-f49d-4825-8beb-056b4582f358-kube-api-access-xjhkj\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.928674 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926941 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8kcn\" (UniqueName: \"kubernetes.io/projected/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-kube-api-access-h8kcn\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.928674 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.926979 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnhzb\" (UniqueName: \"kubernetes.io/projected/a73fde2c-c65f-4374-adef-ee60a09c0b4a-kube-api-access-nnhzb\") pod \"network-metrics-daemon-zcdw6\" (UID: \"a73fde2c-c65f-4374-adef-ee60a09c0b4a\") " pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:38.928674 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927018 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlxdl\" (UniqueName: \"kubernetes.io/projected/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-kube-api-access-qlxdl\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:38.928674 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-run-systemd\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.928674 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927073 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.928674 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927100 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-socket-dir-parent\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.928674 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927124 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-var-lib-kubelet\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.928674 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927151 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-hostroot\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.929260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927177 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-socket-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.929260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927227 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-cni-binary-copy\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.929260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927309 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/491761ca-d114-411c-9196-3bd0065ae73a-iptables-alerter-script\") pod \"iptables-alerter-nx2vb\" (UID: \"491761ca-d114-411c-9196-3bd0065ae73a\") " pod="openshift-network-operator/iptables-alerter-nx2vb" Nov 02 17:32:38.929260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927326 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-systemd-units\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.929260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-var-lib-openvswitch\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.929260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927355 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-node-log\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.929260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927410 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-node-log\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.929260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927461 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e354a328-13db-422e-b0e7-0cc0a5af5a90-tmp-dir\") pod \"node-resolver-278vv\" (UID: \"e354a328-13db-422e-b0e7-0cc0a5af5a90\") " pod="openshift-dns/node-resolver-278vv" Nov 02 17:32:38.929260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927477 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-socket-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.929260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927506 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-run-ovn-kubernetes\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.929260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927530 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-var-lib-kubelet\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.929260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927551 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.929260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927596 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-socket-dir-parent\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.929260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927615 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-cni-netd\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.929260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.927635 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-hostroot\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.929260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.928103 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-log-socket\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.929260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.928160 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e354a328-13db-422e-b0e7-0cc0a5af5a90-hosts-file\") pod \"node-resolver-278vv\" (UID: \"e354a328-13db-422e-b0e7-0cc0a5af5a90\") " pod="openshift-dns/node-resolver-278vv" Nov 02 17:32:38.929260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.928276 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-conf-dir\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.929734 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.928316 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3d709e2-f49d-4825-8beb-056b4582f358-tmp\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.929734 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.928371 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.929734 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.928457 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b6949d0-0006-4488-89ad-296077fcf045-ovn-node-metrics-cert\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.929734 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.928518 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-systemd-units\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.929734 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.928526 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-run-systemd\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.929734 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.928629 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-lib-modules\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.929734 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.928687 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-var-lib-kubelet\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.929734 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.928737 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-var-lib-openvswitch\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.929734 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.929098 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/491761ca-d114-411c-9196-3bd0065ae73a-iptables-alerter-script\") pod \"iptables-alerter-nx2vb\" (UID: \"491761ca-d114-411c-9196-3bd0065ae73a\") " pod="openshift-network-operator/iptables-alerter-nx2vb" Nov 02 17:32:38.929734 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.929105 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3d709e2-f49d-4825-8beb-056b4582f358-etc-tuned\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:38.930158 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.930141 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4a3f746-8b39-4e1e-8f18-8694b16929e1-agent-certs\") pod \"konnectivity-agent-8s6rw\" (UID: \"e4a3f746-8b39-4e1e-8f18-8694b16929e1\") " pod="kube-system/konnectivity-agent-8s6rw" Nov 02 17:32:38.931597 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.931575 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9w5w\" (UniqueName: \"kubernetes.io/projected/2b6949d0-0006-4488-89ad-296077fcf045-kube-api-access-h9w5w\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:38.933247 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.933226 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7t6q\" (UniqueName: \"kubernetes.io/projected/491761ca-d114-411c-9196-3bd0065ae73a-kube-api-access-k7t6q\") pod \"iptables-alerter-nx2vb\" (UID: \"491761ca-d114-411c-9196-3bd0065ae73a\") " pod="openshift-network-operator/iptables-alerter-nx2vb" Nov 02 17:32:38.934555 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.934501 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9g7j\" (UniqueName: \"kubernetes.io/projected/e354a328-13db-422e-b0e7-0cc0a5af5a90-kube-api-access-c9g7j\") pod \"node-resolver-278vv\" (UID: \"e354a328-13db-422e-b0e7-0cc0a5af5a90\") " pod="openshift-dns/node-resolver-278vv" Nov 02 17:32:38.935248 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.935229 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnhzb\" (UniqueName: \"kubernetes.io/projected/a73fde2c-c65f-4374-adef-ee60a09c0b4a-kube-api-access-nnhzb\") pod \"network-metrics-daemon-zcdw6\" (UID: \"a73fde2c-c65f-4374-adef-ee60a09c0b4a\") " pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:38.936049 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.936032 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8kcn\" (UniqueName: \"kubernetes.io/projected/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-kube-api-access-h8kcn\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:32:38.936151 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.936132 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfmgm\" (UniqueName: \"kubernetes.io/projected/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-kube-api-access-cfmgm\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:38.936290 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:38.936275 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjhkj\" (UniqueName: \"kubernetes.io/projected/d3d709e2-f49d-4825-8beb-056b4582f358-kube-api-access-xjhkj\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:39.003322 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:39.003289 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe0e6ace491eef2c5b9302a33c08d4d.slice/crio-117152fb4e06d47da9759eeef3882fced47fe07ca2e5f9b81d4e457754570e1e WatchSource:0}: Error finding container 117152fb4e06d47da9759eeef3882fced47fe07ca2e5f9b81d4e457754570e1e: Status 404 returned error can't find the container with id 117152fb4e06d47da9759eeef3882fced47fe07ca2e5f9b81d4e457754570e1e Nov 02 17:32:39.005245 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:39.005221 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fad33c464d219dfb7d5fa21dbe7ad22.slice/crio-94abd7d1ee7d72da8ddde0f1c9482b5488ef6fc7b9c41523a7facdad001dfc7c WatchSource:0}: Error finding container 94abd7d1ee7d72da8ddde0f1c9482b5488ef6fc7b9c41523a7facdad001dfc7c: Status 404 returned error can't find the container with id 94abd7d1ee7d72da8ddde0f1c9482b5488ef6fc7b9c41523a7facdad001dfc7c Nov 02 17:32:39.008295 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.008281 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:32:39.028090 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.028069 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:39.028160 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.028099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:39.028160 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.028117 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:39.028160 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.028139 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f-host\") pod \"node-ca-dxmdj\" (UID: \"ffdc716d-d67b-41da-aa12-c3ede5dd6b1f\") " pod="openshift-image-registry/node-ca-dxmdj" Nov 02 17:32:39.028281 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.028162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smnk7\" (UniqueName: \"kubernetes.io/projected/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f-kube-api-access-smnk7\") pod \"node-ca-dxmdj\" (UID: \"ffdc716d-d67b-41da-aa12-c3ede5dd6b1f\") " pod="openshift-image-registry/node-ca-dxmdj" Nov 02 17:32:39.028281 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.028223 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f-host\") pod \"node-ca-dxmdj\" (UID: \"ffdc716d-d67b-41da-aa12-c3ede5dd6b1f\") " pod="openshift-image-registry/node-ca-dxmdj" Nov 02 17:32:39.028281 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.028264 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tszcx\" (UniqueName: \"kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx\") pod \"network-check-target-lnbcn\" (UID: \"558e70e5-ac1c-4205-af45-7ef40a464aed\") " pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:39.028396 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.028318 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-cnibin\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:39.028396 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.028341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f-serviceca\") pod \"node-ca-dxmdj\" (UID: \"ffdc716d-d67b-41da-aa12-c3ede5dd6b1f\") " pod="openshift-image-registry/node-ca-dxmdj" Nov 02 17:32:39.028396 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.028386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxdl\" (UniqueName: \"kubernetes.io/projected/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-kube-api-access-qlxdl\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:39.028542 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.028431 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:39.028542 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.028472 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-system-cni-dir\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:39.028542 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.028477 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-cnibin\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:39.028542 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.028529 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-system-cni-dir\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:39.028775 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.028352 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:39.028775 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.028736 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:39.028775 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.028755 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:39.028921 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.028814 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f-serviceca\") pod \"node-ca-dxmdj\" (UID: \"ffdc716d-d67b-41da-aa12-c3ede5dd6b1f\") " pod="openshift-image-registry/node-ca-dxmdj" Nov 02 17:32:39.029032 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.029014 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:39.034170 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:39.034155 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:32:39.034220 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:39.034172 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:32:39.034220 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:39.034184 2578 projected.go:196] Error preparing data for projected volume kube-api-access-tszcx for pod openshift-network-diagnostics/network-check-target-lnbcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:39.034292 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:39.034226 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx podName:558e70e5-ac1c-4205-af45-7ef40a464aed nodeName:}" failed. No retries permitted until 2026-11-02 17:32:39.534214654 +0000 UTC m=+2.101447237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tszcx" (UniqueName: "kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx") pod "network-check-target-lnbcn" (UID: "558e70e5-ac1c-4205-af45-7ef40a464aed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:39.036882 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.036863 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smnk7\" (UniqueName: \"kubernetes.io/projected/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f-kube-api-access-smnk7\") pod \"node-ca-dxmdj\" (UID: \"ffdc716d-d67b-41da-aa12-c3ede5dd6b1f\") " pod="openshift-image-registry/node-ca-dxmdj" Nov 02 17:32:39.036940 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.036911 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlxdl\" (UniqueName: \"kubernetes.io/projected/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-kube-api-access-qlxdl\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:39.146239 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.146202 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nx2vb" Nov 02 17:32:39.155930 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:39.155902 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491761ca_d114_411c_9196_3bd0065ae73a.slice/crio-ddb4c2750aae48a90222d31bb22af2e9ce23331dcb3818f535773f2bae20e02f WatchSource:0}: Error finding container ddb4c2750aae48a90222d31bb22af2e9ce23331dcb3818f535773f2bae20e02f: Status 404 returned error can't find the container with id ddb4c2750aae48a90222d31bb22af2e9ce23331dcb3818f535773f2bae20e02f Nov 02 17:32:39.179791 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.179773 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:39.182451 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.182432 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:32:39.184702 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.184649 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-278vv" Nov 02 17:32:39.188758 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:39.188737 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6949d0_0006_4488_89ad_296077fcf045.slice/crio-4b005971739cb10211378c9b0a94bf2285179c7c51972f4c35877f4ac3d8fa06 WatchSource:0}: Error finding container 4b005971739cb10211378c9b0a94bf2285179c7c51972f4c35877f4ac3d8fa06: Status 404 returned error can't find the container with id 4b005971739cb10211378c9b0a94bf2285179c7c51972f4c35877f4ac3d8fa06 Nov 02 17:32:39.194648 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:39.194626 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d709e2_f49d_4825_8beb_056b4582f358.slice/crio-f390142867f86c4c61914b08e61903e3572109551e831b859c33dec99941fe29 WatchSource:0}: Error finding container f390142867f86c4c61914b08e61903e3572109551e831b859c33dec99941fe29: Status 404 returned error can't find the container with id f390142867f86c4c61914b08e61903e3572109551e831b859c33dec99941fe29 Nov 02 17:32:39.195713 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:39.195695 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode354a328_13db_422e_b0e7_0cc0a5af5a90.slice/crio-89b301aa1389bd1fdafc0b8dabd74be65a483068163c7370cefae600058e32a0 WatchSource:0}: Error finding container 89b301aa1389bd1fdafc0b8dabd74be65a483068163c7370cefae600058e32a0: Status 404 returned error can't find the container with id 89b301aa1389bd1fdafc0b8dabd74be65a483068163c7370cefae600058e32a0 Nov 02 17:32:39.199352 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.199335 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jw64s" Nov 02 17:32:39.207693 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:39.207672 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e84589_7092_4074_bfc6_d5dd9ea2d4cc.slice/crio-328a3d179ba928d8ce082f2f6fd43397b4c113843153f6d55913dd89af0d158b WatchSource:0}: Error finding container 328a3d179ba928d8ce082f2f6fd43397b4c113843153f6d55913dd89af0d158b: Status 404 returned error can't find the container with id 328a3d179ba928d8ce082f2f6fd43397b4c113843153f6d55913dd89af0d158b Nov 02 17:32:39.217022 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.217005 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8s6rw" Nov 02 17:32:39.226028 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:39.226008 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a3f746_8b39_4e1e_8f18_8694b16929e1.slice/crio-0b4290a8c2bddeff8a97107329477306e785e45f2dcafea75b5829d40eeed550 WatchSource:0}: Error finding container 0b4290a8c2bddeff8a97107329477306e785e45f2dcafea75b5829d40eeed550: Status 404 returned error can't find the container with id 0b4290a8c2bddeff8a97107329477306e785e45f2dcafea75b5829d40eeed550 Nov 02 17:32:39.229023 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.228996 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:32:39.237834 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:39.237815 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6e2a1d_9467_4ef5_b493_6e788c0917b6.slice/crio-d2fdba41a69a6707dad0cee32e99cc6cf3fca17653c509f5d9e05d1d05d4d95e WatchSource:0}: Error finding container d2fdba41a69a6707dad0cee32e99cc6cf3fca17653c509f5d9e05d1d05d4d95e: Status 404 returned error can't find the container with id d2fdba41a69a6707dad0cee32e99cc6cf3fca17653c509f5d9e05d1d05d4d95e Nov 02 17:32:39.249946 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.249927 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dxmdj" Nov 02 17:32:39.258581 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:39.258559 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffdc716d_d67b_41da_aa12_c3ede5dd6b1f.slice/crio-56ac149e79f43a9ad9644a11d2ac29a4f6edae69595c394ce0e3363f373237cc WatchSource:0}: Error finding container 56ac149e79f43a9ad9644a11d2ac29a4f6edae69595c394ce0e3363f373237cc: Status 404 returned error can't find the container with id 56ac149e79f43a9ad9644a11d2ac29a4f6edae69595c394ce0e3363f373237cc Nov 02 17:32:39.265363 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.265344 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:32:39.274099 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:32:39.274080 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636356f6_e4d8_475e_8c82_dbbeb5fee0c3.slice/crio-feb45ffd2f207fab99820b663ea19ea9492c80779f5d187e329bf65dc5f9eef8 WatchSource:0}: Error finding container feb45ffd2f207fab99820b663ea19ea9492c80779f5d187e329bf65dc5f9eef8: Status 404 returned error can't find the container with id feb45ffd2f207fab99820b663ea19ea9492c80779f5d187e329bf65dc5f9eef8 Nov 02 17:32:39.432335 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.432303 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:39.432335 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.432341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs\") pod \"network-metrics-daemon-zcdw6\" (UID: \"a73fde2c-c65f-4374-adef-ee60a09c0b4a\") " pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:39.432587 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:39.432454 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:39.432587 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:39.432454 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:39.432587 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:39.432511 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs podName:a73fde2c-c65f-4374-adef-ee60a09c0b4a nodeName:}" failed. No retries permitted until 2026-11-02 17:32:40.432497337 +0000 UTC m=+2.999729921 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs") pod "network-metrics-daemon-zcdw6" (UID: "a73fde2c-c65f-4374-adef-ee60a09c0b4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:39.432587 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:39.432566 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret podName:62013dcf-af15-4041-bdc2-f5437c000082 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:40.432545976 +0000 UTC m=+2.999778564 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret") pod "global-pull-secret-syncer-drvjl" (UID: "62013dcf-af15-4041-bdc2-f5437c000082") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:39.633582 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.633506 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tszcx\" (UniqueName: \"kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx\") pod \"network-check-target-lnbcn\" (UID: \"558e70e5-ac1c-4205-af45-7ef40a464aed\") " pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:39.633743 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:39.633725 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:32:39.633822 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:39.633752 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:32:39.633822 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:39.633765 2578 projected.go:196] Error preparing data for projected volume kube-api-access-tszcx for pod openshift-network-diagnostics/network-check-target-lnbcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:39.633923 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:39.633825 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx podName:558e70e5-ac1c-4205-af45-7ef40a464aed nodeName:}" failed. No retries permitted until 2026-11-02 17:32:40.633806138 +0000 UTC m=+3.201038727 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tszcx" (UniqueName: "kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx") pod "network-check-target-lnbcn" (UID: "558e70e5-ac1c-4205-af45-7ef40a464aed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:39.711211 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.711173 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:32:39.720482 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.720453 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:32:39.861085 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.861019 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:27:38 +0000 UTC" deadline="2028-05-11 20:40:01.967976472 +0000 UTC" Nov 02 17:32:39.861085 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.861082 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13347h7m22.106898078s" Nov 02 17:32:39.892838 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.892007 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:39.892838 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:39.892131 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:32:39.910806 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.910747 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" event={"ID":"7fad33c464d219dfb7d5fa21dbe7ad22","Type":"ContainerStarted","Data":"94abd7d1ee7d72da8ddde0f1c9482b5488ef6fc7b9c41523a7facdad001dfc7c"} Nov 02 17:32:39.927132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.927045 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z4ptm" event={"ID":"636356f6-e4d8-475e-8c82-dbbeb5fee0c3","Type":"ContainerStarted","Data":"feb45ffd2f207fab99820b663ea19ea9492c80779f5d187e329bf65dc5f9eef8"} Nov 02 17:32:39.942479 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.942436 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8s6rw" event={"ID":"e4a3f746-8b39-4e1e-8f18-8694b16929e1","Type":"ContainerStarted","Data":"0b4290a8c2bddeff8a97107329477306e785e45f2dcafea75b5829d40eeed550"} Nov 02 17:32:39.944168 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.944136 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jw64s" event={"ID":"23e84589-7092-4074-bfc6-d5dd9ea2d4cc","Type":"ContainerStarted","Data":"328a3d179ba928d8ce082f2f6fd43397b4c113843153f6d55913dd89af0d158b"} Nov 02 17:32:39.951754 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.951726 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-278vv" event={"ID":"e354a328-13db-422e-b0e7-0cc0a5af5a90","Type":"ContainerStarted","Data":"89b301aa1389bd1fdafc0b8dabd74be65a483068163c7370cefae600058e32a0"} Nov 02 17:32:39.958557 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.958497 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" event={"ID":"2b6949d0-0006-4488-89ad-296077fcf045","Type":"ContainerStarted","Data":"4b005971739cb10211378c9b0a94bf2285179c7c51972f4c35877f4ac3d8fa06"} Nov 02 17:32:39.964462 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.964435 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" event={"ID":"ebe0e6ace491eef2c5b9302a33c08d4d","Type":"ContainerStarted","Data":"117152fb4e06d47da9759eeef3882fced47fe07ca2e5f9b81d4e457754570e1e"} Nov 02 17:32:39.966245 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.966220 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dxmdj" event={"ID":"ffdc716d-d67b-41da-aa12-c3ede5dd6b1f","Type":"ContainerStarted","Data":"56ac149e79f43a9ad9644a11d2ac29a4f6edae69595c394ce0e3363f373237cc"} Nov 02 17:32:39.972223 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.972199 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" event={"ID":"ae6e2a1d-9467-4ef5-b493-6e788c0917b6","Type":"ContainerStarted","Data":"d2fdba41a69a6707dad0cee32e99cc6cf3fca17653c509f5d9e05d1d05d4d95e"} Nov 02 17:32:39.986329 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.986296 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m78lg" event={"ID":"d3d709e2-f49d-4825-8beb-056b4582f358","Type":"ContainerStarted","Data":"f390142867f86c4c61914b08e61903e3572109551e831b859c33dec99941fe29"} Nov 02 17:32:39.995300 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:39.995269 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nx2vb" event={"ID":"491761ca-d114-411c-9196-3bd0065ae73a","Type":"ContainerStarted","Data":"ddb4c2750aae48a90222d31bb22af2e9ce23331dcb3818f535773f2bae20e02f"} Nov 02 17:32:40.201037 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:40.200948 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:32:40.440369 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:40.440330 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:40.440547 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:40.440391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs\") pod \"network-metrics-daemon-zcdw6\" (UID: \"a73fde2c-c65f-4374-adef-ee60a09c0b4a\") " pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:40.440547 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:40.440531 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:40.440650 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:40.440619 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs podName:a73fde2c-c65f-4374-adef-ee60a09c0b4a nodeName:}" failed. No retries permitted until 2026-11-02 17:32:42.440573828 +0000 UTC m=+5.007806425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs") pod "network-metrics-daemon-zcdw6" (UID: "a73fde2c-c65f-4374-adef-ee60a09c0b4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:40.441094 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:40.441075 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:40.441193 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:40.441137 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret podName:62013dcf-af15-4041-bdc2-f5437c000082 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:42.441121592 +0000 UTC m=+5.008354194 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret") pod "global-pull-secret-syncer-drvjl" (UID: "62013dcf-af15-4041-bdc2-f5437c000082") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:40.642354 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:40.642264 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tszcx\" (UniqueName: \"kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx\") pod \"network-check-target-lnbcn\" (UID: \"558e70e5-ac1c-4205-af45-7ef40a464aed\") " pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:40.642520 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:40.642479 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:32:40.642520 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:40.642503 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:32:40.642520 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:40.642515 2578 projected.go:196] Error preparing data for projected volume kube-api-access-tszcx for pod openshift-network-diagnostics/network-check-target-lnbcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:40.642671 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:40.642569 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx podName:558e70e5-ac1c-4205-af45-7ef40a464aed nodeName:}" failed. No retries permitted until 2026-11-02 17:32:42.642552543 +0000 UTC m=+5.209785129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tszcx" (UniqueName: "kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx") pod "network-check-target-lnbcn" (UID: "558e70e5-ac1c-4205-af45-7ef40a464aed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:40.861647 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:40.861602 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:27:38 +0000 UTC" deadline="2028-04-23 07:24:10.785074845 +0000 UTC" Nov 02 17:32:40.861647 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:40.861645 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12901h51m29.923434163s" Nov 02 17:32:40.890086 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:40.890057 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:40.890265 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:40.890175 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:32:40.890549 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:40.890522 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:40.890638 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:40.890608 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:32:41.893287 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:41.893249 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:41.893885 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:41.893382 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:32:42.459436 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:42.458872 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:42.459436 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:42.458912 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs\") pod \"network-metrics-daemon-zcdw6\" (UID: \"a73fde2c-c65f-4374-adef-ee60a09c0b4a\") " pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:42.459436 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:42.459037 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:42.459436 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:42.459065 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:42.459436 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:42.459099 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs podName:a73fde2c-c65f-4374-adef-ee60a09c0b4a nodeName:}" failed. No retries permitted until 2026-11-02 17:32:46.459079696 +0000 UTC m=+9.026312296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs") pod "network-metrics-daemon-zcdw6" (UID: "a73fde2c-c65f-4374-adef-ee60a09c0b4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:42.459436 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:42.459115 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret podName:62013dcf-af15-4041-bdc2-f5437c000082 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:46.459108133 +0000 UTC m=+9.026340715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret") pod "global-pull-secret-syncer-drvjl" (UID: "62013dcf-af15-4041-bdc2-f5437c000082") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:42.661060 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:42.660593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tszcx\" (UniqueName: \"kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx\") pod \"network-check-target-lnbcn\" (UID: \"558e70e5-ac1c-4205-af45-7ef40a464aed\") " pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:42.661060 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:42.660762 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:32:42.661060 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:42.660782 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:32:42.661060 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:42.660794 2578 projected.go:196] Error preparing data for projected volume kube-api-access-tszcx for pod openshift-network-diagnostics/network-check-target-lnbcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:42.661060 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:42.660851 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx podName:558e70e5-ac1c-4205-af45-7ef40a464aed nodeName:}" failed. No retries permitted until 2026-11-02 17:32:46.660834804 +0000 UTC m=+9.228067409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tszcx" (UniqueName: "kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx") pod "network-check-target-lnbcn" (UID: "558e70e5-ac1c-4205-af45-7ef40a464aed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:42.889585 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:42.889554 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:42.889705 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:42.889681 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:32:42.890112 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:42.890092 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:42.890228 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:42.890201 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:32:43.892234 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:43.891922 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:43.892695 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:43.892293 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:32:44.015866 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:44.015824 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8s6rw" event={"ID":"e4a3f746-8b39-4e1e-8f18-8694b16929e1","Type":"ContainerStarted","Data":"32c168eaaca64b412495b488a27f03874e1468e2f707beab836142b50e839ada"} Nov 02 17:32:44.021336 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:44.021304 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" event={"ID":"ebe0e6ace491eef2c5b9302a33c08d4d","Type":"ContainerStarted","Data":"a33817ba9f2e8217d28287ceafd29d00933024bce4ab5f1fb4433a52461b6707"} Nov 02 17:32:44.025045 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:44.024995 2578 generic.go:356] "Generic (PLEG): container finished" podID="7fad33c464d219dfb7d5fa21dbe7ad22" containerID="dd4b4ea611219a8adc29fcc6357b4d7827cbfd182efd2c7520b801d5b68cdd50" exitCode=0 Nov 02 17:32:44.025151 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:44.025067 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" event={"ID":"7fad33c464d219dfb7d5fa21dbe7ad22","Type":"ContainerDied","Data":"dd4b4ea611219a8adc29fcc6357b4d7827cbfd182efd2c7520b801d5b68cdd50"} Nov 02 17:32:44.030895 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:44.030861 2578 generic.go:356] "Generic (PLEG): container finished" podID="636356f6-e4d8-475e-8c82-dbbeb5fee0c3" containerID="e6ff7ac3d5fcabebbf76cb3a5d890cbf1b4f009e6f4a25fcbda6adc03437e94c" exitCode=0 Nov 02 17:32:44.030895 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:44.030892 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z4ptm" event={"ID":"636356f6-e4d8-475e-8c82-dbbeb5fee0c3","Type":"ContainerDied","Data":"e6ff7ac3d5fcabebbf76cb3a5d890cbf1b4f009e6f4a25fcbda6adc03437e94c"} Nov 02 17:32:44.031721 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:44.031678 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8s6rw" podStartSLOduration=1.743380699 podStartE2EDuration="6.031663449s" totalImagesPullingTime="4.28828275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:44.031273581 +0000 UTC m=+6.598506203" watchObservedRunningTime="2026-11-02 17:32:44.031663449 +0000 UTC m=+6.598896056" Nov 02 17:32:44.080240 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:44.080188 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" podStartSLOduration=6.080170872 podStartE2EDuration="6.080170872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:44.059068686 +0000 UTC m=+6.626301292" watchObservedRunningTime="2026-11-02 17:32:44.080170872 +0000 UTC m=+6.647403480" Nov 02 17:32:44.890075 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:44.890041 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:44.890231 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:44.890178 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:32:44.890272 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:44.890261 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:44.890383 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:44.890362 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:32:45.036936 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:45.036853 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dxmdj" event={"ID":"ffdc716d-d67b-41da-aa12-c3ede5dd6b1f","Type":"ContainerStarted","Data":"eeea0b497849c43a313b857a74a436cc4524c90eeb9da0dc71c5f7defb28164f"} Nov 02 17:32:45.044457 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:45.044427 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" event={"ID":"7fad33c464d219dfb7d5fa21dbe7ad22","Type":"ContainerStarted","Data":"36e71d021375da3d57e1cd4dbb75fa72772ae78607c9654114ab31ce583df154"} Nov 02 17:32:45.071852 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:45.071793 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dxmdj" podStartSLOduration=1.683046244 podStartE2EDuration="7.071774132s" totalImagesPullingTime="5.388727888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:45.053418558 +0000 UTC m=+7.620651163" watchObservedRunningTime="2026-11-02 17:32:45.071774132 +0000 UTC m=+7.639006732" Nov 02 17:32:45.890973 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:45.889231 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:45.890973 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:45.889347 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:32:46.051616 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:46.051130 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" event={"ID":"ae6e2a1d-9467-4ef5-b493-6e788c0917b6","Type":"ContainerStarted","Data":"16716f5f27f7109f01e0508bd1e3952191921c010bda2eed8b58a598fe0c0edc"} Nov 02 17:32:46.495169 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:46.494401 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:46.495169 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:46.494452 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs\") pod \"network-metrics-daemon-zcdw6\" (UID: \"a73fde2c-c65f-4374-adef-ee60a09c0b4a\") " pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:46.495169 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:46.494593 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:46.495169 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:46.494651 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs podName:a73fde2c-c65f-4374-adef-ee60a09c0b4a nodeName:}" failed. No retries permitted until 2026-11-02 17:32:54.49463459 +0000 UTC m=+17.061867193 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs") pod "network-metrics-daemon-zcdw6" (UID: "a73fde2c-c65f-4374-adef-ee60a09c0b4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:46.495169 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:46.495083 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:46.495169 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:46.495134 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret podName:62013dcf-af15-4041-bdc2-f5437c000082 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:54.495120263 +0000 UTC m=+17.062352861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret") pod "global-pull-secret-syncer-drvjl" (UID: "62013dcf-af15-4041-bdc2-f5437c000082") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:46.697159 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:46.696473 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tszcx\" (UniqueName: \"kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx\") pod \"network-check-target-lnbcn\" (UID: \"558e70e5-ac1c-4205-af45-7ef40a464aed\") " pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:46.697159 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:46.696678 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:32:46.697159 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:46.696700 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:32:46.697159 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:46.696709 2578 projected.go:196] Error preparing data for projected volume kube-api-access-tszcx for pod openshift-network-diagnostics/network-check-target-lnbcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:46.697159 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:46.696754 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx podName:558e70e5-ac1c-4205-af45-7ef40a464aed nodeName:}" failed. No retries permitted until 2026-11-02 17:32:54.696739862 +0000 UTC m=+17.263972458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tszcx" (UniqueName: "kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx") pod "network-check-target-lnbcn" (UID: "558e70e5-ac1c-4205-af45-7ef40a464aed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:46.773470 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:46.773433 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8s6rw" Nov 02 17:32:46.775116 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:46.774919 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8s6rw" Nov 02 17:32:46.791525 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:46.791064 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" podStartSLOduration=8.791045235 podStartE2EDuration="8.791045235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:45.072976262 +0000 UTC m=+7.640208869" watchObservedRunningTime="2026-11-02 17:32:46.791045235 +0000 UTC m=+9.358277855" Nov 02 17:32:46.890152 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:46.890117 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:46.890302 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:46.890234 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:32:46.890552 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:46.890531 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:46.890628 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:46.890618 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:32:47.031611 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:47.031587 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 17:32:47.057055 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:47.057021 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" event={"ID":"ae6e2a1d-9467-4ef5-b493-6e788c0917b6","Type":"ContainerStarted","Data":"6a8c785e51993eabcdc098f3283c7a5cb188bbb6e4957f8561c5b7c48dea2d8e"} Nov 02 17:32:47.892301 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:47.892263 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:47.892990 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:47.892369 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:32:47.898507 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:47.898397 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:32:47.031602893Z","UUID":"1a87e950-3279-4cce-bfa6-b528902d76a3","Handler":null,"Name":"","Endpoint":""} Nov 02 17:32:47.900975 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:47.900417 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 17:32:47.900975 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:47.900446 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 17:32:48.061630 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:48.061540 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jw64s" event={"ID":"23e84589-7092-4074-bfc6-d5dd9ea2d4cc","Type":"ContainerStarted","Data":"4c7272db5f286fa82f67c4d99fc7f0115be68b2e0bd95d4dbaa69a488891684f"} Nov 02 17:32:48.064672 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:48.064642 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-278vv" event={"ID":"e354a328-13db-422e-b0e7-0cc0a5af5a90","Type":"ContainerStarted","Data":"5fb937d5663ab91bd758b004b4c23bf64d9508310dfb06b9df7159df773edf24"} Nov 02 17:32:48.080821 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:48.080763 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-jw64s" podStartSLOduration=1.867323234 podStartE2EDuration="10.08074375s" totalImagesPullingTime="8.213420516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:48.079976903 +0000 UTC m=+10.647209503" watchObservedRunningTime="2026-11-02 17:32:48.08074375 +0000 UTC m=+10.647976356" Nov 02 17:32:48.094322 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:48.094280 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-278vv" podStartSLOduration=1.636318675 podStartE2EDuration="10.094263772s" totalImagesPullingTime="8.457945097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:48.09400858 +0000 UTC m=+10.661241188" watchObservedRunningTime="2026-11-02 17:32:48.094263772 +0000 UTC m=+10.661496378" Nov 02 17:32:48.889597 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:48.889391 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:48.889754 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:48.889445 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:48.889754 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:48.889711 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:32:48.889849 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:48.889791 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:32:49.069610 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:49.069570 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" event={"ID":"ae6e2a1d-9467-4ef5-b493-6e788c0917b6","Type":"ContainerStarted","Data":"4de9290e0525233112e8e8baf4b588e5e707317b33bc34f5731e01fe4315e13f"} Nov 02 17:32:49.071594 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:49.071560 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nx2vb" event={"ID":"491761ca-d114-411c-9196-3bd0065ae73a","Type":"ContainerStarted","Data":"9bb74541f2a845a6744672bd2c722d50a310f6603d11ccd3af618ecdec4e0755"} Nov 02 17:32:49.073566 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:49.073542 2578 generic.go:356] "Generic (PLEG): container finished" podID="636356f6-e4d8-475e-8c82-dbbeb5fee0c3" containerID="cdd3bc0e464d0af85dde40bf17031871b4166f05dd023372720f4ccb1fb4e448" exitCode=0 Nov 02 17:32:49.073677 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:49.073584 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z4ptm" event={"ID":"636356f6-e4d8-475e-8c82-dbbeb5fee0c3","Type":"ContainerDied","Data":"cdd3bc0e464d0af85dde40bf17031871b4166f05dd023372720f4ccb1fb4e448"} Nov 02 17:32:49.087727 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:49.087671 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" podStartSLOduration=2.5081974750000002 podStartE2EDuration="11.087654141s" totalImagesPullingTime="8.579456666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:49.087525091 +0000 UTC m=+11.654757696" watchObservedRunningTime="2026-11-02 17:32:49.087654141 +0000 UTC m=+11.654886773" Nov 02 17:32:49.102859 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:49.102796 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nx2vb" podStartSLOduration=2.826577485 podStartE2EDuration="11.102777878s" totalImagesPullingTime="8.276200393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:49.102283963 +0000 UTC m=+11.669516584" watchObservedRunningTime="2026-11-02 17:32:49.102777878 +0000 UTC m=+11.670010485" Nov 02 17:32:49.889550 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:49.889524 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:49.889677 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:49.889657 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:32:50.078154 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:50.078065 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m78lg" event={"ID":"d3d709e2-f49d-4825-8beb-056b4582f358","Type":"ContainerStarted","Data":"919f372b02494fed27eeb2ee6e89d6f211d659d027ecd6fe5cb25644178e639e"} Nov 02 17:32:50.080414 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:50.080389 2578 generic.go:356] "Generic (PLEG): container finished" podID="636356f6-e4d8-475e-8c82-dbbeb5fee0c3" containerID="7dfb7538064fae7b2462e74e8ee259371c749ca27333735d6eed4f50a7df5b6a" exitCode=0 Nov 02 17:32:50.080538 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:50.080475 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z4ptm" event={"ID":"636356f6-e4d8-475e-8c82-dbbeb5fee0c3","Type":"ContainerDied","Data":"7dfb7538064fae7b2462e74e8ee259371c749ca27333735d6eed4f50a7df5b6a"} Nov 02 17:32:50.095341 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:50.095296 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m78lg" podStartSLOduration=1.7837453779999999 podStartE2EDuration="12.095280062s" totalImagesPullingTime="10.311534684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:50.095008036 +0000 UTC m=+12.662240666" watchObservedRunningTime="2026-11-02 17:32:50.095280062 +0000 UTC m=+12.662512668" Nov 02 17:32:50.889639 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:50.889603 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:50.889810 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:50.889711 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:32:50.889810 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:50.889762 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:50.889889 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:50.889860 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:32:51.893259 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:51.893229 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:51.893820 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:51.893358 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:32:52.087222 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:52.087182 2578 generic.go:356] "Generic (PLEG): container finished" podID="636356f6-e4d8-475e-8c82-dbbeb5fee0c3" containerID="3ae4901221914788b39c34e1f0bae225aed8b6c3e553ae75eb4e101b6abb31c2" exitCode=0 Nov 02 17:32:52.087377 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:52.087241 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z4ptm" event={"ID":"636356f6-e4d8-475e-8c82-dbbeb5fee0c3","Type":"ContainerDied","Data":"3ae4901221914788b39c34e1f0bae225aed8b6c3e553ae75eb4e101b6abb31c2"} Nov 02 17:32:52.890014 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:52.889760 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:52.890177 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:52.889840 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:52.890177 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:52.890133 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:32:52.890260 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:52.890217 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:32:53.890191 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:53.890157 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:53.890557 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:53.890264 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:32:54.552419 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:54.552223 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:54.552507 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:54.552448 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs\") pod \"network-metrics-daemon-zcdw6\" (UID: \"a73fde2c-c65f-4374-adef-ee60a09c0b4a\") " pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:54.552507 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:54.552361 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:54.552572 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:54.552549 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:54.552572 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:54.552557 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret podName:62013dcf-af15-4041-bdc2-f5437c000082 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:10.552539006 +0000 UTC m=+33.119771603 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret") pod "global-pull-secret-syncer-drvjl" (UID: "62013dcf-af15-4041-bdc2-f5437c000082") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:54.552634 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:54.552583 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs podName:a73fde2c-c65f-4374-adef-ee60a09c0b4a nodeName:}" failed. No retries permitted until 2026-11-02 17:33:10.552573947 +0000 UTC m=+33.119806544 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs") pod "network-metrics-daemon-zcdw6" (UID: "a73fde2c-c65f-4374-adef-ee60a09c0b4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:54.753260 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:54.753234 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tszcx\" (UniqueName: \"kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx\") pod \"network-check-target-lnbcn\" (UID: \"558e70e5-ac1c-4205-af45-7ef40a464aed\") " pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:54.753459 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:54.753406 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:32:54.753459 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:54.753423 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:32:54.753459 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:54.753433 2578 projected.go:196] Error preparing data for projected volume kube-api-access-tszcx for pod openshift-network-diagnostics/network-check-target-lnbcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:54.753604 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:54.753489 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx podName:558e70e5-ac1c-4205-af45-7ef40a464aed nodeName:}" failed. No retries permitted until 2026-11-02 17:33:10.753473635 +0000 UTC m=+33.320706224 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tszcx" (UniqueName: "kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx") pod "network-check-target-lnbcn" (UID: "558e70e5-ac1c-4205-af45-7ef40a464aed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:54.889845 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:54.889809 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:54.889845 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:54.889836 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:54.890081 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:54.889913 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:32:54.890081 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:54.890031 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:32:55.101819 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:55.101742 2578 generic.go:356] "Generic (PLEG): container finished" podID="636356f6-e4d8-475e-8c82-dbbeb5fee0c3" containerID="fe0320edfb9cfc2ba020b576eed3f42390baba91b4dfcc3322a24ef04802d3b1" exitCode=0 Nov 02 17:32:55.102212 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:55.101818 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z4ptm" event={"ID":"636356f6-e4d8-475e-8c82-dbbeb5fee0c3","Type":"ContainerDied","Data":"fe0320edfb9cfc2ba020b576eed3f42390baba91b4dfcc3322a24ef04802d3b1"} Nov 02 17:32:55.104591 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:55.104572 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 17:32:55.104918 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:55.104900 2578 generic.go:356] "Generic (PLEG): container finished" podID="2b6949d0-0006-4488-89ad-296077fcf045" containerID="018155f005a7df5780c9280646f8753de591b949f01e031efac5de73b98b99a0" exitCode=1 Nov 02 17:32:55.104997 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:55.104940 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" event={"ID":"2b6949d0-0006-4488-89ad-296077fcf045","Type":"ContainerStarted","Data":"c73768a4b11d0c0db6ace8552789cade13bb65a991a42bc869db438155f9df19"} Nov 02 17:32:55.105045 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:55.105005 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" event={"ID":"2b6949d0-0006-4488-89ad-296077fcf045","Type":"ContainerStarted","Data":"4cd9f3aaac01807504ff58ac43821760106aac7a3978f54557f5e4621f483f9b"} Nov 02 17:32:55.105045 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:55.105036 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" event={"ID":"2b6949d0-0006-4488-89ad-296077fcf045","Type":"ContainerStarted","Data":"5fde6e02a3d70d19b91272931cc942b415caabb3d844e88646a7ebd9a70f0b2f"} Nov 02 17:32:55.105138 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:55.105065 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" event={"ID":"2b6949d0-0006-4488-89ad-296077fcf045","Type":"ContainerStarted","Data":"84bf8dd62fa7afc142adb12d41d5349228c0a8db124b85f26b67a151c5d2dea8"} Nov 02 17:32:55.105138 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:55.105102 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" event={"ID":"2b6949d0-0006-4488-89ad-296077fcf045","Type":"ContainerStarted","Data":"f40e292c519d774d4df0d4c6161743ed99a56b85419362f60cf79bd44eaebf07"} Nov 02 17:32:55.105213 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:55.105139 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" event={"ID":"2b6949d0-0006-4488-89ad-296077fcf045","Type":"ContainerDied","Data":"018155f005a7df5780c9280646f8753de591b949f01e031efac5de73b98b99a0"} Nov 02 17:32:55.889563 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:55.889534 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:55.889714 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:55.889628 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:32:56.110048 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:56.110016 2578 generic.go:356] "Generic (PLEG): container finished" podID="636356f6-e4d8-475e-8c82-dbbeb5fee0c3" containerID="611742128b8dd7e48c06f5759cb297eb63c701dcf3011373fa849d8dd433f2c1" exitCode=0 Nov 02 17:32:56.110497 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:56.110066 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z4ptm" event={"ID":"636356f6-e4d8-475e-8c82-dbbeb5fee0c3","Type":"ContainerDied","Data":"611742128b8dd7e48c06f5759cb297eb63c701dcf3011373fa849d8dd433f2c1"} Nov 02 17:32:56.116690 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:56.116667 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:32:56.323774 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:56.323714 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:32:56.889392 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:56.889314 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:56.889392 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:56.889346 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:56.889577 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:56.889416 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:32:56.889577 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:56.889482 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:32:57.007518 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:57.007488 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:32:57.115710 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:57.115680 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z4ptm" event={"ID":"636356f6-e4d8-475e-8c82-dbbeb5fee0c3","Type":"ContainerStarted","Data":"8239a30f886f23851a95bf98635434ffd79afea3d14a67e61f879956c01f04b3"} Nov 02 17:32:57.118537 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:57.118517 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 17:32:57.118871 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:57.118852 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" event={"ID":"2b6949d0-0006-4488-89ad-296077fcf045","Type":"ContainerStarted","Data":"47fc47827b81893c80153844dac6deace141578e020102a0a2adb7f7d4191894"} Nov 02 17:32:57.145668 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:57.145576 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-z4ptm" podStartSLOduration=8.410329595 podStartE2EDuration="19.145564346s" totalImagesPullingTime="10.735234751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:57.144425468 +0000 UTC m=+19.711658084" watchObservedRunningTime="2026-11-02 17:32:57.145564346 +0000 UTC m=+19.712796951" Nov 02 17:32:57.300086 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:57.300043 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:32:57.433291 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:57.433175 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:32:57.890767 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:57.890742 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:57.890906 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:57.890818 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:32:58.890182 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:58.890000 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:32:58.891084 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:58.890314 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:32:58.891084 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:58.890005 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:32:58.891084 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:58.890772 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:32:59.125420 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:59.125325 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 17:32:59.125642 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:59.125620 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" event={"ID":"2b6949d0-0006-4488-89ad-296077fcf045","Type":"ContainerStarted","Data":"886366135b9bd0f5e63056a92af6c176cf7ad3af732b68b69e2dfbdcccdb4f38"} Nov 02 17:32:59.125985 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:59.125944 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:59.126132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:59.126033 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:59.126132 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:59.126042 2578 scope.go:122] "RemoveContainer" containerID="018155f005a7df5780c9280646f8753de591b949f01e031efac5de73b98b99a0" Nov 02 17:32:59.142288 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:59.142267 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:59.143201 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:59.143185 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:32:59.287842 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:59.287810 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:32:59.890027 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:32:59.889991 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:32:59.890222 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:32:59.890111 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:33:00.132028 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:00.132004 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 17:33:00.132368 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:00.132344 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" event={"ID":"2b6949d0-0006-4488-89ad-296077fcf045","Type":"ContainerStarted","Data":"386dafe4906b0b94ca2139ae9c0ca340817d3b1df6ff61fe0bc11044ed59ca91"} Nov 02 17:33:00.132473 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:00.132411 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:33:00.162438 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:00.162395 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" podStartSLOduration=7.3427366 podStartE2EDuration="22.162382367s" totalImagesPullingTime="14.819645767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:00.16199907 +0000 UTC m=+22.729231676" watchObservedRunningTime="2026-11-02 17:33:00.162382367 +0000 UTC m=+22.729614968" Nov 02 17:33:00.435844 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:00.435764 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zcdw6"] Nov 02 17:33:00.436098 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:00.435905 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:00.436098 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:00.436037 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:33:00.442886 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:00.442859 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-drvjl"] Nov 02 17:33:00.443025 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:00.442997 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:00.443096 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:00.443079 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:33:00.443409 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:00.443390 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lnbcn"] Nov 02 17:33:00.443478 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:00.443468 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:00.443544 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:00.443530 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:33:01.135017 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:01.134989 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:33:01.889743 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:01.889709 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:01.889743 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:01.889724 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:01.889743 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:01.889732 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:01.890081 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:01.889814 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:33:01.890081 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:01.889886 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:33:01.890081 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:01.889943 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:33:02.541971 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:02.541930 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:02.735900 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:02.735862 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:02.960354 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:02.960321 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:03.111819 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:03.111748 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:03.252609 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:03.252575 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:03.393066 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:03.392983 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:03.859272 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:03.859243 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:03.890201 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:03.890182 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:03.890201 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:03.890198 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:03.890380 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:03.890263 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:33:03.890380 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:03.890302 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:03.890473 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:03.890397 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:33:03.890523 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:03.890497 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:33:04.019473 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:04.019445 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:04.459713 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:04.459668 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:05.346444 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:05.346403 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:05.490967 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:05.490928 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:05.626249 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:05.626166 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:05.756023 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:05.755987 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:05.889863 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:05.889778 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:05.889863 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:05.889794 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:05.890098 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:05.890003 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:05.890098 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:05.889989 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:33:05.890203 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:05.890101 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:33:05.890203 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:05.890161 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:33:05.893110 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:05.893081 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:05.931000 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:05.930980 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:05.931167 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:05.931156 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:33:05.941711 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:05.941689 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:06.041570 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:06.041535 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:06.713518 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:06.713491 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:06.879119 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:06.879091 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:07.174345 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:07.174273 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:07.378188 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:07.378155 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:07.582364 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:07.582326 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:07.765922 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:07.765894 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:07.890797 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:07.890727 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:07.890946 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:07.890812 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:33:07.890946 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:07.890891 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:07.891051 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:07.891008 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:07.891051 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:07.891034 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:33:07.891137 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:07.891077 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:33:07.938878 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:07.938846 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:08.092149 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:08.092113 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:08.360908 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:08.360880 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:08.674052 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:08.673974 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8s6rw" Nov 02 17:33:08.674215 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:08.674143 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:33:08.674609 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:08.674588 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8s6rw" Nov 02 17:33:09.051810 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:09.051781 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-278vv_e354a328-13db-422e-b0e7-0cc0a5af5a90/dns-node-resolver/0.log" Nov 02 17:33:09.167718 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:09.167690 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:09.364885 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:09.364808 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:09.504426 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:09.504380 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:09.638801 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:09.638725 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:09.776000 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:09.775969 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:09.890274 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:09.890201 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:09.890416 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:09.890202 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:09.890416 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:09.890362 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:09.890416 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:09.890382 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:33:09.890522 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:09.890428 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:33:09.890522 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:09.890513 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:33:09.978989 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:09.978942 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:10.200144 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:10.200108 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:10.251070 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:10.251046 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dxmdj_ffdc716d-d67b-41da-aa12-c3ede5dd6b1f/node-ca/0.log" Nov 02 17:33:10.343470 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:10.343436 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:10.475342 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:10.475313 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:10.567705 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:10.567663 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:10.567859 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:10.567713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs\") pod \"network-metrics-daemon-zcdw6\" (UID: \"a73fde2c-c65f-4374-adef-ee60a09c0b4a\") " pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:10.567859 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:10.567820 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:33:10.567924 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:10.567879 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret podName:62013dcf-af15-4041-bdc2-f5437c000082 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:42.567865651 +0000 UTC m=+65.135098238 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret") pod "global-pull-secret-syncer-drvjl" (UID: "62013dcf-af15-4041-bdc2-f5437c000082") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:33:10.567924 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:10.567820 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:33:10.568026 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:10.567948 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs podName:a73fde2c-c65f-4374-adef-ee60a09c0b4a nodeName:}" failed. No retries permitted until 2026-11-02 17:33:42.567937133 +0000 UTC m=+65.135169729 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs") pod "network-metrics-daemon-zcdw6" (UID: "a73fde2c-c65f-4374-adef-ee60a09c0b4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:33:10.768600 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:10.768529 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tszcx\" (UniqueName: \"kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx\") pod \"network-check-target-lnbcn\" (UID: \"558e70e5-ac1c-4205-af45-7ef40a464aed\") " pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:10.768725 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:10.768679 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:33:10.768725 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:10.768695 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:33:10.768725 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:10.768705 2578 projected.go:196] Error preparing data for projected volume kube-api-access-tszcx for pod openshift-network-diagnostics/network-check-target-lnbcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:33:10.768829 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:10.768748 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx podName:558e70e5-ac1c-4205-af45-7ef40a464aed nodeName:}" failed. No retries permitted until 2026-11-02 17:33:42.768737152 +0000 UTC m=+65.335969740 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tszcx" (UniqueName: "kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx") pod "network-check-target-lnbcn" (UID: "558e70e5-ac1c-4205-af45-7ef40a464aed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:33:11.176150 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:11.176061 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:11.318032 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:11.317989 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:11.450910 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:11.450881 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:11.589250 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:11.589216 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:11.775993 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:11.775890 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:11.890220 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:11.890187 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:11.890398 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:11.890195 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:11.890398 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:11.890292 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:33:11.890398 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:11.890305 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:11.890398 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:11.890377 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:33:11.890633 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:11.890430 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:33:11.994426 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:11.994391 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:13.064370 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:13.064337 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:13.200125 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:13.200097 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:13.378200 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:13.378105 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:13.594160 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:13.594127 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:13.845663 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:13.845628 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:13.889335 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:13.889312 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:13.889470 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:13.889314 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:13.889470 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:13.889408 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:33:13.889650 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:13.889318 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:13.889650 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:13.889513 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:33:13.889650 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:13.889534 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:33:14.005271 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:14.005235 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:14.145853 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:14.145769 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:14.890227 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:14.890198 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:14.890384 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:14.890294 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:33:15.186740 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:15.186656 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:15.388070 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:15.388033 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:15.581392 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:15.581355 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:15.729322 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:15.729291 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:15.889855 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:15.889781 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:15.890019 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:15.889912 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:33:15.890019 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:15.889965 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:15.890109 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:15.890036 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:33:15.894402 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:15.894373 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:16.035077 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:16.035042 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:16.397925 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:16.397878 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:16.889444 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:16.889417 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:16.889613 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:16.889542 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:33:17.890231 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:17.890200 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:17.890741 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:17.890294 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:33:17.890741 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:17.890333 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:17.890741 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:17.890388 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:33:18.232170 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:18.232141 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:18.362514 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:18.362481 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:18.889493 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:18.889458 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:18.889664 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:18.889563 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:33:19.297163 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:19.297133 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:19.438971 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:19.438920 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:19.575258 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:19.575187 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:19.889976 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:19.889890 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:19.889976 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:19.889934 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:19.890140 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:19.890066 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:33:19.890197 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:19.890156 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:33:20.357302 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:20.357269 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:20.483748 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:20.483717 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:20.889371 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:20.889337 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:20.889537 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:20.889442 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:33:21.889792 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:21.889762 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:21.890251 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:21.889863 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:33:21.890251 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:21.889904 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:21.890251 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:21.889984 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:33:22.890207 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:22.890171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:22.890640 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:22.890291 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:33:23.889338 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:23.889305 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:23.889523 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:23.889417 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:23.889523 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:23.889418 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnbcn" podUID="558e70e5-ac1c-4205-af45-7ef40a464aed" Nov 02 17:33:23.889523 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:23.889497 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcdw6" podUID="a73fde2c-c65f-4374-adef-ee60a09c0b4a" Nov 02 17:33:24.485199 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:24.485166 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:24.664446 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:24.664410 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:24.889545 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:24.889519 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:24.889663 ip-10-0-142-133 kubenswrapper[2578]: E1102 17:33:24.889615 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drvjl" podUID="62013dcf-af15-4041-bdc2-f5437c000082" Nov 02 17:33:25.181245 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.181177 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeReady" Nov 02 17:33:25.181378 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.181305 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 17:33:25.227322 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.227283 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-48wvf"] Nov 02 17:33:25.229245 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.229231 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:25.230527 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.230506 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vr49c"] Nov 02 17:33:25.232741 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.232727 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:25.233593 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.233569 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 17:33:25.236907 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.236883 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dp4wq\"" Nov 02 17:33:25.237032 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.236968 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 17:33:25.238919 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.238901 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 17:33:25.241751 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.241721 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-48wvf"] Nov 02 17:33:25.247314 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.247293 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vr49c"] Nov 02 17:33:25.249752 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.249732 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8h628"] Nov 02 17:33:25.251336 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.251319 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8h628" Nov 02 17:33:25.265738 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.265712 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8h628"] Nov 02 17:33:25.334329 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.334302 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 17:33:25.334475 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.334424 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 17:33:25.334525 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.334491 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 17:33:25.334525 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.334509 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 17:33:25.334588 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.334532 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 17:33:25.338154 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.338138 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9wn4g\"" Nov 02 17:33:25.338251 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.338196 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 17:33:25.338419 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.338408 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vxnnv\"" Nov 02 17:33:25.372664 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.372640 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/babf1523-fa08-4397-b0ff-946af7e07278-crio-socket\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:25.372729 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.372675 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3df7744d-a04e-42aa-8938-31b27451b921-tmp-dir\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:25.372729 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.372693 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mgbp\" (UniqueName: \"kubernetes.io/projected/babf1523-fa08-4397-b0ff-946af7e07278-kube-api-access-9mgbp\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:25.372729 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.372711 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/babf1523-fa08-4397-b0ff-946af7e07278-data-volume\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:25.372831 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.372771 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3df7744d-a04e-42aa-8938-31b27451b921-config-volume\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:25.372831 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.372788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3df7744d-a04e-42aa-8938-31b27451b921-metrics-tls\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:25.372899 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.372879 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/babf1523-fa08-4397-b0ff-946af7e07278-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:25.372938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.372912 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67a04d53-e707-4b0b-9e32-d84c31e413e5-cert\") pod \"ingress-canary-8h628\" (UID: \"67a04d53-e707-4b0b-9e32-d84c31e413e5\") " pod="openshift-ingress-canary/ingress-canary-8h628" Nov 02 17:33:25.372938 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.372928 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/babf1523-fa08-4397-b0ff-946af7e07278-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:25.373023 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.372994 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zmv4\" (UniqueName: \"kubernetes.io/projected/67a04d53-e707-4b0b-9e32-d84c31e413e5-kube-api-access-6zmv4\") pod \"ingress-canary-8h628\" (UID: \"67a04d53-e707-4b0b-9e32-d84c31e413e5\") " pod="openshift-ingress-canary/ingress-canary-8h628" Nov 02 17:33:25.373066 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.373025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56frp\" (UniqueName: \"kubernetes.io/projected/3df7744d-a04e-42aa-8938-31b27451b921-kube-api-access-56frp\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:25.473469 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.473436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/babf1523-fa08-4397-b0ff-946af7e07278-data-volume\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:25.473584 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.473477 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3df7744d-a04e-42aa-8938-31b27451b921-config-volume\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:25.473584 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.473493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3df7744d-a04e-42aa-8938-31b27451b921-metrics-tls\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:25.473584 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.473529 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/babf1523-fa08-4397-b0ff-946af7e07278-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:25.473584 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.473556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67a04d53-e707-4b0b-9e32-d84c31e413e5-cert\") pod \"ingress-canary-8h628\" (UID: \"67a04d53-e707-4b0b-9e32-d84c31e413e5\") " pod="openshift-ingress-canary/ingress-canary-8h628" Nov 02 17:33:25.473584 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.473570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/babf1523-fa08-4397-b0ff-946af7e07278-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:25.473884 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.473607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zmv4\" (UniqueName: \"kubernetes.io/projected/67a04d53-e707-4b0b-9e32-d84c31e413e5-kube-api-access-6zmv4\") pod \"ingress-canary-8h628\" (UID: \"67a04d53-e707-4b0b-9e32-d84c31e413e5\") " pod="openshift-ingress-canary/ingress-canary-8h628" Nov 02 17:33:25.473884 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.473629 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56frp\" (UniqueName: \"kubernetes.io/projected/3df7744d-a04e-42aa-8938-31b27451b921-kube-api-access-56frp\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:25.473884 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.473670 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/babf1523-fa08-4397-b0ff-946af7e07278-crio-socket\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:25.473884 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.473710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3df7744d-a04e-42aa-8938-31b27451b921-tmp-dir\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:25.473884 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.473734 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mgbp\" (UniqueName: \"kubernetes.io/projected/babf1523-fa08-4397-b0ff-946af7e07278-kube-api-access-9mgbp\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:25.473884 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.473878 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/babf1523-fa08-4397-b0ff-946af7e07278-data-volume\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:25.474186 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.474096 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/babf1523-fa08-4397-b0ff-946af7e07278-crio-socket\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:25.474224 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.474201 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3df7744d-a04e-42aa-8938-31b27451b921-config-volume\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:25.474271 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.474209 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/babf1523-fa08-4397-b0ff-946af7e07278-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:25.474310 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.474279 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3df7744d-a04e-42aa-8938-31b27451b921-tmp-dir\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:25.477504 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.477480 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67a04d53-e707-4b0b-9e32-d84c31e413e5-cert\") pod \"ingress-canary-8h628\" (UID: \"67a04d53-e707-4b0b-9e32-d84c31e413e5\") " pod="openshift-ingress-canary/ingress-canary-8h628" Nov 02 17:33:25.477504 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.477490 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3df7744d-a04e-42aa-8938-31b27451b921-metrics-tls\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:25.477629 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.477490 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/babf1523-fa08-4397-b0ff-946af7e07278-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:25.484996 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.484974 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56frp\" (UniqueName: \"kubernetes.io/projected/3df7744d-a04e-42aa-8938-31b27451b921-kube-api-access-56frp\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:25.485815 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.485799 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mgbp\" (UniqueName: \"kubernetes.io/projected/babf1523-fa08-4397-b0ff-946af7e07278-kube-api-access-9mgbp\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:25.486083 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.486005 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zmv4\" (UniqueName: \"kubernetes.io/projected/67a04d53-e707-4b0b-9e32-d84c31e413e5-kube-api-access-6zmv4\") pod \"ingress-canary-8h628\" (UID: \"67a04d53-e707-4b0b-9e32-d84c31e413e5\") " pod="openshift-ingress-canary/ingress-canary-8h628" Nov 02 17:33:25.547999 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.547970 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:25.552881 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.552860 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:25.559617 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.559597 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8h628" Nov 02 17:33:25.843575 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.843503 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8h628"] Nov 02 17:33:25.849238 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.847917 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vr49c"] Nov 02 17:33:25.849238 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.848002 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-48wvf"] Nov 02 17:33:25.851143 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:33:25.851120 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df7744d_a04e_42aa_8938_31b27451b921.slice/crio-fc423511a76bc5316042893587560d7579bd92ccbe01c5cfae00d72c7c6d986d WatchSource:0}: Error finding container fc423511a76bc5316042893587560d7579bd92ccbe01c5cfae00d72c7c6d986d: Status 404 returned error can't find the container with id fc423511a76bc5316042893587560d7579bd92ccbe01c5cfae00d72c7c6d986d Nov 02 17:33:25.851501 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:33:25.851382 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabf1523_fa08_4397_b0ff_946af7e07278.slice/crio-33140d4e6a3933461db90cac160879545d0b84614a36600d3b5dffbf02127232 WatchSource:0}: Error finding container 33140d4e6a3933461db90cac160879545d0b84614a36600d3b5dffbf02127232: Status 404 returned error can't find the container with id 33140d4e6a3933461db90cac160879545d0b84614a36600d3b5dffbf02127232 Nov 02 17:33:25.890172 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.890147 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:25.890269 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.890187 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:25.894394 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.894371 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 17:33:25.894486 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.894372 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 17:33:25.894603 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.894586 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 17:33:25.894603 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.894599 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jmxfr\"" Nov 02 17:33:25.894685 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:25.894586 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-52q82\"" Nov 02 17:33:26.187569 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:26.187481 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8h628" event={"ID":"67a04d53-e707-4b0b-9e32-d84c31e413e5","Type":"ContainerStarted","Data":"467b0f4d69f4df830fb36f0cbb9bd27f32d920c9ecc16042114bf4434a1f1987"} Nov 02 17:33:26.188879 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:26.188855 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vr49c" event={"ID":"babf1523-fa08-4397-b0ff-946af7e07278","Type":"ContainerStarted","Data":"31c74c4f82536f519fb5272fc19934b5230d56317b06858625f3f3dab57db97d"} Nov 02 17:33:26.189014 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:26.188908 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vr49c" event={"ID":"babf1523-fa08-4397-b0ff-946af7e07278","Type":"ContainerStarted","Data":"33140d4e6a3933461db90cac160879545d0b84614a36600d3b5dffbf02127232"} Nov 02 17:33:26.189870 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:26.189851 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-48wvf" event={"ID":"3df7744d-a04e-42aa-8938-31b27451b921","Type":"ContainerStarted","Data":"fc423511a76bc5316042893587560d7579bd92ccbe01c5cfae00d72c7c6d986d"} Nov 02 17:33:26.890089 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:26.890056 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:26.894858 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:26.894794 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dcw65\"" Nov 02 17:33:26.896091 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:26.896071 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 17:33:27.194555 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.194475 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vr49c" event={"ID":"babf1523-fa08-4397-b0ff-946af7e07278","Type":"ContainerStarted","Data":"1ce5c1a670811a0c460e7bdf895cc40ba2580c8f70edb89c710770be0203b0b5"} Nov 02 17:33:27.512508 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.512469 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-5994g"] Nov 02 17:33:27.515667 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.515623 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:27.527906 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.527880 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 17:33:27.528077 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.527978 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 17:33:27.528285 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.528263 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-5994g"] Nov 02 17:33:27.528424 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.528371 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qw8dn\"" Nov 02 17:33:27.529917 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.528682 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 17:33:27.529917 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.528802 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 17:33:27.529917 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.529499 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 17:33:27.551626 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.551584 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:27.691718 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.691684 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c893dfcd-6f72-45bd-96ab-521e97584a40-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:27.691895 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.691819 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c893dfcd-6f72-45bd-96ab-521e97584a40-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:27.691895 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.691862 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c893dfcd-6f72-45bd-96ab-521e97584a40-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:27.692038 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.691919 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhqlv\" (UniqueName: \"kubernetes.io/projected/c893dfcd-6f72-45bd-96ab-521e97584a40-kube-api-access-bhqlv\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:27.696448 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.696413 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:27.793309 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.793282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhqlv\" (UniqueName: \"kubernetes.io/projected/c893dfcd-6f72-45bd-96ab-521e97584a40-kube-api-access-bhqlv\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:27.793459 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.793323 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c893dfcd-6f72-45bd-96ab-521e97584a40-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:27.793459 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.793406 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c893dfcd-6f72-45bd-96ab-521e97584a40-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:27.793459 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.793435 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c893dfcd-6f72-45bd-96ab-521e97584a40-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:27.794452 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.794406 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c893dfcd-6f72-45bd-96ab-521e97584a40-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:27.796707 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.796071 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c893dfcd-6f72-45bd-96ab-521e97584a40-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:27.796707 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.796389 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c893dfcd-6f72-45bd-96ab-521e97584a40-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:27.807059 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.807038 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhqlv\" (UniqueName: \"kubernetes.io/projected/c893dfcd-6f72-45bd-96ab-521e97584a40-kube-api-access-bhqlv\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:27.831732 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.831708 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:27.842287 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:27.841965 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:28.044977 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.042009 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:28.121755 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.120654 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-5994g"] Nov 02 17:33:28.125651 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:33:28.125622 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc893dfcd_6f72_45bd_96ab_521e97584a40.slice/crio-4c3c9a9c9257b3ab55d41cf4c87481c5756c0be2fa8ef59da9153ad33b2f5b3b WatchSource:0}: Error finding container 4c3c9a9c9257b3ab55d41cf4c87481c5756c0be2fa8ef59da9153ad33b2f5b3b: Status 404 returned error can't find the container with id 4c3c9a9c9257b3ab55d41cf4c87481c5756c0be2fa8ef59da9153ad33b2f5b3b Nov 02 17:33:28.200329 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.200290 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-48wvf" event={"ID":"3df7744d-a04e-42aa-8938-31b27451b921","Type":"ContainerStarted","Data":"8181f81f728b790cefd76ff035c7868acc931ae7309a691794c2f87f63bc63ce"} Nov 02 17:33:28.200524 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.200354 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-48wvf" event={"ID":"3df7744d-a04e-42aa-8938-31b27451b921","Type":"ContainerStarted","Data":"428e09b49e8cd3ee2ab71e508f3b7db242d556a9bd46026eec43345aa5fb87f6"} Nov 02 17:33:28.200524 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.200386 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:28.201944 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.201901 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" event={"ID":"c893dfcd-6f72-45bd-96ab-521e97584a40","Type":"ContainerStarted","Data":"4c3c9a9c9257b3ab55d41cf4c87481c5756c0be2fa8ef59da9153ad33b2f5b3b"} Nov 02 17:33:28.220151 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.220085 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-48wvf" podStartSLOduration=1.349994053 podStartE2EDuration="3.220069856s" totalImagesPullingTime="1.870075803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:28.219679793 +0000 UTC m=+50.786912398" watchObservedRunningTime="2026-11-02 17:33:28.220069856 +0000 UTC m=+50.787302461" Nov 02 17:33:28.756625 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.756593 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5997464558-6x2v8"] Nov 02 17:33:28.758801 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.758778 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:28.763313 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.763287 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h4n2v\"" Nov 02 17:33:28.763473 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.763357 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 17:33:28.763473 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.763373 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 17:33:28.763473 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.763473 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 17:33:28.763630 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.763493 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 17:33:28.763630 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.763547 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 17:33:28.764023 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.764005 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 17:33:28.764209 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.764189 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 17:33:28.768235 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.768214 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 17:33:28.775412 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.775390 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5997464558-6x2v8"] Nov 02 17:33:28.904098 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.904017 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53eac522-fee5-4865-955b-a14da456d693-tmp\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:28.904263 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.904162 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-service-ca\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:28.904263 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.904209 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqx5s\" (UniqueName: \"kubernetes.io/projected/53eac522-fee5-4865-955b-a14da456d693-kube-api-access-hqx5s\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:28.904263 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.904241 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-trusted-ca-bundle\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:28.904263 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.904255 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-oauth-serving-cert\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:28.904434 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.904389 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-oauth-config\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:28.904434 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.904412 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-serving-cert\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:28.904434 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:28.904428 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-console-config\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:29.005575 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.005540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53eac522-fee5-4865-955b-a14da456d693-tmp\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:29.005748 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.005603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-service-ca\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:29.005748 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.005628 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqx5s\" (UniqueName: \"kubernetes.io/projected/53eac522-fee5-4865-955b-a14da456d693-kube-api-access-hqx5s\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:29.005748 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.005657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-trusted-ca-bundle\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:29.005748 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.005682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-oauth-serving-cert\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:29.005748 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.005719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-oauth-config\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:29.005748 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.005745 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-serving-cert\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:29.006087 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.005769 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-console-config\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:29.006145 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.006101 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53eac522-fee5-4865-955b-a14da456d693-tmp\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:29.006720 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.006700 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-service-ca\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:29.006825 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.006701 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-console-config\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:29.006825 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.006731 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-oauth-serving-cert\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:29.006940 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.006859 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-trusted-ca-bundle\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:29.008658 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.008630 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-oauth-config\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:29.010122 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.010098 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-serving-cert\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:29.014541 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.014523 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqx5s\" (UniqueName: \"kubernetes.io/projected/53eac522-fee5-4865-955b-a14da456d693-kube-api-access-hqx5s\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:29.071501 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.071462 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:29.206187 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.206146 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8h628" event={"ID":"67a04d53-e707-4b0b-9e32-d84c31e413e5","Type":"ContainerStarted","Data":"8a5fa2b0fa3994a6f5aa4ae4130c3943fe4b20ad32b9fdde8cccf82855479eb7"} Nov 02 17:33:29.208381 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.208351 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vr49c" event={"ID":"babf1523-fa08-4397-b0ff-946af7e07278","Type":"ContainerStarted","Data":"b23f1d07ef4f68da0503569e15b802e5e4f2837bae90b877fbc0c2851cef733b"} Nov 02 17:33:29.224622 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.224580 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8h628" podStartSLOduration=1.497619872 podStartE2EDuration="4.22456763s" totalImagesPullingTime="2.726947758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:29.223168439 +0000 UTC m=+51.790401045" watchObservedRunningTime="2026-11-02 17:33:29.22456763 +0000 UTC m=+51.791800234" Nov 02 17:33:29.242000 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.241925 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vr49c" podStartSLOduration=1.891975841 podStartE2EDuration="4.241909556s" totalImagesPullingTime="2.349933715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:33:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:29.241167268 +0000 UTC m=+51.808399874" watchObservedRunningTime="2026-11-02 17:33:29.241909556 +0000 UTC m=+51.809142161" Nov 02 17:33:29.345481 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.345431 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5997464558-6x2v8"] Nov 02 17:33:29.348443 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:33:29.348414 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53eac522_fee5_4865_955b_a14da456d693.slice/crio-5a5511f9cffc9fac7dd2b70f2be71a0228fca3de17c8f3f98cd11efcbcd93c3e WatchSource:0}: Error finding container 5a5511f9cffc9fac7dd2b70f2be71a0228fca3de17c8f3f98cd11efcbcd93c3e: Status 404 returned error can't find the container with id 5a5511f9cffc9fac7dd2b70f2be71a0228fca3de17c8f3f98cd11efcbcd93c3e Nov 02 17:33:29.746322 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.746248 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:29.883586 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:29.883549 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:30.032151 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:30.032068 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:30.214009 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:30.213948 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5997464558-6x2v8" event={"ID":"53eac522-fee5-4865-955b-a14da456d693","Type":"ContainerStarted","Data":"5a5511f9cffc9fac7dd2b70f2be71a0228fca3de17c8f3f98cd11efcbcd93c3e"} Nov 02 17:33:30.216082 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:30.216050 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" event={"ID":"c893dfcd-6f72-45bd-96ab-521e97584a40","Type":"ContainerStarted","Data":"69069ebacf361f15fb09005bb6806ff36f41c9e941c0675c76dc6a7098cc239c"} Nov 02 17:33:30.216223 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:30.216104 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" event={"ID":"c893dfcd-6f72-45bd-96ab-521e97584a40","Type":"ContainerStarted","Data":"4c3a70d3ccd044642cf5ea00f9b379d64b0b944cc054f5d2b86e33f14d4b6dfd"} Nov 02 17:33:30.245009 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:30.244966 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" podStartSLOduration=1.987201989 podStartE2EDuration="3.244940361s" totalImagesPullingTime="1.257738372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:30.243933632 +0000 UTC m=+52.811166250" watchObservedRunningTime="2026-11-02 17:33:30.244940361 +0000 UTC m=+52.812172965" Nov 02 17:33:31.724429 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:31.724393 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:31.882803 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:31.882770 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:31.927403 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:31.927372 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-47799"] Nov 02 17:33:31.931091 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:31.931066 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:31.989867 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:31.989775 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 17:33:31.989867 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:31.989787 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 17:33:31.989867 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:31.989821 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 17:33:31.989867 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:31.989834 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6p4gz\"" Nov 02 17:33:32.027139 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.027106 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:32.029028 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.029009 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-wtmp\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.029109 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.029037 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.029109 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.029062 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21691b74-73cc-41de-9c08-0f5e27528dba-metrics-client-ca\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.029192 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.029169 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9wbw\" (UniqueName: \"kubernetes.io/projected/21691b74-73cc-41de-9c08-0f5e27528dba-kube-api-access-n9wbw\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.029235 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.029190 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21691b74-73cc-41de-9c08-0f5e27528dba-root\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.029235 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.029213 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-accelerators-collector-config\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.029235 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.029233 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-tls\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.029338 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.029266 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21691b74-73cc-41de-9c08-0f5e27528dba-sys\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.029369 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.029335 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-textfile\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.130365 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.130333 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9wbw\" (UniqueName: \"kubernetes.io/projected/21691b74-73cc-41de-9c08-0f5e27528dba-kube-api-access-n9wbw\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.130365 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.130368 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21691b74-73cc-41de-9c08-0f5e27528dba-root\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.130538 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.130395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-accelerators-collector-config\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.130538 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.130436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-tls\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.130538 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.130458 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21691b74-73cc-41de-9c08-0f5e27528dba-sys\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.130538 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.130456 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21691b74-73cc-41de-9c08-0f5e27528dba-root\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.130538 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.130484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-textfile\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.130538 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.130528 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-wtmp\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.130785 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.130535 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21691b74-73cc-41de-9c08-0f5e27528dba-sys\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.130785 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.130553 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.130785 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.130594 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21691b74-73cc-41de-9c08-0f5e27528dba-metrics-client-ca\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.130785 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.130674 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-wtmp\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.131023 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.130982 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-textfile\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.131141 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.131126 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-accelerators-collector-config\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.131202 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.131139 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21691b74-73cc-41de-9c08-0f5e27528dba-metrics-client-ca\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.132815 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.132798 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-tls\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.132893 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.132837 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.141847 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.141828 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9wbw\" (UniqueName: \"kubernetes.io/projected/21691b74-73cc-41de-9c08-0f5e27528dba-kube-api-access-n9wbw\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.164576 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.164551 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:32.223458 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.223429 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5997464558-6x2v8" event={"ID":"53eac522-fee5-4865-955b-a14da456d693","Type":"ContainerStarted","Data":"598fc09e0cec98cce0d54ddd1af221d7293a81a36e99723e74fbc32c33e6de51"} Nov 02 17:33:32.263543 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.263400 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:32.263543 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.263368 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5997464558-6x2v8" podStartSLOduration=1.780233533 podStartE2EDuration="4.263346573s" totalImagesPullingTime="2.48311304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:32.242845088 +0000 UTC m=+54.810077690" watchObservedRunningTime="2026-11-02 17:33:32.263346573 +0000 UTC m=+54.830579169" Nov 02 17:33:32.274394 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:33:32.274369 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21691b74_73cc_41de_9c08_0f5e27528dba.slice/crio-b56787b1149f33c8e1c2e99c8075b6d1d15514e11a0f42716ef22c26ddf92785 WatchSource:0}: Error finding container b56787b1149f33c8e1c2e99c8075b6d1d15514e11a0f42716ef22c26ddf92785: Status 404 returned error can't find the container with id b56787b1149f33c8e1c2e99c8075b6d1d15514e11a0f42716ef22c26ddf92785 Nov 02 17:33:32.307826 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.307794 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:32.447111 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.447074 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:32.596974 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.596875 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:32.945283 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.945250 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:32.967412 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.967386 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:33:32.971184 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.971163 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:32.975216 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.974946 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 17:33:32.975216 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.974973 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 17:33:32.975216 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.975039 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 17:33:32.975216 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.975102 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 17:33:32.975216 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.975114 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 17:33:32.975216 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.975152 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 17:33:32.975216 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.975166 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 17:33:32.975216 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.975187 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gql28\"" Nov 02 17:33:32.975879 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.975857 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 17:33:32.990898 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:32.990872 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:33:33.076274 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.076240 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 17:33:33.137936 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.137895 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-config-out\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.138110 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.137965 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.138110 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.137996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.138110 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.138023 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.138110 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.138077 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-web-config\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.138312 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.138158 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.138312 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.138281 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhglv\" (UniqueName: \"kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-kube-api-access-bhglv\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.138409 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.138348 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.138409 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.138386 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.138512 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.138414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.138512 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.138479 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.138611 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.138554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.138648 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.138633 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.149562 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.149527 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:33.227994 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.227969 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-47799" event={"ID":"21691b74-73cc-41de-9c08-0f5e27528dba","Type":"ContainerStarted","Data":"b56787b1149f33c8e1c2e99c8075b6d1d15514e11a0f42716ef22c26ddf92785"} Nov 02 17:33:33.239730 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.239710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.239793 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.239739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.239793 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.239756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.239793 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.239770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-web-config\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.239793 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.239789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.240002 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.239819 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhglv\" (UniqueName: \"kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-kube-api-access-bhglv\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.240002 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.239853 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.240002 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.239872 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.240002 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.239895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.240002 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.239919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.240002 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.239965 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.240269 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.240015 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.240269 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.240058 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-config-out\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.240269 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.240185 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.241765 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.241739 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.242048 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.242024 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.242928 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.242902 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.243243 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.243220 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.243601 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.243575 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-config-out\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.243876 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.243846 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.244078 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.244060 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.244521 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.244503 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.244999 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.244980 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.244999 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.244993 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.245344 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.245327 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-web-config\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.247535 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.247515 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhglv\" (UniqueName: \"kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-kube-api-access-bhglv\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.296879 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.296851 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:33.446127 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.446086 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:33.577495 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.577398 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:33:33.579943 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:33:33.579916 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b083135_c135_437c_9446_5267ab9cb71f.slice/crio-33cc3c3636522c210a2c55e06391e20ffab4fd130b5a0973557180aa5e1650fb WatchSource:0}: Error finding container 33cc3c3636522c210a2c55e06391e20ffab4fd130b5a0973557180aa5e1650fb: Status 404 returned error can't find the container with id 33cc3c3636522c210a2c55e06391e20ffab4fd130b5a0973557180aa5e1650fb Nov 02 17:33:33.589496 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:33.589470 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:34.233547 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:34.233514 2578 generic.go:356] "Generic (PLEG): container finished" podID="21691b74-73cc-41de-9c08-0f5e27528dba" containerID="f4e37af1131c13b2208b02984bdf354d26e9a581774b63b44c9ab0130986b31e" exitCode=0 Nov 02 17:33:34.234049 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:34.233597 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-47799" event={"ID":"21691b74-73cc-41de-9c08-0f5e27528dba","Type":"ContainerDied","Data":"f4e37af1131c13b2208b02984bdf354d26e9a581774b63b44c9ab0130986b31e"} Nov 02 17:33:34.235528 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:34.235506 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b083135-c135-437c-9446-5267ab9cb71f","Type":"ContainerStarted","Data":"33cc3c3636522c210a2c55e06391e20ffab4fd130b5a0973557180aa5e1650fb"} Nov 02 17:33:34.965232 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:34.965183 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:35.197377 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:35.197340 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:35.240404 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:35.240323 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-47799" event={"ID":"21691b74-73cc-41de-9c08-0f5e27528dba","Type":"ContainerStarted","Data":"bb69a5c5c7582a30f3e2028227fa7694edc2046b7a174e3ea8909ef7b9996fdc"} Nov 02 17:33:35.240839 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:35.240410 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-47799" event={"ID":"21691b74-73cc-41de-9c08-0f5e27528dba","Type":"ContainerStarted","Data":"fb2f82d9213515a842b215445887a3ef34dcbf94e14cdc89b02ada93b656a795"} Nov 02 17:33:35.241845 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:35.241823 2578 generic.go:356] "Generic (PLEG): container finished" podID="1b083135-c135-437c-9446-5267ab9cb71f" containerID="1e32330efff8fe8589858662cbf717d812c6465e667dcc1f671e29ea86034ab7" exitCode=0 Nov 02 17:33:35.241934 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:35.241873 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b083135-c135-437c-9446-5267ab9cb71f","Type":"ContainerDied","Data":"1e32330efff8fe8589858662cbf717d812c6465e667dcc1f671e29ea86034ab7"} Nov 02 17:33:35.262059 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:35.262018 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-47799" podStartSLOduration=3.240479848 podStartE2EDuration="4.262007256s" totalImagesPullingTime="948.193327ms" totalInitContainerRuntime="73.334081ms" isStatefulPod=true podCreationTimestamp="2026-11-02 17:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:35.260678004 +0000 UTC m=+57.827910610" watchObservedRunningTime="2026-11-02 17:33:35.262007256 +0000 UTC m=+57.829239882" Nov 02 17:33:35.340066 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:35.340033 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:35.536574 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:35.536495 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:36.742786 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:36.742684 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:36.908996 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:36.908918 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:37.061214 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.061167 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:37.114758 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.114731 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-fd64789dc-hcqt6"] Nov 02 17:33:37.118074 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.118059 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.122258 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.122240 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 17:33:37.123311 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.123294 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 17:33:37.123396 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.123334 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 17:33:37.126195 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.125666 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 17:33:37.126405 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.126380 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 17:33:37.127187 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.127167 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-smp64\"" Nov 02 17:33:37.128453 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.128434 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 17:33:37.131790 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.131770 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-fd64789dc-hcqt6"] Nov 02 17:33:37.170929 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.170902 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/290c0bd1-b749-4ec9-88e1-384ac37be16f-metrics-client-ca\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.171062 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.170941 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.171062 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.171038 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-secret-telemeter-client\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.171163 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.171087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-telemeter-client-tls\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.171198 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.171174 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvw4s\" (UniqueName: \"kubernetes.io/projected/290c0bd1-b749-4ec9-88e1-384ac37be16f-kube-api-access-gvw4s\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.171227 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.171200 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-federate-client-tls\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.171227 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.171222 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/290c0bd1-b749-4ec9-88e1-384ac37be16f-serving-certs-ca-bundle\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.171308 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.171247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/290c0bd1-b749-4ec9-88e1-384ac37be16f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.251497 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.251467 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b083135-c135-437c-9446-5267ab9cb71f","Type":"ContainerStarted","Data":"e7b7ed98d053e64b85b97cea7ff349f2132a9a6eb40a4cd485466ae34a81fc3c"} Nov 02 17:33:37.251657 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.251532 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b083135-c135-437c-9446-5267ab9cb71f","Type":"ContainerStarted","Data":"51ff36c3b5f28fc2eef6f476ee40097f0be3b41a5506e631d42efc3c56da1b64"} Nov 02 17:33:37.251657 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.251557 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b083135-c135-437c-9446-5267ab9cb71f","Type":"ContainerStarted","Data":"d1a1531f37c2c54e81a18a913cfdd1b61dc5031fa128c3c420e27d5e058cef36"} Nov 02 17:33:37.251657 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.251579 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b083135-c135-437c-9446-5267ab9cb71f","Type":"ContainerStarted","Data":"7f8bee42ef2baea4c975aa3066e5bbcc94be4a7e5bf918e440d9fe573893f266"} Nov 02 17:33:37.251657 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.251604 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b083135-c135-437c-9446-5267ab9cb71f","Type":"ContainerStarted","Data":"32100d40e27e116fa94ce3cd4f88a18c31d2f73730b501aae36e15387d8bd2fb"} Nov 02 17:33:37.271850 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.271825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/290c0bd1-b749-4ec9-88e1-384ac37be16f-metrics-client-ca\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.271898 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.271863 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.271898 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.271884 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-secret-telemeter-client\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.271995 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.271914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-telemeter-client-tls\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.271995 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.271984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvw4s\" (UniqueName: \"kubernetes.io/projected/290c0bd1-b749-4ec9-88e1-384ac37be16f-kube-api-access-gvw4s\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.272094 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.272016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-federate-client-tls\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.272094 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.272037 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/290c0bd1-b749-4ec9-88e1-384ac37be16f-serving-certs-ca-bundle\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.272094 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.272071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/290c0bd1-b749-4ec9-88e1-384ac37be16f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.272852 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.272790 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/290c0bd1-b749-4ec9-88e1-384ac37be16f-serving-certs-ca-bundle\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.272852 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.272832 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/290c0bd1-b749-4ec9-88e1-384ac37be16f-metrics-client-ca\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.272852 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.272849 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/290c0bd1-b749-4ec9-88e1-384ac37be16f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.275042 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.275017 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.275143 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.275091 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-secret-telemeter-client\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.275208 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.275166 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-federate-client-tls\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.275264 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.275247 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-telemeter-client-tls\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.280397 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.280379 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvw4s\" (UniqueName: \"kubernetes.io/projected/290c0bd1-b749-4ec9-88e1-384ac37be16f-kube-api-access-gvw4s\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.435406 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.435374 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:37.444863 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.444825 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:37.582882 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.582788 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:37.716252 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:37.716223 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-fd64789dc-hcqt6"] Nov 02 17:33:37.720483 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:33:37.720455 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290c0bd1_b749_4ec9_88e1_384ac37be16f.slice/crio-5cc3d45cd04b69a4845d7f5ea07058de3bf85d842ebdefe66c247a3b149fc9a2 WatchSource:0}: Error finding container 5cc3d45cd04b69a4845d7f5ea07058de3bf85d842ebdefe66c247a3b149fc9a2: Status 404 returned error can't find the container with id 5cc3d45cd04b69a4845d7f5ea07058de3bf85d842ebdefe66c247a3b149fc9a2 Nov 02 17:33:38.211214 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:38.211185 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:38.250697 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:38.250614 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:38.258990 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:38.258366 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" event={"ID":"290c0bd1-b749-4ec9-88e1-384ac37be16f","Type":"ContainerStarted","Data":"5cc3d45cd04b69a4845d7f5ea07058de3bf85d842ebdefe66c247a3b149fc9a2"} Nov 02 17:33:38.263115 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:38.263085 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b083135-c135-437c-9446-5267ab9cb71f","Type":"ContainerStarted","Data":"a48c41142da65c840ebace4f02e42440fbc361d6df950be23f4b4220dcc5890f"} Nov 02 17:33:38.291690 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:38.291642 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.91315729 podStartE2EDuration="6.291625216s" totalImagesPullingTime="3.36870187s" totalInitContainerRuntime="9.766056ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:38.291386026 +0000 UTC m=+60.858618632" watchObservedRunningTime="2026-11-02 17:33:38.291625216 +0000 UTC m=+60.858857821" Nov 02 17:33:38.550552 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:38.550470 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:38.679594 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:38.679557 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:39.072327 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:39.072292 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:39.072493 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:39.072362 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:39.077443 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:39.077315 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:39.269781 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:39.269752 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:39.675369 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:39.675279 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:39.821329 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:39.821289 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:39.960059 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:39.960031 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:40.270378 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.270295 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" event={"ID":"290c0bd1-b749-4ec9-88e1-384ac37be16f","Type":"ContainerStarted","Data":"b77110010b5d9c4244e355eed4ba74b95bdc3bf0673d590257439e9e074cd13c"} Nov 02 17:33:40.270378 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.270355 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" event={"ID":"290c0bd1-b749-4ec9-88e1-384ac37be16f","Type":"ContainerStarted","Data":"9d69840bae17ac3972ebd83ccaa7be66f8cda3c40c928698e4894a77c31bd9c2"} Nov 02 17:33:40.270378 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.270375 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" event={"ID":"290c0bd1-b749-4ec9-88e1-384ac37be16f","Type":"ContainerStarted","Data":"bbd5555a14bcbf9b18ce4d7ceb5a45d18fda44926be207c1a3c591beef96b2bf"} Nov 02 17:33:40.294106 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.294056 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" podStartSLOduration=1.727782151 podStartE2EDuration="3.294041709s" totalImagesPullingTime="1.566259558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:40.292631458 +0000 UTC m=+62.859864063" watchObservedRunningTime="2026-11-02 17:33:40.294041709 +0000 UTC m=+62.861274315" Nov 02 17:33:40.362171 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.362132 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-558b6fbcdf-n4st4"] Nov 02 17:33:40.365239 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.365215 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.375777 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.375753 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-558b6fbcdf-n4st4"] Nov 02 17:33:40.402174 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.402147 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-oauth-config\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.402330 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.402182 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-console-config\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.402330 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.402210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjxms\" (UniqueName: \"kubernetes.io/projected/b69d99f5-8b53-47cf-8a76-917f606488b9-kube-api-access-pjxms\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.402330 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.402269 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-service-ca\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.402484 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.402338 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-oauth-serving-cert\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.402484 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.402375 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-serving-cert\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.402484 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.402464 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b69d99f5-8b53-47cf-8a76-917f606488b9-tmp\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.402601 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.402495 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-trusted-ca-bundle\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.503393 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.503355 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-console-config\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.503393 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.503399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjxms\" (UniqueName: \"kubernetes.io/projected/b69d99f5-8b53-47cf-8a76-917f606488b9-kube-api-access-pjxms\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.503638 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.503418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-service-ca\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.503638 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.503436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-oauth-serving-cert\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.503638 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.503452 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-serving-cert\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.503638 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.503496 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b69d99f5-8b53-47cf-8a76-917f606488b9-tmp\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.503638 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.503516 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-trusted-ca-bundle\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.503850 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.503686 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-oauth-config\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.504087 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.504060 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b69d99f5-8b53-47cf-8a76-917f606488b9-tmp\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.504309 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.504276 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-oauth-serving-cert\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.504309 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.504291 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-service-ca\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.504418 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.504285 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-console-config\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.504550 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.504523 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-trusted-ca-bundle\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.506103 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.506078 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-serving-cert\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.506204 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.506186 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-oauth-config\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.511048 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.511025 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjxms\" (UniqueName: \"kubernetes.io/projected/b69d99f5-8b53-47cf-8a76-917f606488b9-kube-api-access-pjxms\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.677522 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.677431 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:40.952742 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:40.952707 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-558b6fbcdf-n4st4"] Nov 02 17:33:40.956065 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:33:40.956035 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69d99f5_8b53_47cf_8a76_917f606488b9.slice/crio-2ce00f5670141c9170eeeaf58029aaa782c2afbb5545da08b06be9b391dbb4e7 WatchSource:0}: Error finding container 2ce00f5670141c9170eeeaf58029aaa782c2afbb5545da08b06be9b391dbb4e7: Status 404 returned error can't find the container with id 2ce00f5670141c9170eeeaf58029aaa782c2afbb5545da08b06be9b391dbb4e7 Nov 02 17:33:41.275121 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:41.275038 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-558b6fbcdf-n4st4" event={"ID":"b69d99f5-8b53-47cf-8a76-917f606488b9","Type":"ContainerStarted","Data":"5bb18eb75a166b592fceed1afd3b2495f1761ce94e331d60827b2098f94eeecf"} Nov 02 17:33:41.275121 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:41.275087 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-558b6fbcdf-n4st4" event={"ID":"b69d99f5-8b53-47cf-8a76-917f606488b9","Type":"ContainerStarted","Data":"2ce00f5670141c9170eeeaf58029aaa782c2afbb5545da08b06be9b391dbb4e7"} Nov 02 17:33:41.293557 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:41.293508 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-558b6fbcdf-n4st4" podStartSLOduration=1.2934944640000001 podStartE2EDuration="1.293494464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:41.292599145 +0000 UTC m=+63.859831751" watchObservedRunningTime="2026-11-02 17:33:41.293494464 +0000 UTC m=+63.860727147" Nov 02 17:33:41.457297 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:41.457259 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:41.599591 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:41.599507 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:42.204548 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:42.204510 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:42.345347 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:42.345312 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:42.621153 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:42.621071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:42.621153 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:42.621129 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs\") pod \"network-metrics-daemon-zcdw6\" (UID: \"a73fde2c-c65f-4374-adef-ee60a09c0b4a\") " pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:42.624157 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:42.624137 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 17:33:42.625063 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:42.625048 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 17:33:42.634334 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:42.634311 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:42.634447 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:42.634427 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs\") pod \"network-metrics-daemon-zcdw6\" (UID: \"a73fde2c-c65f-4374-adef-ee60a09c0b4a\") " pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:42.710078 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:42.710045 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jmxfr\"" Nov 02 17:33:42.718242 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:42.718223 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:42.805589 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:42.805555 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dcw65\"" Nov 02 17:33:42.813581 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:42.813561 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:42.822866 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:42.822842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tszcx\" (UniqueName: \"kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx\") pod \"network-check-target-lnbcn\" (UID: \"558e70e5-ac1c-4205-af45-7ef40a464aed\") " pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:42.825485 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:42.825466 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 17:33:42.835912 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:42.835888 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 17:33:42.846538 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:42.846514 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tszcx\" (UniqueName: \"kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx\") pod \"network-check-target-lnbcn\" (UID: \"558e70e5-ac1c-4205-af45-7ef40a464aed\") " pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:42.993932 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:42.993895 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zcdw6"] Nov 02 17:33:42.997520 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:33:42.997492 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73fde2c_c65f_4374_adef_ee60a09c0b4a.slice/crio-83c43619022e0e860a985f870be3b4af3fd8192dd3c00b5f5e92cedcfb143cc1 WatchSource:0}: Error finding container 83c43619022e0e860a985f870be3b4af3fd8192dd3c00b5f5e92cedcfb143cc1: Status 404 returned error can't find the container with id 83c43619022e0e860a985f870be3b4af3fd8192dd3c00b5f5e92cedcfb143cc1 Nov 02 17:33:43.004743 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.004724 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-52q82\"" Nov 02 17:33:43.012637 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.012611 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:43.084781 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.084750 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-drvjl"] Nov 02 17:33:43.093986 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:33:43.093930 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62013dcf_af15_4041_bdc2_f5437c000082.slice/crio-d742c4e4b17cbc9c9527de69d2c5c7bdffddc869ba92a947a1ba804e84f8733b WatchSource:0}: Error finding container d742c4e4b17cbc9c9527de69d2c5c7bdffddc869ba92a947a1ba804e84f8733b: Status 404 returned error can't find the container with id d742c4e4b17cbc9c9527de69d2c5c7bdffddc869ba92a947a1ba804e84f8733b Nov 02 17:33:43.168734 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.168699 2578 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-558b6fbcdf-n4st4"] Nov 02 17:33:43.195605 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.195570 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7979dc65db-56v4t"] Nov 02 17:33:43.201144 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.201115 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.211858 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.211834 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7979dc65db-56v4t"] Nov 02 17:33:43.226919 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.226897 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-oauth-serving-cert\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.227047 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.226933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k229c\" (UniqueName: \"kubernetes.io/projected/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-kube-api-access-k229c\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.227047 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.226976 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-serving-cert\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.227120 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.227092 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-trusted-ca-bundle\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.227149 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.227117 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-tmp\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.227223 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.227199 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-service-ca\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.227265 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.227241 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-config\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.227334 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.227317 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-oauth-config\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.285428 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.285332 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zcdw6" event={"ID":"a73fde2c-c65f-4374-adef-ee60a09c0b4a","Type":"ContainerStarted","Data":"83c43619022e0e860a985f870be3b4af3fd8192dd3c00b5f5e92cedcfb143cc1"} Nov 02 17:33:43.286637 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.286608 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-drvjl" event={"ID":"62013dcf-af15-4041-bdc2-f5437c000082","Type":"ContainerStarted","Data":"d742c4e4b17cbc9c9527de69d2c5c7bdffddc869ba92a947a1ba804e84f8733b"} Nov 02 17:33:43.286771 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.286705 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lnbcn"] Nov 02 17:33:43.291146 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:33:43.291053 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558e70e5_ac1c_4205_af45_7ef40a464aed.slice/crio-18dccc9f1e70001661daa816359127248b0dd741e83971552f35aa25039d4056 WatchSource:0}: Error finding container 18dccc9f1e70001661daa816359127248b0dd741e83971552f35aa25039d4056: Status 404 returned error can't find the container with id 18dccc9f1e70001661daa816359127248b0dd741e83971552f35aa25039d4056 Nov 02 17:33:43.328280 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.328246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-service-ca\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.328422 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.328302 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-config\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.328422 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.328330 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-oauth-config\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.328422 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.328375 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-oauth-serving-cert\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.328422 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.328411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k229c\" (UniqueName: \"kubernetes.io/projected/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-kube-api-access-k229c\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.328681 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.328451 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-serving-cert\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.328681 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.328483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-trusted-ca-bundle\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.328681 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.328516 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-tmp\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.328895 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.328872 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-tmp\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.329139 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.329120 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-service-ca\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.329241 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.329184 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-config\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.329478 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.329454 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-trusted-ca-bundle\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.329589 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.329489 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-oauth-serving-cert\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.331314 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.331291 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-serving-cert\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.331393 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.331298 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-oauth-config\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.337121 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.337097 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k229c\" (UniqueName: \"kubernetes.io/projected/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-kube-api-access-k229c\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.515149 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.514908 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:43.794306 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:43.794266 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7979dc65db-56v4t"] Nov 02 17:33:43.803121 ip-10-0-142-133 kubenswrapper[2578]: W1102 17:33:43.799047 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40f5b0d_64b1_4e58_afec_4dc4b19363d1.slice/crio-f86bb36cade1759273de3d44e951ef8f897a140526e207664028c81fa9128f21 WatchSource:0}: Error finding container f86bb36cade1759273de3d44e951ef8f897a140526e207664028c81fa9128f21: Status 404 returned error can't find the container with id f86bb36cade1759273de3d44e951ef8f897a140526e207664028c81fa9128f21 Nov 02 17:33:44.085309 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:44.085268 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:44.295174 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:44.295137 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zcdw6" event={"ID":"a73fde2c-c65f-4374-adef-ee60a09c0b4a","Type":"ContainerStarted","Data":"3dc96ae7bd347cd4c96aa9eb0a79d8c92d008c5bbf20bd5ce3610e8416fe1f2c"} Nov 02 17:33:44.295320 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:44.295198 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zcdw6" event={"ID":"a73fde2c-c65f-4374-adef-ee60a09c0b4a","Type":"ContainerStarted","Data":"d709832162919875fb3e20db1ab5f864d6f3543f30f106c75f19cfcbdc3fb1b0"} Nov 02 17:33:44.302086 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:44.301773 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lnbcn" event={"ID":"558e70e5-ac1c-4205-af45-7ef40a464aed","Type":"ContainerStarted","Data":"18dccc9f1e70001661daa816359127248b0dd741e83971552f35aa25039d4056"} Nov 02 17:33:44.304531 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:44.304505 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7979dc65db-56v4t" event={"ID":"c40f5b0d-64b1-4e58-afec-4dc4b19363d1","Type":"ContainerStarted","Data":"71381505bc2c86e03f375a654de061a5de203d0d9af8823982362c1d7a3e01b1"} Nov 02 17:33:44.304628 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:44.304552 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7979dc65db-56v4t" event={"ID":"c40f5b0d-64b1-4e58-afec-4dc4b19363d1","Type":"ContainerStarted","Data":"f86bb36cade1759273de3d44e951ef8f897a140526e207664028c81fa9128f21"} Nov 02 17:33:44.314662 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:44.314626 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zcdw6" podStartSLOduration=65.321962358 podStartE2EDuration="1m6.314611461s" totalImagesPullingTime="992.649103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:44.31305529 +0000 UTC m=+66.880287897" watchObservedRunningTime="2026-11-02 17:33:44.314611461 +0000 UTC m=+66.881844086" Nov 02 17:33:44.332189 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:44.331517 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7979dc65db-56v4t" podStartSLOduration=1.3315007589999999 podStartE2EDuration="1.331500759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:44.330668717 +0000 UTC m=+66.897901322" watchObservedRunningTime="2026-11-02 17:33:44.331500759 +0000 UTC m=+66.898733370" Nov 02 17:33:45.873508 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:45.873468 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:46.038246 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:46.038206 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:47.320601 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:47.320557 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lnbcn" event={"ID":"558e70e5-ac1c-4205-af45-7ef40a464aed","Type":"ContainerStarted","Data":"023773645dbc8470a025a97f50f5e38cac11219053294221092ddf084e78de87"} Nov 02 17:33:47.321051 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:47.320676 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:47.340447 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:47.340386 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lnbcn" podStartSLOduration=65.813767913 podStartE2EDuration="1m9.340367541s" totalImagesPullingTime="3.526599628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:47.33956672 +0000 UTC m=+69.906799348" watchObservedRunningTime="2026-11-02 17:33:47.340367541 +0000 UTC m=+69.907600147" Nov 02 17:33:47.439429 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:47.439391 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:47.589975 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:47.589871 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:47.976153 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:47.976118 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:48.114381 ip-10-0-142-133 kubenswrapper[2578]: I1102 17:33:48.114336 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:48.147000 ip-10-0-142-133 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 17:33:48.164403 ip-10-0-142-133 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 17:33:48.164568 ip-10-0-142-133 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 17:33:48.164617 ip-10-0-142-133 systemd[1]: kubelet.service: Consumed 4.129s CPU time. Nov 02 17:33:48.179613 ip-10-0-142-133 systemd[1]: Starting Kubernetes Kubelet... Nov 02 17:33:48.238560 ip-10-0-142-133 kubenswrapper[5549]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:33:48.238560 ip-10-0-142-133 kubenswrapper[5549]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 17:33:48.238560 ip-10-0-142-133 kubenswrapper[5549]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:33:48.238560 ip-10-0-142-133 kubenswrapper[5549]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:33:48.240570 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240491 5549 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:33:48.240570 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240571 5549 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240575 5549 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240579 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240582 5549 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240584 5549 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240588 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240591 5549 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240594 5549 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240597 5549 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240599 5549 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240602 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240605 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240607 5549 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240610 5549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240613 5549 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240615 5549 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240618 5549 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240620 5549 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240623 5549 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:33:48.240637 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240629 5549 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240633 5549 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240638 5549 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240641 5549 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240644 5549 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240647 5549 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240650 5549 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240652 5549 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240655 5549 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240658 5549 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240660 5549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240663 5549 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240666 5549 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240668 5549 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240671 5549 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240674 5549 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240676 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240679 5549 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240682 5549 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240684 5549 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:33:48.241114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240687 5549 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240690 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240692 5549 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240694 5549 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240697 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240700 5549 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240703 5549 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240706 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240709 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240712 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240714 5549 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240718 5549 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240723 5549 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240726 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240729 5549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240732 5549 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240734 5549 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240737 5549 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240740 5549 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:33:48.241582 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240742 5549 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240745 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240747 5549 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240750 5549 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240752 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240755 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240758 5549 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240761 5549 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240765 5549 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240768 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240770 5549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240773 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240776 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240779 5549 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240782 5549 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240784 5549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240787 5549 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240790 5549 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240793 5549 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:33:48.242054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240795 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240798 5549 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240801 5549 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240804 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240807 5549 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240809 5549 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240812 5549 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240815 5549 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240818 5549 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240820 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240823 5549 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240825 5549 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240828 5549 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240831 5549 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240833 5549 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240836 5549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240839 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240842 5549 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240845 5549 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:33:48.242505 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240848 5549 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240850 5549 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240853 5549 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240856 5549 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240858 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240861 5549 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240864 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240867 5549 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.240870 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241637 5549 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241645 5549 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241648 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241652 5549 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241655 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241659 5549 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241662 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241665 5549 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241667 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241671 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241674 5549 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:33:48.242971 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241677 5549 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241679 5549 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241682 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241685 5549 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241687 5549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241690 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241692 5549 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241695 5549 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241698 5549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241702 5549 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241706 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241709 5549 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241712 5549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241716 5549 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241719 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241722 5549 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241725 5549 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241728 5549 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241731 5549 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:33:48.243450 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241734 5549 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241737 5549 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241740 5549 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241743 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241746 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241749 5549 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241751 5549 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241754 5549 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241756 5549 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241759 5549 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241762 5549 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241765 5549 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241767 5549 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241770 5549 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241773 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241775 5549 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241778 5549 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241783 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241786 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:33:48.243900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241788 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241791 5549 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241794 5549 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241797 5549 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241799 5549 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241802 5549 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241805 5549 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241808 5549 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241811 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241813 5549 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241816 5549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241819 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241821 5549 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241824 5549 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241826 5549 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241829 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241833 5549 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241836 5549 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241838 5549 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:33:48.244379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241841 5549 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241844 5549 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241846 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241849 5549 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241851 5549 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241854 5549 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241856 5549 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241859 5549 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241861 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241864 5549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241866 5549 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241871 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241874 5549 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241877 5549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241879 5549 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241882 5549 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241885 5549 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241888 5549 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241890 5549 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241893 5549 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:33:48.244836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241895 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241898 5549 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241901 5549 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241903 5549 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241906 5549 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241908 5549 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241911 5549 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241913 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241916 5549 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241919 5549 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241922 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241925 5549 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241927 5549 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241930 5549 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241932 5549 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241935 5549 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241937 5549 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.241940 5549 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242409 5549 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242414 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:33:48.245432 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242417 5549 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242420 5549 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242423 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242426 5549 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242429 5549 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242431 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242434 5549 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242437 5549 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242439 5549 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242442 5549 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242444 5549 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242447 5549 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242450 5549 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242452 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242455 5549 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242457 5549 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242460 5549 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242462 5549 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242468 5549 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242470 5549 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:33:48.245911 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242473 5549 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242476 5549 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242479 5549 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242481 5549 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242484 5549 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242487 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242489 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242492 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242494 5549 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242498 5549 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242503 5549 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242506 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242509 5549 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242512 5549 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242515 5549 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242518 5549 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242520 5549 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242523 5549 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242526 5549 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:33:48.246444 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242528 5549 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242531 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242534 5549 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242538 5549 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242542 5549 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242546 5549 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242548 5549 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242551 5549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242554 5549 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242557 5549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242559 5549 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242562 5549 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242565 5549 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242567 5549 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242570 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242572 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242575 5549 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242578 5549 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242581 5549 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242583 5549 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:33:48.246913 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242586 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242589 5549 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242591 5549 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242594 5549 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242597 5549 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242599 5549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242602 5549 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242605 5549 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242607 5549 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242610 5549 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242613 5549 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242616 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242618 5549 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242621 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242624 5549 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242627 5549 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242630 5549 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242632 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242635 5549 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242637 5549 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:33:48.247415 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242640 5549 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242642 5549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242645 5549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242647 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242650 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242653 5549 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242656 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242658 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242661 5549 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242664 5549 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242667 5549 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242670 5549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242673 5549 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242675 5549 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242678 5549 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242680 5549 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242683 5549 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242685 5549 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242688 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:33:48.247898 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242691 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242693 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242696 5549 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242698 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242701 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.242703 5549 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242808 5549 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242819 5549 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242828 5549 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242834 5549 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242846 5549 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242849 5549 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242856 5549 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242861 5549 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242864 5549 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242867 5549 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242871 5549 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242874 5549 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242877 5549 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242880 5549 flags.go:64] FLAG: --cgroup-root="" Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242883 5549 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242887 5549 flags.go:64] FLAG: --client-ca-file="" Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242890 5549 flags.go:64] FLAG: --cloud-provider="external" Nov 02 17:33:48.248368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242893 5549 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242898 5549 flags.go:64] FLAG: --cluster-domain="" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242901 5549 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242904 5549 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242907 5549 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242911 5549 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242915 5549 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242918 5549 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242921 5549 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242925 5549 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242928 5549 flags.go:64] FLAG: --contention-profiling="false" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242931 5549 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242934 5549 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242937 5549 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242940 5549 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242944 5549 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242947 5549 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242964 5549 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242967 5549 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242970 5549 flags.go:64] FLAG: --enable-server="true" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242973 5549 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242979 5549 flags.go:64] FLAG: --event-burst="100" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242984 5549 flags.go:64] FLAG: --event-qps="50" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242987 5549 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 17:33:48.248919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242991 5549 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242994 5549 flags.go:64] FLAG: --eviction-hard="" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.242998 5549 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243001 5549 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243004 5549 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243007 5549 flags.go:64] FLAG: --eviction-soft="" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243011 5549 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243014 5549 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243017 5549 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243020 5549 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243023 5549 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243026 5549 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243029 5549 flags.go:64] FLAG: --feature-gates="" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243038 5549 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243041 5549 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243045 5549 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243048 5549 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243052 5549 flags.go:64] FLAG: --healthz-port="10248" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243055 5549 flags.go:64] FLAG: --help="false" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243058 5549 flags.go:64] FLAG: --hostname-override="ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243062 5549 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243065 5549 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243068 5549 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 17:33:48.249499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243089 5549 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243094 5549 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243097 5549 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243100 5549 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243104 5549 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243107 5549 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243112 5549 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243117 5549 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243120 5549 flags.go:64] FLAG: --kube-reserved="" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243123 5549 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243126 5549 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243129 5549 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243132 5549 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243135 5549 flags.go:64] FLAG: --lock-file="" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243138 5549 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243141 5549 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243144 5549 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243150 5549 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243153 5549 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243157 5549 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243160 5549 flags.go:64] FLAG: --logging-format="text" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243163 5549 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243167 5549 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243170 5549 flags.go:64] FLAG: --manifest-url="" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243173 5549 flags.go:64] FLAG: --manifest-url-header="" Nov 02 17:33:48.250058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243177 5549 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243180 5549 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243185 5549 flags.go:64] FLAG: --max-pods="110" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243188 5549 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243191 5549 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243194 5549 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243197 5549 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243200 5549 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243203 5549 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243206 5549 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243213 5549 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243216 5549 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243219 5549 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243224 5549 flags.go:64] FLAG: --pod-cidr="" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243228 5549 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243231 5549 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243234 5549 flags.go:64] FLAG: --pods-per-core="0" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243237 5549 flags.go:64] FLAG: --port="10250" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243240 5549 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243244 5549 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0589b06646a566b08" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243247 5549 flags.go:64] FLAG: --qos-reserved="" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243250 5549 flags.go:64] FLAG: --read-only-port="10255" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243253 5549 flags.go:64] FLAG: --register-node="true" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243256 5549 flags.go:64] FLAG: --register-with-taints="" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243260 5549 flags.go:64] FLAG: --registry-burst="10" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243262 5549 flags.go:64] FLAG: --registry-qps="5" Nov 02 17:33:48.250695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243265 5549 flags.go:64] FLAG: --reserved-cpus="" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243270 5549 flags.go:64] FLAG: --reserved-memory="" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243274 5549 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243277 5549 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243280 5549 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243283 5549 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243286 5549 flags.go:64] FLAG: --runonce="false" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243289 5549 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243292 5549 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243296 5549 flags.go:64] FLAG: --seccomp-default="false" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243298 5549 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243301 5549 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243304 5549 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243308 5549 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243311 5549 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243313 5549 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243316 5549 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243319 5549 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243322 5549 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243326 5549 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243331 5549 flags.go:64] FLAG: --system-cgroups="" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243335 5549 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243340 5549 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243343 5549 flags.go:64] FLAG: --tls-cert-file="" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243345 5549 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 17:33:48.251707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243350 5549 flags.go:64] FLAG: --tls-min-version="" Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243353 5549 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243356 5549 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243359 5549 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243362 5549 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243365 5549 flags.go:64] FLAG: --v="2" Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243370 5549 flags.go:64] FLAG: --version="false" Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243373 5549 flags.go:64] FLAG: --vmodule="" Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243377 5549 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243381 5549 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243513 5549 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243517 5549 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243521 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243523 5549 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243526 5549 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243530 5549 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243534 5549 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243538 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243542 5549 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243544 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243547 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:33:48.252390 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243550 5549 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243553 5549 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243556 5549 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243558 5549 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243561 5549 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243565 5549 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243569 5549 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243572 5549 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243574 5549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243577 5549 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243580 5549 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243582 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243585 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243588 5549 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243590 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243593 5549 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243595 5549 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243598 5549 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243600 5549 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243603 5549 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:33:48.252902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243605 5549 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243608 5549 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243612 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243614 5549 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243617 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243620 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243623 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243625 5549 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243627 5549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243630 5549 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243633 5549 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243636 5549 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243638 5549 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243641 5549 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243643 5549 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243646 5549 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243649 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243652 5549 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243657 5549 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:33:48.253405 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243659 5549 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243662 5549 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243665 5549 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243668 5549 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243670 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243673 5549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243675 5549 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243678 5549 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243680 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243683 5549 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243686 5549 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243689 5549 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243691 5549 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243694 5549 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243696 5549 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243706 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243708 5549 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243711 5549 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243713 5549 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243716 5549 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:33:48.254044 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243718 5549 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243721 5549 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243724 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243726 5549 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243729 5549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243731 5549 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243736 5549 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243739 5549 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243743 5549 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243746 5549 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243750 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243755 5549 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243757 5549 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243760 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243763 5549 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243766 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243768 5549 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243771 5549 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243774 5549 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:33:48.254596 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243776 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:33:48.255161 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243779 5549 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:33:48.255161 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243781 5549 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:33:48.255161 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243784 5549 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:33:48.255161 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243787 5549 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:33:48.255161 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243790 5549 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:33:48.255161 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243792 5549 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:33:48.255161 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243795 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:33:48.255161 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243797 5549 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:33:48.255161 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243800 5549 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:33:48.255161 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243803 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:33:48.255161 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243806 5549 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:33:48.255161 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243808 5549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:33:48.255161 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243811 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:33:48.255161 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243813 5549 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:33:48.255161 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243816 5549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:33:48.255161 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243819 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:33:48.255573 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.243846 5549 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:33:48.255573 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243885 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:33:48.255573 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243890 5549 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:33:48.255573 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243893 5549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:33:48.255573 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243896 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:33:48.255573 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243900 5549 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:33:48.255573 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243903 5549 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:33:48.255573 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243908 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:33:48.255573 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243911 5549 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:33:48.255573 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243914 5549 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:33:48.255573 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243918 5549 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:33:48.255573 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243921 5549 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:33:48.255573 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243924 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:33:48.255573 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243927 5549 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:33:48.255573 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243930 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243932 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243935 5549 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243938 5549 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243940 5549 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243943 5549 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243946 5549 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243960 5549 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243963 5549 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243966 5549 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243968 5549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243971 5549 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243974 5549 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243977 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243980 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243982 5549 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243985 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243988 5549 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243990 5549 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243993 5549 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:33:48.256010 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243996 5549 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.243998 5549 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244001 5549 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244003 5549 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244006 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244013 5549 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244016 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244018 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244021 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244024 5549 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244026 5549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244029 5549 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244031 5549 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244034 5549 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244037 5549 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244040 5549 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244042 5549 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244045 5549 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244047 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:33:48.256579 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244050 5549 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244054 5549 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244057 5549 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244059 5549 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244062 5549 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244065 5549 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244068 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244071 5549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244074 5549 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244076 5549 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244079 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244081 5549 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244084 5549 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244086 5549 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244089 5549 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244091 5549 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244094 5549 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244097 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244102 5549 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244104 5549 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:33:48.257106 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244107 5549 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244110 5549 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244112 5549 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244115 5549 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244118 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244120 5549 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244123 5549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244125 5549 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244130 5549 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244133 5549 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244136 5549 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244139 5549 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244141 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244144 5549 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244147 5549 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244149 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244152 5549 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244154 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244158 5549 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:33:48.257662 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244161 5549 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244164 5549 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244166 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244169 5549 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244172 5549 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244174 5549 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244177 5549 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244180 5549 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244182 5549 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244185 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244187 5549 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244192 5549 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244195 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244198 5549 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.244200 5549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.247983 5549 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.247999 5549 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.248016 5549 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.248021 5549 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248104 5549 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:33:48.258222 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248109 5549 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248113 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248118 5549 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248122 5549 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248127 5549 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248130 5549 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248133 5549 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248135 5549 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248138 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248141 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248144 5549 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248152 5549 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248156 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248158 5549 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248161 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248164 5549 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248167 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248170 5549 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248173 5549 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:33:48.258703 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248176 5549 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248179 5549 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248182 5549 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248184 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248187 5549 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248190 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248193 5549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248195 5549 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248198 5549 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248202 5549 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248205 5549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248207 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248210 5549 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248213 5549 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248216 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248218 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248221 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248223 5549 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248226 5549 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248228 5549 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:33:48.259170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248231 5549 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248234 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248236 5549 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248239 5549 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248241 5549 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248250 5549 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248253 5549 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248256 5549 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248259 5549 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248263 5549 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248266 5549 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248269 5549 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248272 5549 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248275 5549 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248277 5549 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248280 5549 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248282 5549 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248285 5549 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248288 5549 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:33:48.259653 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248290 5549 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248293 5549 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248295 5549 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248298 5549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248301 5549 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248303 5549 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248306 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248309 5549 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248311 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248314 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248316 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248319 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248322 5549 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248324 5549 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248327 5549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248329 5549 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248332 5549 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248334 5549 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248337 5549 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:33:48.260226 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248345 5549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248348 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248351 5549 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248354 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248357 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248360 5549 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248363 5549 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248365 5549 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248368 5549 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248370 5549 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248373 5549 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248376 5549 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248379 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248382 5549 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248385 5549 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248387 5549 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248390 5549 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248393 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248395 5549 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248398 5549 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:33:48.260676 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248400 5549 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:33:48.261176 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248403 5549 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:33:48.261176 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248405 5549 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:33:48.261176 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248408 5549 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:33:48.261176 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248410 5549 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:33:48.261176 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248413 5549 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:33:48.261176 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248415 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:33:48.261176 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248418 5549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:33:48.261176 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.248444 5549 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:33:48.261176 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248634 5549 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:33:48.261176 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248640 5549 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:33:48.261176 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248643 5549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:33:48.261176 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248645 5549 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:33:48.261176 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248654 5549 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:33:48.261176 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248658 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:33:48.261176 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248661 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:33:48.261176 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248663 5549 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248666 5549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248669 5549 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248672 5549 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248676 5549 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248679 5549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248682 5549 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248685 5549 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248688 5549 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248691 5549 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248693 5549 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248696 5549 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248698 5549 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248701 5549 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248704 5549 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248706 5549 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248709 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248711 5549 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248714 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248717 5549 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:33:48.261564 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248719 5549 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248721 5549 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248724 5549 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248726 5549 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248729 5549 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248732 5549 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248735 5549 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248737 5549 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248740 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248742 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248751 5549 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248754 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248757 5549 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248760 5549 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248762 5549 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248765 5549 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248767 5549 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248770 5549 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248772 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:33:48.262067 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248775 5549 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248777 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248780 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248782 5549 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248785 5549 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248789 5549 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248792 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248795 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248798 5549 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248800 5549 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248803 5549 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248806 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248808 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248811 5549 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248814 5549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248816 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248819 5549 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248821 5549 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248824 5549 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:33:48.262619 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248826 5549 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248829 5549 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248831 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248834 5549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248837 5549 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248845 5549 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248848 5549 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248851 5549 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248854 5549 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248856 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248859 5549 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248861 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248864 5549 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248867 5549 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248869 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248872 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248874 5549 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248881 5549 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248884 5549 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:33:48.263111 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248887 5549 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248889 5549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248892 5549 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248894 5549 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248897 5549 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248900 5549 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248902 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248905 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248908 5549 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248910 5549 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248913 5549 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248916 5549 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248919 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248921 5549 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248924 5549 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248927 5549 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248930 5549 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248932 5549 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248935 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248963 5549 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:33:48.263566 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248967 5549 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:33:48.264244 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.248970 5549 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:33:48.264244 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.248994 5549 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:33:48.264244 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249043 5549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:33:48.264244 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249047 5549 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:33:48.264244 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249050 5549 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:33:48.264244 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249054 5549 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:33:48.264244 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249058 5549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:33:48.264244 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249061 5549 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:33:48.264244 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249063 5549 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:33:48.264244 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249066 5549 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:33:48.264244 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249071 5549 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:33:48.264244 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249074 5549 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:33:48.264244 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249077 5549 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:33:48.264244 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249079 5549 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:33:48.264244 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249082 5549 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:33:48.264244 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249085 5549 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249088 5549 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249090 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249093 5549 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249095 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249098 5549 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249101 5549 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249103 5549 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249106 5549 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249109 5549 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249111 5549 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249114 5549 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249117 5549 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249119 5549 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249122 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249125 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249133 5549 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249136 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249139 5549 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249141 5549 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:33:48.264628 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249144 5549 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249147 5549 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249149 5549 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249152 5549 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249154 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249157 5549 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249159 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249164 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249166 5549 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249169 5549 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249173 5549 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249177 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249181 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249184 5549 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249186 5549 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249190 5549 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249193 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249195 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249198 5549 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:33:48.265116 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249200 5549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249203 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249205 5549 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249208 5549 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249211 5549 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249213 5549 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249216 5549 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249219 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249221 5549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249224 5549 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249232 5549 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249235 5549 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249237 5549 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249240 5549 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249242 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249245 5549 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249247 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249250 5549 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249253 5549 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:33:48.265616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249255 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249259 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249262 5549 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249265 5549 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249268 5549 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249270 5549 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249273 5549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249276 5549 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249281 5549 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249284 5549 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249287 5549 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249289 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249292 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249295 5549 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249298 5549 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249300 5549 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249303 5549 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249306 5549 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249308 5549 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249311 5549 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:33:48.266082 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249313 5549 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249316 5549 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249318 5549 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249321 5549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249329 5549 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249333 5549 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249335 5549 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249338 5549 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249340 5549 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249343 5549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249345 5549 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249348 5549 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249350 5549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249354 5549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:33:48.249357 5549 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.249404 5549 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.250400 5549 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.250461 5549 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.251016 5549 server.go:1014] "Starting client certificate rotation" Nov 02 17:33:48.266556 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.251987 5549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:33:48.267024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.252164 5549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:27:37 +0000 UTC" deadline="2028-05-08 10:13:56.685905116 +0000 UTC" Nov 02 17:33:48.267024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.252226 5549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13264h40m8.433681442s" Nov 02 17:33:48.267024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.252451 5549 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:33:48.267024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.253494 5549 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:33:48.267024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.254785 5549 log.go:25] "Validated CRI v1 runtime API" Nov 02 17:33:48.267024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.256547 5549 log.go:25] "Validated CRI v1 image API" Nov 02 17:33:48.267024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.257267 5549 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 17:33:48.267024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.259333 5549 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c861d13b-7785-48cd-be99-049f18fba5b5:/dev/nvme0n1p4 cce73ded-b912-458e-b0ac-911a50df920c:/dev/nvme0n1p3] Nov 02 17:33:48.267246 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.259358 5549 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b4290a8c2bddeff8a97107329477306e785e45f2dcafea75b5829d40eeed550/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b4290a8c2bddeff8a97107329477306e785e45f2dcafea75b5829d40eeed550/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/117152fb4e06d47da9759eeef3882fced47fe07ca2e5f9b81d4e457754570e1e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/117152fb4e06d47da9759eeef3882fced47fe07ca2e5f9b81d4e457754570e1e/userdata/shm major:0 minor:58 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18dccc9f1e70001661daa816359127248b0dd741e83971552f35aa25039d4056/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18dccc9f1e70001661daa816359127248b0dd741e83971552f35aa25039d4056/userdata/shm major:0 minor:390 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ce00f5670141c9170eeeaf58029aaa782c2afbb5545da08b06be9b391dbb4e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ce00f5670141c9170eeeaf58029aaa782c2afbb5545da08b06be9b391dbb4e7/userdata/shm major:0 minor:371 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/328a3d179ba928d8ce082f2f6fd43397b4c113843153f6d55913dd89af0d158b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/328a3d179ba928d8ce082f2f6fd43397b4c113843153f6d55913dd89af0d158b/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33140d4e6a3933461db90cac160879545d0b84614a36600d3b5dffbf02127232/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33140d4e6a3933461db90cac160879545d0b84614a36600d3b5dffbf02127232/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33cc3c3636522c210a2c55e06391e20ffab4fd130b5a0973557180aa5e1650fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33cc3c3636522c210a2c55e06391e20ffab4fd130b5a0973557180aa5e1650fb/userdata/shm major:0 minor:300 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/467b0f4d69f4df830fb36f0cbb9bd27f32d920c9ecc16042114bf4434a1f1987/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/467b0f4d69f4df830fb36f0cbb9bd27f32d920c9ecc16042114bf4434a1f1987/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b005971739cb10211378c9b0a94bf2285179c7c51972f4c35877f4ac3d8fa06/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b005971739cb10211378c9b0a94bf2285179c7c51972f4c35877f4ac3d8fa06/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c3c9a9c9257b3ab55d41cf4c87481c5756c0be2fa8ef59da9153ad33b2f5b3b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c3c9a9c9257b3ab55d41cf4c87481c5756c0be2fa8ef59da9153ad33b2f5b3b/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56ac149e79f43a9ad9644a11d2ac29a4f6edae69595c394ce0e3363f373237cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56ac149e79f43a9ad9644a11d2ac29a4f6edae69595c394ce0e3363f373237cc/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a5511f9cffc9fac7dd2b70f2be71a0228fca3de17c8f3f98cd11efcbcd93c3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a5511f9cffc9fac7dd2b70f2be71a0228fca3de17c8f3f98cd11efcbcd93c3e/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5cc3d45cd04b69a4845d7f5ea07058de3bf85d842ebdefe66c247a3b149fc9a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5cc3d45cd04b69a4845d7f5ea07058de3bf85d842ebdefe66c247a3b149fc9a2/userdata/shm major:0 minor:343 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/83c43619022e0e860a985f870be3b4af3fd8192dd3c00b5f5e92cedcfb143cc1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/83c43619022e0e860a985f870be3b4af3fd8192dd3c00b5f5e92cedcfb143cc1/userdata/shm major:0 minor:383 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89b301aa1389bd1fdafc0b8dabd74be65a483068163c7370cefae600058e32a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89b301aa1389bd1fdafc0b8dabd74be65a483068163c7370cefae600058e32a0/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94abd7d1ee7d72da8ddde0f1c9482b5488ef6fc7b9c41523a7facdad001dfc7c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94abd7d1ee7d72da8ddde0f1c9482b5488ef6fc7b9c41523a7facdad001dfc7c/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b56787b1149f33c8e1c2e99c8075b6d1d15514e11a0f42716ef22c26ddf92785/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b56787b1149f33c8e1c2e99c8075b6d1d15514e11a0f42716ef22c26ddf92785/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2fdba41a69a6707dad0cee32e99cc6cf3fca17653c509f5d9e05d1d05d4d95e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2fdba41a69a6707dad0cee32e99cc6cf3fca17653c509f5d9e05d1d05d4d95e/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d742c4e4b17cbc9c9527de69d2c5c7bdffddc869ba92a947a1ba804e84f8733b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d742c4e4b17cbc9c9527de69d2c5c7bdffddc869ba92a947a1ba804e84f8733b/userdata/shm major:0 minor:385 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ddb4c2750aae48a90222d31bb22af2e9ce23331dcb3818f535773f2bae20e02f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ddb4c2750aae48a90222d31bb22af2e9ce23331dcb3818f535773f2bae20e02f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f86bb36cade1759273de3d44e951ef8f897a140526e207664028c81fa9128f21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f86bb36cade1759273de3d44e951ef8f897a140526e207664028c81fa9128f21/userdata/shm major:0 minor:399 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc423511a76bc5316042893587560d7579bd92ccbe01c5cfae00d72c7c6d986d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc423511a76bc5316042893587560d7579bd92ccbe01c5cfae00d72c7c6d986d/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/feb45ffd2f207fab99820b663ea19ea9492c80779f5d187e329bf65dc5f9eef8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/feb45ffd2f207fab99820b663ea19ea9492c80779f5d187e329bf65dc5f9eef8/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volume-subpaths/web-config/alertmanager/9 major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volume-subpaths/web-config/config-reloader/6 major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~empty-dir/config-out major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~projected/kube-api-access-bhglv:{mountpoint:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~projected/kube-api-access-bhglv major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~projected/tls-assets major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/config-volume major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/web-config major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21691b74-73cc-41de-9c08-0f5e27528dba/volumes/kubernetes.io~projected/kube-api-access-n9wbw:{mountpoint:/var/lib/kubelet/pods/21691b74-73cc-41de-9c08-0f5e27528dba/volumes/kubernetes.io~projected/kube-api-access-n9wbw major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21691b74-73cc-41de-9c08-0f5e27528dba/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/21691b74-73cc-41de-9c08-0f5e27528dba/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21691b74-73cc-41de-9c08-0f5e27528dba/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/21691b74-73cc-41de-9c08-0f5e27528dba/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23e84589-7092-4074-bfc6-d5dd9ea2d4cc/volumes/kubernetes.io~projected/kube-api-access-h8kcn:{mountpoint:/var/lib/kubelet/pods/23e84589-7092-4074-bfc6-d5dd9ea2d4cc/volumes/kubernetes.io~projected/kube-api-access-h8kcn major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/290c0bd1-b749-4ec9-88e1-384ac37be16f/volumes/kubernetes.io~projected/kube-api-access-gvw4s:{mountpoint:/var/lib/kubelet/pods/290c0bd1-b749-4ec9-88e1-384ac37be16f/volumes/kubernetes.io~projected/kube-api-access-gvw4s major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/290c0bd1-b749-4ec9-88e1-384ac37be16f/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/290c0bd1-b749-4ec9-88e1-384ac37be16f/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/290c0bd1-b749-4ec9-88e1-384ac37be16f/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/290c0bd1-b749-4ec9-88e1-384ac37be16f/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/290c0bd1-b749-4ec9-88e1-384ac37be16f/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/290c0bd1-b749-4ec9-88e1-384ac37be16f/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/290c0bd1-b749-4ec9-88e1-384ac37be16f/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/290c0bd1-b749-4ec9-88e1-384ac37be16f/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b6949d0-0006-4488-89ad-296077fcf045/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2b6949d0-0006-4488-89ad-296077fcf045/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b6949d0-0006-4488-89ad-296077fcf045/volumes/kubernetes.io~projected/kube-api-access-h9w5w:{mountpoint:/var/lib/kubelet/pods/2b6949d0-0006-4488-89ad-296077fcf045/volumes/kubernetes.io~projected/kube-api-access-h9w5w major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b6949d0-0006-4488-89ad-296077fcf045/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2b6949d0-0006-4488-89ad-296077fcf045/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3df7744d-a04e-42aa-8938-31b27451b921/volumes/kubernetes.io~projected/kube-api-access-56frp:{mountpoint:/var/lib/kubelet/pods/3df7744d-a04e-42aa-8938-31b27451b921/volumes/kubernetes.io~projected/kube-api-access-56frp major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3df7744d-a04e-42aa-8938-31b27451b921/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/3df7744d-a04e-42aa-8938-31b27451b921/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/491761ca-d114-411c-9196-3bd0065ae73a/volumes/kubernetes.io~projected/kube-api-access-k7t6q:{mountpoint:/var/lib/kubelet/pods/491761ca-d114-411c-9196-3bd0065ae73a/volumes/kubernetes.io~projected/kube-api-access-k7t6q major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53eac522-fee5-4865-955b-a14da456d693/volumes/kubernetes.io~projected/kube-api-access-hqx5s:{mountpoint:/var/lib/kubelet/pods/53eac522-fee5-4865-955b-a14da456d693/volumes/kubernetes.io~projected/kube-api-access-hqx5s major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53eac522-fee5-4865-955b-a14da456d693/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/53eac522-fee5-4865-955b-a14da456d693/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53eac522-fee5-4865-955b-a14da456d693/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/53eac522-fee5-4865-955b-a14da456d693/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/558e70e5-ac1c-4205-af45-7ef40a464aed/volumes/kubernetes.io~projected/kube-api-access-tszcx:{mountpoint:/var/lib/kubelet/pods/558e70e5-ac1c-4205-af45-7ef40a464aed/volumes/kubernetes.io~projected/kube-api-access-tszcx major:0 minor:387 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62013dcf-af15-4041-bdc2-f5437c000082/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/62013dcf-af15-4041-bdc2-f5437c000082/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/636356f6-e4d8-475e-8c82-dbbeb5fee0c3/volumes/kubernetes.io~projected/kube-api-access-qlxdl:{mountpoint:/var/lib/kubelet/pods/636356f6-e4d8-475e-8c82-dbbeb5fee0c3/volumes/kubernetes.io~projected/kube-api-access-qlxdl major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67a04d53-e707-4b0b-9e32-d84c31e413e5/volumes/kubernetes.io~projected/kube-api-access-6zmv4:{mountpoint:/var/lib/kubelet/pods/67a04d53-e707-4b0b-9e32-d84c31e413e5/volumes/kubernetes.io~projected/kube-api-access-6zmv4 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67a04d53-e707-4b0b-9e32-d84c31e413e5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/67a04d53-e707-4b0b-9e32-d84c31e413e5/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a73fde2c-c65f-4374-adef-ee60a09c0b4a/volumes/kubernetes.io~projected/kube-api-access-nnhzb:{mountpoint:/var/lib/kubelet/pods/a73fde2c-c65f-4374-adef-ee60a09c0b4a/volumes/kubernetes.io~projected/kube-api-access-nnhzb major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a73fde2c-c65f-4374-adef-ee60a09c0b4a/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/a73fde2c-c65f-4374-adef-ee60a09c0b4a/volumes/kubernetes.io~secret/metrics-certs major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae6e2a1d-9467-4ef5-b493-6e788c0917b6/volumes/kubernetes.io~projected/kube-api-access-cfmgm:{mountpoint:/var/lib/kubelet/pods/ae6e2a1d-9467-4ef5-b493-6e788c0917b6/volumes/kubernetes.io~projected/kube-api-access-cfmgm major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b69d99f5-8b53-47cf-8a76-917f606488b9/volumes/kubernetes.io~projected/kube-api-access-pjxms:{mountpoint:/var/lib/kubelet/pods/b69d99f5-8b53-47cf-8a76-917f606488b9/volumes/kubernetes.io~projected/kube-api-access-pjxms major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b69d99f5-8b53-47cf-8a76-917f606488b9/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/b69d99f5-8b53-47cf-8a76-917f606488b9/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b69d99f5-8b53-47cf-8a76-917f606488b9/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/b69d99f5-8b53-47cf-8a76-917f606488b9/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/babf1523-fa08-4397-b0ff-946af7e07278/volumes/kubernetes.io~projected/kube-api-access-9mgbp:{mountpoint:/var/lib/kubelet/pods/babf1523-fa08-4397-b0ff-946af7e07278/volumes/kubernetes.io~projected/kube-api-access-9mgbp major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/babf1523-fa08-4397-b0ff-946af7e07278/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/babf1523-fa08-4397-b0ff-946af7e07278/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c40f5b0d-64b1-4e58-afec-4dc4b19363d1/volumes/kubernetes.io~projected/kube-api-access-k229c:{mountpoint:/var/lib/kubelet/pods/c40f5b0d-64b1-4e58-afec-4dc4b19363d1/volumes/kubernetes.io~projected/kube-api-access-k229c major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c40f5b0d-64b1-4e58-afec-4dc4b19363d1/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/c40f5b0d-64b1-4e58-afec-4dc4b19363d1/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c40f5b0d-64b1-4e58-afec-4dc4b19363d1/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/c40f5b0d-64b1-4e58-afec-4dc4b19363d1/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c893dfcd-6f72-45bd-96ab-521e97584a40/volumes/kubernetes.io~projected/kube-api-access-bhqlv:{mountpoint:/var/lib/kubelet/pods/c893dfcd-6f72-45bd-96ab-521e97584a40/volumes/kubernetes.io~projected/kube-api-access-bhqlv major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c893dfcd-6f72-45bd-96ab-521e97584a40/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/c893dfcd-6f72-45bd-96ab-521e97584a40/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c893dfcd-6f72-45bd-96ab-521e97584a40/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/c893dfcd-6f72-45bd-96ab-521e97584a40/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3d709e2-f49d-4825-8beb-056b4582f358/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d3d709e2-f49d-4825-8beb-056b4582f358/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3d709e2-f49d-4825-8beb-056b4582f358/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d3d709e2-f49d-4825-8beb-056b4582f358/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3d709e2-f49d-4825-8beb-056b4582f358/volumes/kubernetes.io~projected/kube-api-access-xjhkj:{mountpoint:/var/lib/kubelet/pods/d3d709e2-f49d-4825-8beb-056b4582f358/volumes/kubernetes.io~projected/kube-api-access-xjhkj major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e354a328-13db-422e-b0e7-0cc0a5af5a90/volumes/kubernetes.io~projected/kube-api-access-c9g7j:{mountpoint:/var/lib/kubelet/pods/e354a328-13db-422e-b0e7-0cc0a5af5a90/volumes/kubernetes.io~projected/kube-api-access-c9g7j major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4a3f746-8b39-4e1e-8f18-8694b16929e1/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e4a3f746-8b39-4e1e-8f18-8694b16929e1/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f/volumes/kubernetes.io~projected/kube-api-access-smnk7:{mountpoint:/var/lib/kubelet/pods/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f/volumes/kubernetes.io~projected/kube-api-access-smnk7 major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c6fcd805d62767424de0df96028015ffb0056d48759d6b4ce471161b76ea9037/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/03af9577bfd3e4a3c44472097109dfaecd61923ddeacf19fe2669252327cadc5/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c2b52136818138ff604ddbd5209433d6128d819ea8168d08e371d2563fe86206/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/2d7f730bb529f9a5e8e11dd60262e21ec72c83af5ccc3a9609df95ce3f675aea/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/6da6131afe68b4cf0b73a64a6168c8f8c0c8648746cbfb94e129b4ce2fb2339e/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d0ec5d84414e992714c8b2677f3332910d44b38cc823b6d8ff0e04b6b7887a7b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/75d46fa4d82d635ab1388374082560fe975e94b4dcd82f516c5da6fa6750d52c/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/1fcb0e905a7c8009972ac654aac73831320a8cb5fdd7143460065a7481892d56/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/049ca3907022c57f7743933c6477d14ff339c2921303759646ecc1ca167862bc/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/0c664ba63df6d29dae4036fa5e2214e8c60d7dbd9bb6329dc2547e8f6b0a6483/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/b79f4f658ff885cdf46bd390b7b06561e94b56aa8346318d4964e2d79f98f0f8/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/51e684fe2998e187c821e4bda10da70fe3146c746d25a061d93ac6be0f300026/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/189b1b4f976393478ed29f23b495f9ace8e0db724d44632d985a08bdb7207c99/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/df6c662aeb296b905310b08ca2655d3dc0377bc95997c450f3416e929ed9fac4/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/6d9736e3a27bf28fb54385435cc67eea3118d48f393f54464adeca9de2ff528f/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/a9eebbc4c51626ef5180253c71f839de0726af1e64ef918db039c4b73f937dc9/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/7ce0bb5909acc8a68d87f6ba2a3a1b880a8c87dda8f83d4083d8d24fde52b9e5/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/eba7f090944f22ad798fe0a88983f98a04d7ddba52c27a64c2be9bbb88776be8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/9a277e6e71473605dccc1458855de372b3169baf4a7744fc73298fbb58c5db02/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f23b6f2cf3cc723829d804dcf1d8a170107caaee55bea2c3e6e418b391506247/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/374901c7fe326af3167bcab766e663f43bb10420008458a71087f5dc011ffd85/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/ebca5907ac0d39b24cbf76f157becc0b967b4ceb16d3059b71401085c4bde081/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/f78f2339a53bf71b49f14b4a5e9e315d759fc5abf0f60df2ac1b50fb7c4d0391/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/6a1a3e1b45b2cdf079b5885f0ad953490128b0b9fca98cc7720d725bb9a2b887/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/dec9fdfbf41d0b1d1ac06827ec6b1bb73dad2bf4f49c13309802664b56e9d76f/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/4279a2d98fa820455754b6cd91c333f265870592aa73906787e4749fd6f4e9d7/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/ea85fe72406f4bc058403246c3acddac1a8d2c0adbda60f0775a61aea1dd00a6/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/74057667e6d9c8ab4a94a50ad99855e0ae351d8e978876bb59c7befc0df81599/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/b584e20319c1d42c967d73bab044792aae8963cb78d508c572a3f2dbf6a6a629/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/a816210668286e81c0001dd770403aff4d85bc9479b35fad49c0ba0fdc0929f4/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/7291e43252c8f5e42b1ee7c116afaf297dcdaedffe8c7140a6749a12ff13247c/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/dcf14f5df68d5a9e0dbe95f1d3ecbeb5f0dffa4a5d83715ebffbcde511d7a21f/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/b95067cf9b8fa91581243931404979b98810018f159add285178cb34d21de2a6/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/3d11ffa739b1e3c547b1a821528094ef5522b8b76970e84159f24320eebaec4d/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/5b7986ae6a0370690d7ef2c58aa0e6204f5814a2a2495ca37613536ac52b723a/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/9a88e4916927b50f90588dddac0fad4c0c9b63b14898bebce487ed6042a3dfb5/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/b5e8a0edafd7e6ed7de88e72196418834293ac8c83dbac3adca8ed64fe48998d/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/4e693ec7cbcb90f1717ab30d59ae09e75f766314cf7e48248a0da28fe08c0fb2/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/5f817e1bda8ba73b8022f67c0c372ac77f438194aa005a01fe846fee7ecc034f/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/11f411e28466bc91c441f09a9deb723ba1e37dc428c7541182ab9fafd397ad4f/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/3722dbc49e99e167a0010a59e92ca5e91fdda9bf216a22efde0252a27d8764ec/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/433996e02966de94ae4b55452a26940fe8ae1d817dbe17b740d3dab0cf62dc83/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/95d1bb7c78cecc690247f8fc95554f4831d99e721cda2c2786de5dbe9206185a/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/1c91f3e4a937f4581424d9aca35392966d0fb33badd9589368b746ff184efbb4/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/6e70267ef5f851531178dead591a905c0b44e9f95660f5811e6bd61eb0ebca65/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/7b7f9e56564b50b9c4202d062d397877b9c663bad9e0529c66ea578d1c52cb51/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/c4bf5bff628e4ef8724212404db4b122f61892cf3ab76dd351e56bdc82257949/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/8af934acc855aca2ba8c2b707c432845b4994f35fbfa482843245b4182203d6b/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/7f7c9d86614fb8ec0600e49e32478c5f49fb06b2c62ca214f1bdc508a8a5f9b6/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/d67e31c40e6eac73d7ac656717e1e0c1996a73075dd17b34897125d290f91e80/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/fecd338d06035c3b85b22321b6745d9bad391451421e98e651a4f9b5218c0530/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/1d59022a04735c33ff7ce63aa8e3a3102f2b684df3542ca7bf6f75a6e01c897f/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/c616870dde515f1f0d260438162acdde8436620dbfe7c85889d1e2b41f0eadab/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/2df144519b3ff088a9c1e093d0f05fa89b9c100e2994b5cfb0e592c863657235/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/1b1296b04d76fa11005548b72eca9ed846fa66a347c4b5e72f713128ba5c1ed5/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/ee01e13e5e377bf2cf736c8ebbcfac92b020c6a6507a54dad30d4d58cd58b43d/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/c224ce647421a5005905ee9623ea78f09e5acc8c92654f3c12b90d801edbcfba/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/fad4ef5a33cfae42a2795cc38d9a8281788f9e8b93917f93c3dfdecc74f24791/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/a4a65677cf5579fcfb7b8583bba956dae57250afa4dac159066f1bfe108ad81e/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/a4d8c2ec18d465daf0da42d7a123d5207174a3c71496115dfa69c16250c24698/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-61:{mountpoint:/var/lib/containers/storage/overlay/a70ac4b9c0c78718a592e3f643cf92e48d7ea5b9439b13a8820e3265aee37490/merged major:0 minor:61 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/bc17c14961ce9feac3adc644c2e493e64fa0d9c29c2e2d6aed9b05c1ec2c352f/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/0d11f0354ffc8f9b73562244797d735c8eb061a2a05932a08ae92dadb27d2c44/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/42a4f88415195fddc1f99a85488b21f84d4bf3a7bc8873bc62026a79eb514174/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/ec82b5e1fff79a894666c33bb2eaa55e54ed22092dc5f62f86fa9fc6f21a6f7b/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/c3caa344d31b39eaaf9585fc4bbce2a18bddd9c08739f8faa8251fb8cdd9373a/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/910857bcaa2209d78fa65108d854f846a20460969f9ed578209a3d223ff13984/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/b189cca185aa957917f82630908f500643e5e67d8edb4a8e997c6969e6fb56b7/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/f842640175fdc4b3eea1d4be50f2191e05810c54fcc9e4f9ddc467bf8cef456b/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/6c9f170f8cd2051178847364a8cc5e9bf1e38047e4dc210169033de1909d8d2a/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 17:33:48.268176 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.267773 5549 manager.go:217] Machine: {Timestamp:2026-11-02 17:33:48.267307026 +0000 UTC m=+0.055812552 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100072 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec268d136db8825b1663df7af2e7ecac SystemUUID:ec268d13-6db8-825b-1663-df7af2e7ecac BootID:21454693-700e-486d-ac79-41437dc05655 Filesystems:[{Device:/var/lib/kubelet/pods/e4a3f746-8b39-4e1e-8f18-8694b16929e1/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d3d709e2-f49d-4825-8beb-056b4582f358/volumes/kubernetes.io~projected/kube-api-access-xjhkj DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/babf1523-fa08-4397-b0ff-946af7e07278/volumes/kubernetes.io~projected/kube-api-access-9mgbp DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc423511a76bc5316042893587560d7579bd92ccbe01c5cfae00d72c7c6d986d/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/53eac522-fee5-4865-955b-a14da456d693/volumes/kubernetes.io~projected/kube-api-access-hqx5s DeviceMajor:0 DeviceMinor:259 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:298 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2fdba41a69a6707dad0cee32e99cc6cf3fca17653c509f5d9e05d1d05d4d95e/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21691b74-73cc-41de-9c08-0f5e27528dba/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:282 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~projected/kube-api-access-bhglv DeviceMajor:0 DeviceMinor:299 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b69d99f5-8b53-47cf-8a76-917f606488b9/volumes/kubernetes.io~projected/kube-api-access-pjxms DeviceMajor:0 DeviceMinor:370 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c40f5b0d-64b1-4e58-afec-4dc4b19363d1/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:396 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/feb45ffd2f207fab99820b663ea19ea9492c80779f5d187e329bf65dc5f9eef8/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b56787b1149f33c8e1c2e99c8075b6d1d15514e11a0f42716ef22c26ddf92785/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:290 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b6949d0-0006-4488-89ad-296077fcf045/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c40f5b0d-64b1-4e58-afec-4dc4b19363d1/volumes/kubernetes.io~projected/kube-api-access-k229c DeviceMajor:0 DeviceMinor:398 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67a04d53-e707-4b0b-9e32-d84c31e413e5/volumes/kubernetes.io~projected/kube-api-access-6zmv4 DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:298 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/83c43619022e0e860a985f870be3b4af3fd8192dd3c00b5f5e92cedcfb143cc1/userdata/shm DeviceMajor:0 DeviceMinor:383 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/558e70e5-ac1c-4205-af45-7ef40a464aed/volumes/kubernetes.io~projected/kube-api-access-tszcx DeviceMajor:0 DeviceMinor:387 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c893dfcd-6f72-45bd-96ab-521e97584a40/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:297 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5cc3d45cd04b69a4845d7f5ea07058de3bf85d842ebdefe66c247a3b149fc9a2/userdata/shm DeviceMajor:0 DeviceMinor:343 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b6949d0-0006-4488-89ad-296077fcf045/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/53eac522-fee5-4865-955b-a14da456d693/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:253 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:298 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d3d709e2-f49d-4825-8beb-056b4582f358/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/babf1523-fa08-4397-b0ff-946af7e07278/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c893dfcd-6f72-45bd-96ab-521e97584a40/volumes/kubernetes.io~projected/kube-api-access-bhqlv DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a5511f9cffc9fac7dd2b70f2be71a0228fca3de17c8f3f98cd11efcbcd93c3e/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3df7744d-a04e-42aa-8938-31b27451b921/volumes/kubernetes.io~projected/kube-api-access-56frp DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/467b0f4d69f4df830fb36f0cbb9bd27f32d920c9ecc16042114bf4434a1f1987/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:292 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/290c0bd1-b749-4ec9-88e1-384ac37be16f/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b69d99f5-8b53-47cf-8a76-917f606488b9/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:369 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f/volumes/kubernetes.io~projected/kube-api-access-smnk7 DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b4290a8c2bddeff8a97107329477306e785e45f2dcafea75b5829d40eeed550/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/290c0bd1-b749-4ec9-88e1-384ac37be16f/volumes/kubernetes.io~projected/kube-api-access-gvw4s DeviceMajor:0 DeviceMinor:342 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a73fde2c-c65f-4374-adef-ee60a09c0b4a/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:382 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c40f5b0d-64b1-4e58-afec-4dc4b19363d1/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:397 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:297 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18dccc9f1e70001661daa816359127248b0dd741e83971552f35aa25039d4056/userdata/shm DeviceMajor:0 DeviceMinor:390 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33cc3c3636522c210a2c55e06391e20ffab4fd130b5a0973557180aa5e1650fb/userdata/shm DeviceMajor:0 DeviceMinor:300 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d3d709e2-f49d-4825-8beb-056b4582f358/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62013dcf-af15-4041-bdc2-f5437c000082/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:377 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/21691b74-73cc-41de-9c08-0f5e27528dba/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:277 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c893dfcd-6f72-45bd-96ab-521e97584a40/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:231 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/290c0bd1-b749-4ec9-88e1-384ac37be16f/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:341 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f86bb36cade1759273de3d44e951ef8f897a140526e207664028c81fa9128f21/userdata/shm DeviceMajor:0 DeviceMinor:399 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/491761ca-d114-411c-9196-3bd0065ae73a/volumes/kubernetes.io~projected/kube-api-access-k7t6q DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-61 DeviceMajor:0 DeviceMinor:61 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89b301aa1389bd1fdafc0b8dabd74be65a483068163c7370cefae600058e32a0/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/117152fb4e06d47da9759eeef3882fced47fe07ca2e5f9b81d4e457754570e1e/userdata/shm DeviceMajor:0 DeviceMinor:58 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/290c0bd1-b749-4ec9-88e1-384ac37be16f/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:339 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33140d4e6a3933461db90cac160879545d0b84614a36600d3b5dffbf02127232/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ce00f5670141c9170eeeaf58029aaa782c2afbb5545da08b06be9b391dbb4e7/userdata/shm DeviceMajor:0 DeviceMinor:371 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae6e2a1d-9467-4ef5-b493-6e788c0917b6/volumes/kubernetes.io~projected/kube-api-access-cfmgm DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/636356f6-e4d8-475e-8c82-dbbeb5fee0c3/volumes/kubernetes.io~projected/kube-api-access-qlxdl DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/53eac522-fee5-4865-955b-a14da456d693/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:258 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b005971739cb10211378c9b0a94bf2285179c7c51972f4c35877f4ac3d8fa06/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/328a3d179ba928d8ce082f2f6fd43397b4c113843153f6d55913dd89af0d158b/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:296 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94abd7d1ee7d72da8ddde0f1c9482b5488ef6fc7b9c41523a7facdad001dfc7c/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e354a328-13db-422e-b0e7-0cc0a5af5a90/volumes/kubernetes.io~projected/kube-api-access-c9g7j DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56ac149e79f43a9ad9644a11d2ac29a4f6edae69595c394ce0e3363f373237cc/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ddb4c2750aae48a90222d31bb22af2e9ce23331dcb3818f535773f2bae20e02f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3df7744d-a04e-42aa-8938-31b27451b921/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c3c9a9c9257b3ab55d41cf4c87481c5756c0be2fa8ef59da9153ad33b2f5b3b/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:291 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/67a04d53-e707-4b0b-9e32-d84c31e413e5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21691b74-73cc-41de-9c08-0f5e27528dba/volumes/kubernetes.io~projected/kube-api-access-n9wbw DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/23e84589-7092-4074-bfc6-d5dd9ea2d4cc/volumes/kubernetes.io~projected/kube-api-access-h8kcn DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:294 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:298 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/290c0bd1-b749-4ec9-88e1-384ac37be16f/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:335 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d742c4e4b17cbc9c9527de69d2c5c7bdffddc869ba92a947a1ba804e84f8733b/userdata/shm DeviceMajor:0 DeviceMinor:385 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b6949d0-0006-4488-89ad-296077fcf045/volumes/kubernetes.io~projected/kube-api-access-h9w5w DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a73fde2c-c65f-4374-adef-ee60a09c0b4a/volumes/kubernetes.io~projected/kube-api-access-nnhzb DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b69d99f5-8b53-47cf-8a76-917f606488b9/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:365 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:18dccc9f1e70001 MacAddress:22:e0:4c:92:5f:4b Speed:10000 Mtu:8901} {Name:2ce00f5670141c9 MacAddress:c2:86:61:e3:3b:1a Speed:10000 Mtu:8901} {Name:33140d4e6a39334 MacAddress:1a:dd:af:42:5d:e1 Speed:10000 Mtu:8901} {Name:33cc3c3636522c2 MacAddress:f2:6f:25:ff:96:c6 Speed:10000 Mtu:8901} {Name:467b0f4d69f4df8 MacAddress:5e:3f:46:8d:08:5e Speed:10000 Mtu:8901} {Name:4c3c9a9c9257b3a MacAddress:66:00:a0:c5:71:2d Speed:10000 Mtu:8901} {Name:5a5511f9cffc9fa MacAddress:36:5d:35:62:38:95 Speed:10000 Mtu:8901} {Name:5cc3d45cd04b69a MacAddress:9a:76:28:01:e3:e7 Speed:10000 Mtu:8901} {Name:83c43619022e0e8 MacAddress:ba:f5:57:fc:4f:c6 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cf:d2:70:db Speed:0 Mtu:9001} {Name:br-int MacAddress:6a:96:94:bb:27:ba Speed:0 Mtu:8901} {Name:d742c4e4b17cbc9 MacAddress:d6:af:57:81:1f:c4 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cf:d2:70:db Speed:0 Mtu:9001} {Name:f86bb36cade1759 MacAddress:ea:1e:ff:fd:78:27 Speed:10000 Mtu:8901} {Name:fc423511a76bc53 MacAddress:06:2c:ae:0a:c9:03 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:46:80:b9:97:48:97 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4a:f5:46:68:9f:79 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 17:33:48.268415 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.268180 5549 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 17:33:48.268415 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.268224 5549 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 17:33:48.268967 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.268905 5549 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 17:33:48.269613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.268970 5549 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-133.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 17:33:48.269613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.269416 5549 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 17:33:48.269613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.269435 5549 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 17:33:48.269613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.269455 5549 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:33:48.269613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.269499 5549 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:33:48.269836 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.269807 5549 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 17:33:48.270119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.270102 5549 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 17:33:48.270251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.270239 5549 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 17:33:48.270321 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.270290 5549 kubelet.go:493] "Attempting to sync node with API server" Nov 02 17:33:48.270321 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.270312 5549 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 17:33:48.270413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.270329 5549 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 17:33:48.270413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.270342 5549 kubelet.go:405] "Adding apiserver pod source" Nov 02 17:33:48.270413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.270354 5549 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 17:33:48.271089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.271072 5549 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 17:33:48.271089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.271088 5549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:33:48.271605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.271585 5549 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 17:33:48.273999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.273985 5549 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 17:33:48.273999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.273998 5549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:33:48.274107 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.274097 5549 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 17:33:48.274292 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.274282 5549 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 17:33:48.274335 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.274310 5549 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 17:33:48.274421 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.274411 5549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 17:33:48.274454 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.274432 5549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 17:33:48.274454 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.274442 5549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 17:33:48.274454 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.274453 5549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 17:33:48.274527 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.274459 5549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 17:33:48.274527 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.274468 5549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 17:33:48.274527 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.274478 5549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 17:33:48.274527 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.274487 5549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 17:33:48.274527 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.274497 5549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 17:33:48.274527 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.274503 5549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 17:33:48.274527 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.274512 5549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 17:33:48.274527 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.274522 5549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 17:33:48.274728 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.274545 5549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 17:33:48.274728 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.274553 5549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 17:33:48.277921 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.277908 5549 server.go:1263] "Started kubelet" Nov 02 17:33:48.279043 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.278019 5549 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 17:33:48.279043 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.278354 5549 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 17:33:48.279043 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.278743 5549 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 17:33:48.278495 ip-10-0-142-133 systemd[1]: Started Kubernetes Kubelet. Nov 02 17:33:48.279304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.279261 5549 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 17:33:48.280343 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.280325 5549 server.go:317] "Adding debug handlers to kubelet server" Nov 02 17:33:48.283129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.282914 5549 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 17:33:48.283656 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.283632 5549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 17:33:48.283779 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:33:48.283692 5549 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-133.us-east-2.compute.internal\" not found" Nov 02 17:33:48.283779 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.283735 5549 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 17:33:48.283779 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.283743 5549 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 17:33:48.284028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.284008 5549 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 17:33:48.284171 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.284058 5549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:27:38 +0000 UTC" deadline="2028-06-16 14:52:47.563255449 +0000 UTC" Nov 02 17:33:48.284263 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.284247 5549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14205h18m59.279015192s" Nov 02 17:33:48.286644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.286621 5549 factory.go:55] Registering systemd factory Nov 02 17:33:48.286644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.286646 5549 factory.go:223] Registration of the systemd container factory successfully Nov 02 17:33:48.287748 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.287728 5549 factory.go:153] Registering CRI-O factory Nov 02 17:33:48.287748 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.287749 5549 factory.go:223] Registration of the crio container factory successfully Nov 02 17:33:48.287892 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.287816 5549 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 17:33:48.287892 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.287843 5549 factory.go:103] Registering Raw factory Nov 02 17:33:48.287892 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.287868 5549 manager.go:1199] Started watching for new ooms in manager Nov 02 17:33:48.288056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.287879 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c40f5b0d-64b1-4e58-afec-4dc4b19363d1" volumeName="kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-oauth-serving-cert" seLinuxMountContext="" Nov 02 17:33:48.288056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.287930 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a73fde2c-c65f-4374-adef-ee60a09c0b4a" volumeName="kubernetes.io/projected/a73fde2c-c65f-4374-adef-ee60a09c0b4a-kube-api-access-nnhzb" seLinuxMountContext="" Nov 02 17:33:48.288056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.287964 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c40f5b0d-64b1-4e58-afec-4dc4b19363d1" volumeName="kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-serving-cert" seLinuxMountContext="" Nov 02 17:33:48.288056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.287987 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3df7744d-a04e-42aa-8938-31b27451b921" volumeName="kubernetes.io/secret/3df7744d-a04e-42aa-8938-31b27451b921-metrics-tls" seLinuxMountContext="" Nov 02 17:33:48.288056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288004 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67a04d53-e707-4b0b-9e32-d84c31e413e5" volumeName="kubernetes.io/projected/67a04d53-e707-4b0b-9e32-d84c31e413e5-kube-api-access-6zmv4" seLinuxMountContext="" Nov 02 17:33:48.288056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288019 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b69d99f5-8b53-47cf-8a76-917f606488b9" volumeName="kubernetes.io/empty-dir/b69d99f5-8b53-47cf-8a76-917f606488b9-tmp" seLinuxMountContext="" Nov 02 17:33:48.288056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288032 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3d709e2-f49d-4825-8beb-056b4582f358" volumeName="kubernetes.io/projected/d3d709e2-f49d-4825-8beb-056b4582f358-kube-api-access-xjhkj" seLinuxMountContext="" Nov 02 17:33:48.288056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288044 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e354a328-13db-422e-b0e7-0cc0a5af5a90" volumeName="kubernetes.io/projected/e354a328-13db-422e-b0e7-0cc0a5af5a90-kube-api-access-c9g7j" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288061 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21691b74-73cc-41de-9c08-0f5e27528dba" volumeName="kubernetes.io/secret/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-tls" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288075 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="290c0bd1-b749-4ec9-88e1-384ac37be16f" volumeName="kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-telemeter-client-tls" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288088 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="636356f6-e4d8-475e-8c82-dbbeb5fee0c3" volumeName="kubernetes.io/configmap/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288100 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b69d99f5-8b53-47cf-8a76-917f606488b9" volumeName="kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-console-config" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288113 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21691b74-73cc-41de-9c08-0f5e27528dba" volumeName="kubernetes.io/projected/21691b74-73cc-41de-9c08-0f5e27528dba-kube-api-access-n9wbw" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288128 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="290c0bd1-b749-4ec9-88e1-384ac37be16f" volumeName="kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288145 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="491761ca-d114-411c-9196-3bd0065ae73a" volumeName="kubernetes.io/projected/491761ca-d114-411c-9196-3bd0065ae73a-kube-api-access-k7t6q" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288161 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53eac522-fee5-4865-955b-a14da456d693" volumeName="kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-oauth-serving-cert" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288173 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c40f5b0d-64b1-4e58-afec-4dc4b19363d1" volumeName="kubernetes.io/empty-dir/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-tmp" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288189 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c40f5b0d-64b1-4e58-afec-4dc4b19363d1" volumeName="kubernetes.io/projected/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-kube-api-access-k229c" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288203 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e354a328-13db-422e-b0e7-0cc0a5af5a90" volumeName="kubernetes.io/empty-dir/e354a328-13db-422e-b0e7-0cc0a5af5a90-tmp-dir" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288217 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b083135-c135-437c-9446-5267ab9cb71f" volumeName="kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-kube-api-access-bhglv" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288229 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3df7744d-a04e-42aa-8938-31b27451b921" volumeName="kubernetes.io/projected/3df7744d-a04e-42aa-8938-31b27451b921-kube-api-access-56frp" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288247 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="558e70e5-ac1c-4205-af45-7ef40a464aed" volumeName="kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288260 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b69d99f5-8b53-47cf-8a76-917f606488b9" volumeName="kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-oauth-serving-cert" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288274 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="babf1523-fa08-4397-b0ff-946af7e07278" volumeName="kubernetes.io/secret/babf1523-fa08-4397-b0ff-946af7e07278-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288292 5549 manager.go:319] Starting recovery of all containers Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288286 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4a3f746-8b39-4e1e-8f18-8694b16929e1" volumeName="kubernetes.io/secret/e4a3f746-8b39-4e1e-8f18-8694b16929e1-agent-certs" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288312 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffdc716d-d67b-41da-aa12-c3ede5dd6b1f" volumeName="kubernetes.io/configmap/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f-serviceca" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288349 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b083135-c135-437c-9446-5267ab9cb71f" volumeName="kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288367 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21691b74-73cc-41de-9c08-0f5e27528dba" volumeName="kubernetes.io/configmap/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288393 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="290c0bd1-b749-4ec9-88e1-384ac37be16f" volumeName="kubernetes.io/configmap/290c0bd1-b749-4ec9-88e1-384ac37be16f-metrics-client-ca" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288408 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c40f5b0d-64b1-4e58-afec-4dc4b19363d1" volumeName="kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-config" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288425 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c40f5b0d-64b1-4e58-afec-4dc4b19363d1" volumeName="kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-trusted-ca-bundle" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288438 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3d709e2-f49d-4825-8beb-056b4582f358" volumeName="kubernetes.io/empty-dir/d3d709e2-f49d-4825-8beb-056b4582f358-tmp" seLinuxMountContext="" Nov 02 17:33:48.288435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288450 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b083135-c135-437c-9446-5267ab9cb71f" volumeName="kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-tls-assets" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288463 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="636356f6-e4d8-475e-8c82-dbbeb5fee0c3" volumeName="kubernetes.io/configmap/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-cni-binary-copy" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288478 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae6e2a1d-9467-4ef5-b493-6e788c0917b6" volumeName="kubernetes.io/projected/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-kube-api-access-cfmgm" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288489 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c893dfcd-6f72-45bd-96ab-521e97584a40" volumeName="kubernetes.io/projected/c893dfcd-6f72-45bd-96ab-521e97584a40-kube-api-access-bhqlv" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288501 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4a3f746-8b39-4e1e-8f18-8694b16929e1" volumeName="kubernetes.io/configmap/e4a3f746-8b39-4e1e-8f18-8694b16929e1-konnectivity-ca" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288513 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b083135-c135-437c-9446-5267ab9cb71f" volumeName="kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288524 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b083135-c135-437c-9446-5267ab9cb71f" volumeName="kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288540 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b083135-c135-437c-9446-5267ab9cb71f" volumeName="kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-cluster-tls-config" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288551 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3df7744d-a04e-42aa-8938-31b27451b921" volumeName="kubernetes.io/empty-dir/3df7744d-a04e-42aa-8938-31b27451b921-tmp-dir" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288564 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="636356f6-e4d8-475e-8c82-dbbeb5fee0c3" volumeName="kubernetes.io/configmap/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288576 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c893dfcd-6f72-45bd-96ab-521e97584a40" volumeName="kubernetes.io/secret/c893dfcd-6f72-45bd-96ab-521e97584a40-prometheus-operator-tls" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288590 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b083135-c135-437c-9446-5267ab9cb71f" volumeName="kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-web-config" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288601 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b083135-c135-437c-9446-5267ab9cb71f" volumeName="kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288613 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="290c0bd1-b749-4ec9-88e1-384ac37be16f" volumeName="kubernetes.io/configmap/290c0bd1-b749-4ec9-88e1-384ac37be16f-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288625 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b6949d0-0006-4488-89ad-296077fcf045" volumeName="kubernetes.io/configmap/2b6949d0-0006-4488-89ad-296077fcf045-ovnkube-script-lib" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288640 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53eac522-fee5-4865-955b-a14da456d693" volumeName="kubernetes.io/empty-dir/53eac522-fee5-4865-955b-a14da456d693-tmp" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288652 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62013dcf-af15-4041-bdc2-f5437c000082" volumeName="kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288677 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b69d99f5-8b53-47cf-8a76-917f606488b9" volumeName="kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-serving-cert" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288689 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="babf1523-fa08-4397-b0ff-946af7e07278" volumeName="kubernetes.io/configmap/babf1523-fa08-4397-b0ff-946af7e07278-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288706 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b083135-c135-437c-9446-5267ab9cb71f" volumeName="kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-main-db" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288717 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="290c0bd1-b749-4ec9-88e1-384ac37be16f" volumeName="kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-secret-telemeter-client" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288729 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53eac522-fee5-4865-955b-a14da456d693" volumeName="kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-console-config" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288741 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="636356f6-e4d8-475e-8c82-dbbeb5fee0c3" volumeName="kubernetes.io/projected/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-kube-api-access-qlxdl" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288754 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="babf1523-fa08-4397-b0ff-946af7e07278" volumeName="kubernetes.io/projected/babf1523-fa08-4397-b0ff-946af7e07278-kube-api-access-9mgbp" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288775 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c893dfcd-6f72-45bd-96ab-521e97584a40" volumeName="kubernetes.io/configmap/c893dfcd-6f72-45bd-96ab-521e97584a40-metrics-client-ca" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288787 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c893dfcd-6f72-45bd-96ab-521e97584a40" volumeName="kubernetes.io/secret/c893dfcd-6f72-45bd-96ab-521e97584a40-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288799 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3d709e2-f49d-4825-8beb-056b4582f358" volumeName="kubernetes.io/empty-dir/d3d709e2-f49d-4825-8beb-056b4582f358-etc-tuned" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288811 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b083135-c135-437c-9446-5267ab9cb71f" volumeName="kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-config-out" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288825 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23e84589-7092-4074-bfc6-d5dd9ea2d4cc" volumeName="kubernetes.io/configmap/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-cni-binary-copy" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288839 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="290c0bd1-b749-4ec9-88e1-384ac37be16f" volumeName="kubernetes.io/configmap/290c0bd1-b749-4ec9-88e1-384ac37be16f-serving-certs-ca-bundle" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288850 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53eac522-fee5-4865-955b-a14da456d693" volumeName="kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-oauth-config" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288863 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b69d99f5-8b53-47cf-8a76-917f606488b9" volumeName="kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-trusted-ca-bundle" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288874 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c40f5b0d-64b1-4e58-afec-4dc4b19363d1" volumeName="kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-oauth-config" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288886 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21691b74-73cc-41de-9c08-0f5e27528dba" volumeName="kubernetes.io/secret/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288908 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="290c0bd1-b749-4ec9-88e1-384ac37be16f" volumeName="kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-federate-client-tls" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288920 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3df7744d-a04e-42aa-8938-31b27451b921" volumeName="kubernetes.io/configmap/3df7744d-a04e-42aa-8938-31b27451b921-config-volume" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288933 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b69d99f5-8b53-47cf-8a76-917f606488b9" volumeName="kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-oauth-config" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288945 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffdc716d-d67b-41da-aa12-c3ede5dd6b1f" volumeName="kubernetes.io/projected/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f-kube-api-access-smnk7" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288973 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53eac522-fee5-4865-955b-a14da456d693" volumeName="kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-service-ca" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.288987 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b69d99f5-8b53-47cf-8a76-917f606488b9" volumeName="kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-service-ca" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289000 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b083135-c135-437c-9446-5267ab9cb71f" volumeName="kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-metrics-client-ca" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289012 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b083135-c135-437c-9446-5267ab9cb71f" volumeName="kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-config-volume" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289024 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="290c0bd1-b749-4ec9-88e1-384ac37be16f" volumeName="kubernetes.io/projected/290c0bd1-b749-4ec9-88e1-384ac37be16f-kube-api-access-gvw4s" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289043 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b6949d0-0006-4488-89ad-296077fcf045" volumeName="kubernetes.io/projected/2b6949d0-0006-4488-89ad-296077fcf045-kube-api-access-h9w5w" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289056 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="491761ca-d114-411c-9196-3bd0065ae73a" volumeName="kubernetes.io/configmap/491761ca-d114-411c-9196-3bd0065ae73a-iptables-alerter-script" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289071 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53eac522-fee5-4865-955b-a14da456d693" volumeName="kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-serving-cert" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289083 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a73fde2c-c65f-4374-adef-ee60a09c0b4a" volumeName="kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289097 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="babf1523-fa08-4397-b0ff-946af7e07278" volumeName="kubernetes.io/empty-dir/babf1523-fa08-4397-b0ff-946af7e07278-data-volume" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289109 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b083135-c135-437c-9446-5267ab9cb71f" volumeName="kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289122 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21691b74-73cc-41de-9c08-0f5e27528dba" volumeName="kubernetes.io/empty-dir/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-textfile" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289135 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23e84589-7092-4074-bfc6-d5dd9ea2d4cc" volumeName="kubernetes.io/projected/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-kube-api-access-h8kcn" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289147 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b6949d0-0006-4488-89ad-296077fcf045" volumeName="kubernetes.io/configmap/2b6949d0-0006-4488-89ad-296077fcf045-ovnkube-config" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289159 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b6949d0-0006-4488-89ad-296077fcf045" volumeName="kubernetes.io/configmap/2b6949d0-0006-4488-89ad-296077fcf045-env-overrides" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289172 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b6949d0-0006-4488-89ad-296077fcf045" volumeName="kubernetes.io/secret/2b6949d0-0006-4488-89ad-296077fcf045-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289184 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53eac522-fee5-4865-955b-a14da456d693" volumeName="kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-trusted-ca-bundle" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289198 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c40f5b0d-64b1-4e58-afec-4dc4b19363d1" volumeName="kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-service-ca" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289313 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21691b74-73cc-41de-9c08-0f5e27528dba" volumeName="kubernetes.io/configmap/21691b74-73cc-41de-9c08-0f5e27528dba-metrics-client-ca" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289327 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23e84589-7092-4074-bfc6-d5dd9ea2d4cc" volumeName="kubernetes.io/configmap/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-daemon-config" seLinuxMountContext="" Nov 02 17:33:48.289304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289339 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53eac522-fee5-4865-955b-a14da456d693" volumeName="kubernetes.io/projected/53eac522-fee5-4865-955b-a14da456d693-kube-api-access-hqx5s" seLinuxMountContext="" Nov 02 17:33:48.291006 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289353 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67a04d53-e707-4b0b-9e32-d84c31e413e5" volumeName="kubernetes.io/secret/67a04d53-e707-4b0b-9e32-d84c31e413e5-cert" seLinuxMountContext="" Nov 02 17:33:48.291006 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289371 5549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b69d99f5-8b53-47cf-8a76-917f606488b9" volumeName="kubernetes.io/projected/b69d99f5-8b53-47cf-8a76-917f606488b9-kube-api-access-pjxms" seLinuxMountContext="" Nov 02 17:33:48.291006 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289387 5549 reconstruct.go:97] "Volume reconstruction finished" Nov 02 17:33:48.291006 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.289394 5549 reconciler.go:29] "Reconciler: start to sync state" Nov 02 17:33:48.292927 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.292908 5549 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:33:48.293156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.293140 5549 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:33:48.295181 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.295154 5549 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 17:33:48.309897 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.309830 5549 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 17:33:48.311316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.311296 5549 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 17:33:48.311316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.311316 5549 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 17:33:48.311476 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.311333 5549 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 17:33:48.311476 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:33:48.311376 5549 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 17:33:48.316377 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.316351 5549 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:33:48.320149 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.320127 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 17:33:48.327834 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.327811 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 17:33:48.330551 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.330509 5549 generic.go:356] "Generic (PLEG): container finished" podID="1b083135-c135-437c-9446-5267ab9cb71f" containerID="1e32330efff8fe8589858662cbf717d812c6465e667dcc1f671e29ea86034ab7" exitCode=0 Nov 02 17:33:48.332216 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.332195 5549 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:33:48.338010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.337985 5549 generic.go:356] "Generic (PLEG): container finished" podID="21691b74-73cc-41de-9c08-0f5e27528dba" containerID="f4e37af1131c13b2208b02984bdf354d26e9a581774b63b44c9ab0130986b31e" exitCode=0 Nov 02 17:33:48.341358 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.341332 5549 generic.go:356] "Generic (PLEG): container finished" podID="636356f6-e4d8-475e-8c82-dbbeb5fee0c3" containerID="611742128b8dd7e48c06f5759cb297eb63c701dcf3011373fa849d8dd433f2c1" exitCode=0 Nov 02 17:33:48.341358 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.341357 5549 generic.go:356] "Generic (PLEG): container finished" podID="636356f6-e4d8-475e-8c82-dbbeb5fee0c3" containerID="fe0320edfb9cfc2ba020b576eed3f42390baba91b4dfcc3322a24ef04802d3b1" exitCode=0 Nov 02 17:33:48.341474 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.341371 5549 generic.go:356] "Generic (PLEG): container finished" podID="636356f6-e4d8-475e-8c82-dbbeb5fee0c3" containerID="3ae4901221914788b39c34e1f0bae225aed8b6c3e553ae75eb4e101b6abb31c2" exitCode=0 Nov 02 17:33:48.341474 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.341381 5549 generic.go:356] "Generic (PLEG): container finished" podID="636356f6-e4d8-475e-8c82-dbbeb5fee0c3" containerID="7dfb7538064fae7b2462e74e8ee259371c749ca27333735d6eed4f50a7df5b6a" exitCode=0 Nov 02 17:33:48.341474 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.341390 5549 generic.go:356] "Generic (PLEG): container finished" podID="636356f6-e4d8-475e-8c82-dbbeb5fee0c3" containerID="cdd3bc0e464d0af85dde40bf17031871b4166f05dd023372720f4ccb1fb4e448" exitCode=0 Nov 02 17:33:48.341474 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.341398 5549 generic.go:356] "Generic (PLEG): container finished" podID="636356f6-e4d8-475e-8c82-dbbeb5fee0c3" containerID="e6ff7ac3d5fcabebbf76cb3a5d890cbf1b4f009e6f4a25fcbda6adc03437e94c" exitCode=0 Nov 02 17:33:48.359413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.359386 5549 generic.go:356] "Generic (PLEG): container finished" podID="7fad33c464d219dfb7d5fa21dbe7ad22" containerID="dd4b4ea611219a8adc29fcc6357b4d7827cbfd182efd2c7520b801d5b68cdd50" exitCode=0 Nov 02 17:33:48.372749 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.372724 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 17:33:48.373091 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.373065 5549 generic.go:356] "Generic (PLEG): container finished" podID="2b6949d0-0006-4488-89ad-296077fcf045" containerID="018155f005a7df5780c9280646f8753de591b949f01e031efac5de73b98b99a0" exitCode=1 Nov 02 17:33:48.394560 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.394544 5549 manager.go:324] Recovery completed Nov 02 17:33:48.411699 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:33:48.411637 5549 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 17:33:48.417586 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.417567 5549 cpu_manager.go:226] "Starting" policy="none" Nov 02 17:33:48.417586 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.417581 5549 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 17:33:48.417751 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.417598 5549 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 17:33:48.417751 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.417712 5549 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 17:33:48.417751 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.417722 5549 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 17:33:48.417751 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.417739 5549 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 17:33:48.417751 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.417744 5549 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 17:33:48.417751 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.417750 5549 policy_none.go:50] "Start" Nov 02 17:33:48.417751 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.417757 5549 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 17:33:48.417986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.417766 5549 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 17:33:48.419725 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.419708 5549 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 17:33:48.419828 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.419727 5549 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 17:33:48.419828 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.419735 5549 policy_none.go:44] "Start" Nov 02 17:33:48.424286 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.424261 5549 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 17:33:48.427007 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.426993 5549 manager.go:341] "Starting Device Plugin manager" Nov 02 17:33:48.427099 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:33:48.427018 5549 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 17:33:48.427099 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.427030 5549 server.go:87] "Starting device plugin registration server" Nov 02 17:33:48.427345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.427332 5549 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 17:33:48.427388 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.427349 5549 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 17:33:48.427448 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.427429 5549 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 17:33:48.427572 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.427537 5549 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 17:33:48.427628 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.427580 5549 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 17:33:48.427628 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.427587 5549 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 17:33:48.428082 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.427999 5549 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:33:48.427569235Z","UUID":"25b22f1d-bb48-4e0f-98e3-cbfde1125acf","Handler":null,"Name":"","Endpoint":""} Nov 02 17:33:48.429174 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:33:48.429148 5549 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 17:33:48.429736 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.429652 5549 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 17:33:48.430029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.430017 5549 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 17:33:48.527468 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.527440 5549 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:33:48.529791 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.529772 5549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:33:48.529895 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.529806 5549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:33:48.529895 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.529820 5549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-133.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:33:48.529895 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.529875 5549 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:48.544532 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.544502 5549 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:48.544634 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.544579 5549 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:48.612745 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.612702 5549 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal"] Nov 02 17:33:48.613390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.613375 5549 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:48.613450 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.613387 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" event={"ID":"7fad33c464d219dfb7d5fa21dbe7ad22","Type":"ContainerStarted","Data":"36e71d021375da3d57e1cd4dbb75fa72772ae78607c9654114ab31ce583df154"} Nov 02 17:33:48.613450 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.613422 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" event={"ID":"7fad33c464d219dfb7d5fa21dbe7ad22","Type":"ContainerDied","Data":"dd4b4ea611219a8adc29fcc6357b4d7827cbfd182efd2c7520b801d5b68cdd50"} Nov 02 17:33:48.613450 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.613435 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" event={"ID":"7fad33c464d219dfb7d5fa21dbe7ad22","Type":"ContainerStarted","Data":"94abd7d1ee7d72da8ddde0f1c9482b5488ef6fc7b9c41523a7facdad001dfc7c"} Nov 02 17:33:48.613585 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.613457 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" event={"ID":"ebe0e6ace491eef2c5b9302a33c08d4d","Type":"ContainerStarted","Data":"a33817ba9f2e8217d28287ceafd29d00933024bce4ab5f1fb4433a52461b6707"} Nov 02 17:33:48.613585 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.613468 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" event={"ID":"ebe0e6ace491eef2c5b9302a33c08d4d","Type":"ContainerStarted","Data":"117152fb4e06d47da9759eeef3882fced47fe07ca2e5f9b81d4e457754570e1e"} Nov 02 17:33:48.613648 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.613600 5549 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:48.628035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.628014 5549 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:33:48.628124 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:33:48.628069 5549 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:48.629083 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.629063 5549 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:33:48.629139 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:33:48.629104 5549 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:48.697423 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.697398 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7fad33c464d219dfb7d5fa21dbe7ad22-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal\" (UID: \"7fad33c464d219dfb7d5fa21dbe7ad22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:48.697517 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.697429 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fad33c464d219dfb7d5fa21dbe7ad22-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal\" (UID: \"7fad33c464d219dfb7d5fa21dbe7ad22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:48.697517 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.697455 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebe0e6ace491eef2c5b9302a33c08d4d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal\" (UID: \"ebe0e6ace491eef2c5b9302a33c08d4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:48.797996 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.797945 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7fad33c464d219dfb7d5fa21dbe7ad22-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal\" (UID: \"7fad33c464d219dfb7d5fa21dbe7ad22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:48.798156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.798002 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fad33c464d219dfb7d5fa21dbe7ad22-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal\" (UID: \"7fad33c464d219dfb7d5fa21dbe7ad22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:48.798156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.798038 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebe0e6ace491eef2c5b9302a33c08d4d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal\" (UID: \"ebe0e6ace491eef2c5b9302a33c08d4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:48.798156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.798050 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7fad33c464d219dfb7d5fa21dbe7ad22-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal\" (UID: \"7fad33c464d219dfb7d5fa21dbe7ad22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:48.798156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.798082 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebe0e6ace491eef2c5b9302a33c08d4d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal\" (UID: \"ebe0e6ace491eef2c5b9302a33c08d4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:48.798156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:48.798132 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fad33c464d219dfb7d5fa21dbe7ad22-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal\" (UID: \"7fad33c464d219dfb7d5fa21dbe7ad22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:49.270658 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.270623 5549 apiserver.go:52] "Watching apiserver" Nov 02 17:33:49.287847 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.287822 5549 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 17:33:49.288590 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.288564 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-fd64789dc-hcqt6","openshift-network-diagnostics/network-check-target-lnbcn","openshift-monitoring/prometheus-operator-9cd89b74c-5994g","openshift-console/console-5997464558-6x2v8","openshift-multus/multus-jw64s","kube-system/konnectivity-agent-8s6rw","openshift-console/console-7979dc65db-56v4t","openshift-network-operator/iptables-alerter-nx2vb","openshift-ovn-kubernetes/ovnkube-node-956tn","openshift-insights/insights-runtime-extractor-vr49c","kube-system/global-pull-secret-syncer-drvjl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal","openshift-monitoring/node-exporter-47799","openshift-console/console-558b6fbcdf-n4st4","openshift-ingress-canary/ingress-canary-8h628","openshift-image-registry/node-ca-dxmdj","openshift-multus/network-metrics-daemon-zcdw6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8","openshift-dns/dns-default-48wvf","openshift-monitoring/alertmanager-main-0","kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-m78lg","openshift-dns/node-resolver-278vv","openshift-multus/multus-additional-cni-plugins-z4ptm"] Nov 02 17:33:49.294329 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.294300 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 17:33:49.296579 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.296558 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 17:33:49.296884 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.296860 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 17:33:49.298560 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.298540 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 17:33:49.298647 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.298546 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 17:33:49.299311 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.299287 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 17:33:49.299401 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.299345 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 17:33:49.299401 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.299353 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 17:33:49.299401 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.299366 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dcw65\"" Nov 02 17:33:49.299401 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.299371 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sbqnj\"" Nov 02 17:33:49.299780 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.299758 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 17:33:49.299780 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.299764 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jl4vs\"" Nov 02 17:33:49.299903 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.299825 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 17:33:49.299903 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.299889 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 17:33:49.300025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.299999 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 17:33:49.300119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.300096 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 17:33:49.300578 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.300558 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:33:49.300672 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.300614 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 17:33:49.300672 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.300636 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 17:33:49.300672 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.300658 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 17:33:49.300784 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.300674 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 17:33:49.300784 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.300682 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 17:33:49.301071 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.301059 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 17:33:49.301240 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.301223 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 17:33:49.301386 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.301371 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 17:33:49.301386 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.301375 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 17:33:49.301386 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.301392 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 17:33:49.301533 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.301429 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lkfrw\"" Nov 02 17:33:49.301533 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.301455 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jmxfr\"" Nov 02 17:33:49.301658 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.301645 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 17:33:49.301799 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.301785 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 17:33:49.301799 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.301791 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zg2g5\"" Nov 02 17:33:49.302044 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.302030 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 17:33:49.302149 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.302067 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k78hz\"" Nov 02 17:33:49.302149 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.302067 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wjf7q\"" Nov 02 17:33:49.302149 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.302123 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 17:33:49.302483 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.302471 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-52q82\"" Nov 02 17:33:49.302904 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.302885 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 17:33:49.302904 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.302910 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xjkc9\"" Nov 02 17:33:49.303095 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.302987 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 17:33:49.303095 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.303037 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dtld9\"" Nov 02 17:33:49.303507 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.303490 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 17:33:49.303585 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.303514 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:33:49.303585 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.303543 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 17:33:49.303585 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.303553 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 17:33:49.303813 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.303784 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 17:33:49.304094 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.304077 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 17:33:49.304242 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.304222 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ttfxd\"" Nov 02 17:33:49.304314 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.304285 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vxnnv\"" Nov 02 17:33:49.304385 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.304372 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9wn4g\"" Nov 02 17:33:49.304447 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.304431 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 17:33:49.304842 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.304823 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 17:33:49.304917 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.304839 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 17:33:49.304917 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.304842 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 17:33:49.304917 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.304858 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qw8dn\"" Nov 02 17:33:49.304917 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.304879 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dp4wq\"" Nov 02 17:33:49.305444 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.305421 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 17:33:49.306057 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.306035 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 17:33:49.306141 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.306094 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 17:33:49.306141 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.306100 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 17:33:49.306434 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.306418 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 17:33:49.306495 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.306463 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 17:33:49.306774 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.306754 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 17:33:49.306863 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.306776 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 17:33:49.307220 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.307201 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 17:33:49.307410 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.307393 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 17:33:49.307885 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.307865 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 17:33:49.311072 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.311051 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 17:33:49.311151 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.311084 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 17:33:49.311151 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.311095 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 17:33:49.311151 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.311098 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 17:33:49.311269 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.311156 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 17:33:49.311362 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.311345 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 17:33:49.312091 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.312073 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 17:33:49.312091 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.312075 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 17:33:49.312204 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.312092 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 17:33:49.312274 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.312254 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 17:33:49.312274 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.312265 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 17:33:49.312274 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.312272 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 17:33:49.312487 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.312279 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 17:33:49.312487 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.312304 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 17:33:49.312487 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.312454 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 17:33:49.312566 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.312545 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gql28\"" Nov 02 17:33:49.312597 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.312577 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 17:33:49.312597 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.312594 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 17:33:49.312649 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.312630 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6p4gz\"" Nov 02 17:33:49.312677 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.312659 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 17:33:49.312705 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.312684 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h4n2v\"" Nov 02 17:33:49.312732 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.312724 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-smp64\"" Nov 02 17:33:49.319855 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.319833 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 17:33:49.319855 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.319846 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 17:33:49.377495 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.377467 5549 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:49.377703 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.377641 5549 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:49.386797 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.386779 5549 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 17:33:49.390535 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.390515 5549 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:33:49.390718 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:33:49.390559 5549 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:49.391374 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.391352 5549 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:33:49.391447 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:33:49.391390 5549 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-133.us-east-2.compute.internal" Nov 02 17:33:49.399697 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.399676 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-telemeter-client-tls\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:49.399747 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.399705 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.399747 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.399726 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-cnibin\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:33:49.399747 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.399743 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-console-config\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:49.399872 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.399761 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k229c\" (UniqueName: \"kubernetes.io/projected/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-kube-api-access-k229c\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:49.399872 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.399776 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62013dcf-af15-4041-bdc2-f5437c000082-dbus\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:49.399872 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.399844 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-console-config\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:49.400036 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.399899 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-slash\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.400036 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400029 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.400130 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400034 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-telemeter-client-tls\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:49.400130 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400034 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.400130 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400074 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-federate-client-tls\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:49.400275 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400128 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-oauth-config\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:49.400275 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400154 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4a3f746-8b39-4e1e-8f18-8694b16929e1-konnectivity-ca\") pod \"konnectivity-agent-8s6rw\" (UID: \"e4a3f746-8b39-4e1e-8f18-8694b16929e1\") " pod="kube-system/konnectivity-agent-8s6rw" Nov 02 17:33:49.400275 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400180 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-oauth-serving-cert\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:49.400275 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400208 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3df7744d-a04e-42aa-8938-31b27451b921-config-volume\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:49.400275 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400255 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.400525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400258 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-run-systemd\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.400525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400347 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-oauth-config\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:49.400525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400369 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4a3f746-8b39-4e1e-8f18-8694b16929e1-konnectivity-ca\") pod \"konnectivity-agent-8s6rw\" (UID: \"e4a3f746-8b39-4e1e-8f18-8694b16929e1\") " pod="kube-system/konnectivity-agent-8s6rw" Nov 02 17:33:49.400525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400389 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/491761ca-d114-411c-9196-3bd0065ae73a-host-slash\") pod \"iptables-alerter-nx2vb\" (UID: \"491761ca-d114-411c-9196-3bd0065ae73a\") " pod="openshift-network-operator/iptables-alerter-nx2vb" Nov 02 17:33:49.400525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400413 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-federate-client-tls\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:49.400525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400424 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqx5s\" (UniqueName: \"kubernetes.io/projected/53eac522-fee5-4865-955b-a14da456d693-kube-api-access-hqx5s\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:49.400525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400424 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-oauth-serving-cert\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:49.400525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400462 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3d709e2-f49d-4825-8beb-056b4582f358-etc-tuned\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.400525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400489 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnhzb\" (UniqueName: \"kubernetes.io/projected/a73fde2c-c65f-4374-adef-ee60a09c0b4a-kube-api-access-nnhzb\") pod \"network-metrics-daemon-zcdw6\" (UID: \"a73fde2c-c65f-4374-adef-ee60a09c0b4a\") " pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:49.400525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400514 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-system-cni-dir\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:33:49.400525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400517 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3df7744d-a04e-42aa-8938-31b27451b921-config-volume\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:49.400999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400549 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-run-multus-certs\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.400999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400573 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3d709e2-f49d-4825-8beb-056b4582f358-etc-tuned\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.400999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400593 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-console-config\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:49.400999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400618 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-console-config\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:49.400999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400584 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b69d99f5-8b53-47cf-8a76-917f606488b9-tmp\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:49.400999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400655 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b69d99f5-8b53-47cf-8a76-917f606488b9-tmp\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:49.400999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400678 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-oauth-config\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:49.400999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400708 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/290c0bd1-b749-4ec9-88e1-384ac37be16f-metrics-client-ca\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:49.400999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400738 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.400999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400765 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9wbw\" (UniqueName: \"kubernetes.io/projected/21691b74-73cc-41de-9c08-0f5e27528dba-kube-api-access-n9wbw\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.400999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400788 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-tmp\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:49.400999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400814 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-sysconfig\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.400999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400829 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-oauth-config\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:49.400999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400839 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-sysctl-d\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.400999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400877 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-etc-openvswitch\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.400999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400907 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-tmp\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:49.400999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400934 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/290c0bd1-b749-4ec9-88e1-384ac37be16f-metrics-client-ca\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:49.400999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.400975 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401013 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-cni-netd\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401041 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b6949d0-0006-4488-89ad-296077fcf045-ovnkube-config\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401062 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401182 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401205 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b6949d0-0006-4488-89ad-296077fcf045-ovnkube-config\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401212 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-run\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401238 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3df7744d-a04e-42aa-8938-31b27451b921-tmp-dir\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401266 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-system-cni-dir\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401294 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c893dfcd-6f72-45bd-96ab-521e97584a40-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401321 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56frp\" (UniqueName: \"kubernetes.io/projected/3df7744d-a04e-42aa-8938-31b27451b921-kube-api-access-56frp\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401340 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3df7744d-a04e-42aa-8938-31b27451b921-tmp-dir\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401348 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401358 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-kubelet\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401391 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/491761ca-d114-411c-9196-3bd0065ae73a-iptables-alerter-script\") pod \"iptables-alerter-nx2vb\" (UID: \"491761ca-d114-411c-9196-3bd0065ae73a\") " pod="openshift-network-operator/iptables-alerter-nx2vb" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401431 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-sys-fs\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401462 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62013dcf-af15-4041-bdc2-f5437c000082-kubelet-config\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401495 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvw4s\" (UniqueName: \"kubernetes.io/projected/290c0bd1-b749-4ec9-88e1-384ac37be16f-kube-api-access-gvw4s\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401500 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c893dfcd-6f72-45bd-96ab-521e97584a40-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401522 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/491761ca-d114-411c-9196-3bd0065ae73a-iptables-alerter-script\") pod \"iptables-alerter-nx2vb\" (UID: \"491761ca-d114-411c-9196-3bd0065ae73a\") " pod="openshift-network-operator/iptables-alerter-nx2vb" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401536 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-log-socket\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401677 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c893dfcd-6f72-45bd-96ab-521e97584a40-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401702 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-serving-cert\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401717 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67a04d53-e707-4b0b-9e32-d84c31e413e5-cert\") pod \"ingress-canary-8h628\" (UID: \"67a04d53-e707-4b0b-9e32-d84c31e413e5\") " pod="openshift-ingress-canary/ingress-canary-8h628" Nov 02 17:33:49.401765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401760 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401786 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401816 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c893dfcd-6f72-45bd-96ab-521e97584a40-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401833 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-run-openvswitch\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401859 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-socket-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401889 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-device-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401901 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-serving-cert\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401940 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402007 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67a04d53-e707-4b0b-9e32-d84c31e413e5-cert\") pod \"ingress-canary-8h628\" (UID: \"67a04d53-e707-4b0b-9e32-d84c31e413e5\") " pod="openshift-ingress-canary/ingress-canary-8h628" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.401941 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21691b74-73cc-41de-9c08-0f5e27528dba-sys\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402027 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402046 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21691b74-73cc-41de-9c08-0f5e27528dba-root\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402065 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-cni-dir\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402090 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-cni-binary-copy\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402127 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-run-netns\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402186 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-conf-dir\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402216 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-serving-cert\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402241 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-oauth-serving-cert\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402286 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tszcx\" (UniqueName: \"kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx\") pod \"network-check-target-lnbcn\" (UID: \"558e70e5-ac1c-4205-af45-7ef40a464aed\") " pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402304 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-cni-binary-copy\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402404 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-oauth-serving-cert\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402405 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402450 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-node-log\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402490 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-registration-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402492 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-serving-cert\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402539 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs\") pod \"network-metrics-daemon-zcdw6\" (UID: \"a73fde2c-c65f-4374-adef-ee60a09c0b4a\") " pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402564 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-service-ca\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402580 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402600 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-var-lib-kubelet\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402627 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3d709e2-f49d-4825-8beb-056b4582f358-tmp\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402651 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/babf1523-fa08-4397-b0ff-946af7e07278-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402675 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402698 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-serving-cert\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402720 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.402715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402733 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-service-ca\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402749 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhglv\" (UniqueName: \"kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-kube-api-access-bhglv\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402736 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3d709e2-f49d-4825-8beb-056b4582f358-tmp\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402772 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-wtmp\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402807 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21691b74-73cc-41de-9c08-0f5e27528dba-metrics-client-ca\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402804 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a73fde2c-c65f-4374-adef-ee60a09c0b4a-metrics-certs\") pod \"network-metrics-daemon-zcdw6\" (UID: \"a73fde2c-c65f-4374-adef-ee60a09c0b4a\") " pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402892 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402900 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/babf1523-fa08-4397-b0ff-946af7e07278-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402933 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f-host\") pod \"node-ca-dxmdj\" (UID: \"ffdc716d-d67b-41da-aa12-c3ede5dd6b1f\") " pod="openshift-image-registry/node-ca-dxmdj" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402941 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62013dcf-af15-4041-bdc2-f5437c000082-original-pull-secret\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402974 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.402986 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403027 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-var-lib-cni-multus\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403118 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-serving-cert\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403123 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21691b74-73cc-41de-9c08-0f5e27528dba-metrics-client-ca\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403139 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-systemd\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403165 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-accelerators-collector-config\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403187 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-daemon-config\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403205 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403231 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhqlv\" (UniqueName: \"kubernetes.io/projected/c893dfcd-6f72-45bd-96ab-521e97584a40-kube-api-access-bhqlv\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403333 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-trusted-ca-bundle\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403343 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-accelerators-collector-config\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403356 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-daemon-config\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403431 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403512 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403544 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-var-lib-openvswitch\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403597 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjhkj\" (UniqueName: \"kubernetes.io/projected/d3d709e2-f49d-4825-8beb-056b4582f358-kube-api-access-xjhkj\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403621 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f-serviceca\") pod \"node-ca-dxmdj\" (UID: \"ffdc716d-d67b-41da-aa12-c3ede5dd6b1f\") " pod="openshift-image-registry/node-ca-dxmdj" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403688 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-trusted-ca-bundle\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403708 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smnk7\" (UniqueName: \"kubernetes.io/projected/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f-kube-api-access-smnk7\") pod \"node-ca-dxmdj\" (UID: \"ffdc716d-d67b-41da-aa12-c3ede5dd6b1f\") " pod="openshift-image-registry/node-ca-dxmdj" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403756 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-web-config\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403780 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4a3f746-8b39-4e1e-8f18-8694b16929e1-agent-certs\") pod \"konnectivity-agent-8s6rw\" (UID: \"e4a3f746-8b39-4e1e-8f18-8694b16929e1\") " pod="kube-system/konnectivity-agent-8s6rw" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403811 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/290c0bd1-b749-4ec9-88e1-384ac37be16f-serving-certs-ca-bundle\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403725 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403826 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f-serviceca\") pod \"node-ca-dxmdj\" (UID: \"ffdc716d-d67b-41da-aa12-c3ede5dd6b1f\") " pod="openshift-image-registry/node-ca-dxmdj" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403843 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-config-out\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403885 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403912 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7t6q\" (UniqueName: \"kubernetes.io/projected/491761ca-d114-411c-9196-3bd0065ae73a-kube-api-access-k7t6q\") pod \"iptables-alerter-nx2vb\" (UID: \"491761ca-d114-411c-9196-3bd0065ae73a\") " pod="openshift-network-operator/iptables-alerter-nx2vb" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403919 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-config-out\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403978 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-var-lib-kubelet\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403981 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-web-config\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.403995 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/290c0bd1-b749-4ec9-88e1-384ac37be16f-serving-certs-ca-bundle\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404005 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-sysctl-conf\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404042 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-secret-telemeter-client\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404070 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-trusted-ca-bundle\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404092 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404092 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4a3f746-8b39-4e1e-8f18-8694b16929e1-agent-certs\") pod \"konnectivity-agent-8s6rw\" (UID: \"e4a3f746-8b39-4e1e-8f18-8694b16929e1\") " pod="kube-system/konnectivity-agent-8s6rw" Nov 02 17:33:49.404129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404094 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3df7744d-a04e-42aa-8938-31b27451b921-metrics-tls\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404156 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404189 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-run-netns\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404219 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxdl\" (UniqueName: \"kubernetes.io/projected/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-kube-api-access-qlxdl\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404226 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404246 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-cnibin\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404259 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-secret-telemeter-client\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404290 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-service-ca\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404307 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/babf1523-fa08-4397-b0ff-946af7e07278-crio-socket\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404323 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mgbp\" (UniqueName: \"kubernetes.io/projected/babf1523-fa08-4397-b0ff-946af7e07278-kube-api-access-9mgbp\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404328 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3df7744d-a04e-42aa-8938-31b27451b921-metrics-tls\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404338 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-run-ovn\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404352 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-run-k8s-cni-cncf-io\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404369 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9g7j\" (UniqueName: \"kubernetes.io/projected/e354a328-13db-422e-b0e7-0cc0a5af5a90-kube-api-access-c9g7j\") pod \"node-resolver-278vv\" (UID: \"e354a328-13db-422e-b0e7-0cc0a5af5a90\") " pod="openshift-dns/node-resolver-278vv" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404389 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-oauth-serving-cert\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404411 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zmv4\" (UniqueName: \"kubernetes.io/projected/67a04d53-e707-4b0b-9e32-d84c31e413e5-kube-api-access-6zmv4\") pod \"ingress-canary-8h628\" (UID: \"67a04d53-e707-4b0b-9e32-d84c31e413e5\") " pod="openshift-ingress-canary/ingress-canary-8h628" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404462 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-oauth-config\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404514 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-trusted-ca-bundle\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404567 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/babf1523-fa08-4397-b0ff-946af7e07278-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404595 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfmgm\" (UniqueName: \"kubernetes.io/projected/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-kube-api-access-cfmgm\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404626 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-service-ca\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404737 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9w5w\" (UniqueName: \"kubernetes.io/projected/2b6949d0-0006-4488-89ad-296077fcf045-kube-api-access-h9w5w\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404753 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/babf1523-fa08-4397-b0ff-946af7e07278-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404772 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-modprobe-d\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404799 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-cni-bin\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404838 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b6949d0-0006-4488-89ad-296077fcf045-ovn-node-metrics-cert\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404797 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-oauth-config\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404869 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-var-lib-cni-bin\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404907 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-oauth-serving-cert\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.404980 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-etc-kubernetes\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405032 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e354a328-13db-422e-b0e7-0cc0a5af5a90-tmp-dir\") pod \"node-resolver-278vv\" (UID: \"e354a328-13db-422e-b0e7-0cc0a5af5a90\") " pod="openshift-dns/node-resolver-278vv" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405058 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-config\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405082 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-run-ovn-kubernetes\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405122 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-tls\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405135 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e354a328-13db-422e-b0e7-0cc0a5af5a90-tmp-dir\") pod \"node-resolver-278vv\" (UID: \"e354a328-13db-422e-b0e7-0cc0a5af5a90\") " pod="openshift-dns/node-resolver-278vv" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405153 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53eac522-fee5-4865-955b-a14da456d693-tmp\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405213 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-lib-modules\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405218 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53eac522-fee5-4865-955b-a14da456d693-tmp\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405281 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-config\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405285 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b6949d0-0006-4488-89ad-296077fcf045-ovn-node-metrics-cert\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405309 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405357 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b6949d0-0006-4488-89ad-296077fcf045-ovnkube-script-lib\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405390 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-tls\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405404 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405442 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-sys\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405466 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405505 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-textfile\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405545 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b6949d0-0006-4488-89ad-296077fcf045-ovnkube-script-lib\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405561 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-textfile\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405572 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c893dfcd-6f72-45bd-96ab-521e97584a40-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405555 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/290c0bd1-b749-4ec9-88e1-384ac37be16f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405597 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b6949d0-0006-4488-89ad-296077fcf045-env-overrides\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405623 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/290c0bd1-b749-4ec9-88e1-384ac37be16f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405647 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-systemd-units\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405672 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405695 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-trusted-ca-bundle\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405718 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-kubernetes\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405741 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjxms\" (UniqueName: \"kubernetes.io/projected/b69d99f5-8b53-47cf-8a76-917f606488b9-kube-api-access-pjxms\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405783 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/babf1523-fa08-4397-b0ff-946af7e07278-data-volume\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405804 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c893dfcd-6f72-45bd-96ab-521e97584a40-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405806 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-socket-dir-parent\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405844 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-service-ca\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405864 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-host\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405886 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-hostroot\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405909 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8kcn\" (UniqueName: \"kubernetes.io/projected/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-kube-api-access-h8kcn\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.405934 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e354a328-13db-422e-b0e7-0cc0a5af5a90-hosts-file\") pod \"node-resolver-278vv\" (UID: \"e354a328-13db-422e-b0e7-0cc0a5af5a90\") " pod="openshift-dns/node-resolver-278vv" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.406029 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b6949d0-0006-4488-89ad-296077fcf045-env-overrides\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.406112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.406119 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/babf1523-fa08-4397-b0ff-946af7e07278-data-volume\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:49.412158 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.410947 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-trusted-ca-bundle\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:49.412158 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.411456 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.412158 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.411615 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-service-ca\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:49.412158 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.411643 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:33:49.412158 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.411708 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56frp\" (UniqueName: \"kubernetes.io/projected/3df7744d-a04e-42aa-8938-31b27451b921-kube-api-access-56frp\") pod \"dns-default-48wvf\" (UID: \"3df7744d-a04e-42aa-8938-31b27451b921\") " pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.412728 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9wbw\" (UniqueName: \"kubernetes.io/projected/21691b74-73cc-41de-9c08-0f5e27528dba-kube-api-access-n9wbw\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.412742 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqx5s\" (UniqueName: \"kubernetes.io/projected/53eac522-fee5-4865-955b-a14da456d693-kube-api-access-hqx5s\") pod \"console-5997464558-6x2v8\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.413233 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smnk7\" (UniqueName: \"kubernetes.io/projected/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f-kube-api-access-smnk7\") pod \"node-ca-dxmdj\" (UID: \"ffdc716d-d67b-41da-aa12-c3ede5dd6b1f\") " pod="openshift-image-registry/node-ca-dxmdj" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.413319 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k229c\" (UniqueName: \"kubernetes.io/projected/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-kube-api-access-k229c\") pod \"console-7979dc65db-56v4t\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.413739 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/290c0bd1-b749-4ec9-88e1-384ac37be16f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.414014 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvw4s\" (UniqueName: \"kubernetes.io/projected/290c0bd1-b749-4ec9-88e1-384ac37be16f-kube-api-access-gvw4s\") pod \"telemeter-client-fd64789dc-hcqt6\" (UID: \"290c0bd1-b749-4ec9-88e1-384ac37be16f\") " pod="openshift-monitoring/telemeter-client-fd64789dc-hcqt6" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.414393 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnhzb\" (UniqueName: \"kubernetes.io/projected/a73fde2c-c65f-4374-adef-ee60a09c0b4a-kube-api-access-nnhzb\") pod \"network-metrics-daemon-zcdw6\" (UID: \"a73fde2c-c65f-4374-adef-ee60a09c0b4a\") " pod="openshift-multus/network-metrics-daemon-zcdw6" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.414991 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjhkj\" (UniqueName: \"kubernetes.io/projected/d3d709e2-f49d-4825-8beb-056b4582f358-kube-api-access-xjhkj\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.415171 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9w5w\" (UniqueName: \"kubernetes.io/projected/2b6949d0-0006-4488-89ad-296077fcf045-kube-api-access-h9w5w\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.415278 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7t6q\" (UniqueName: \"kubernetes.io/projected/491761ca-d114-411c-9196-3bd0065ae73a-kube-api-access-k7t6q\") pod \"iptables-alerter-nx2vb\" (UID: \"491761ca-d114-411c-9196-3bd0065ae73a\") " pod="openshift-network-operator/iptables-alerter-nx2vb" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.415621 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjxms\" (UniqueName: \"kubernetes.io/projected/b69d99f5-8b53-47cf-8a76-917f606488b9-kube-api-access-pjxms\") pod \"console-558b6fbcdf-n4st4\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.415690 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mgbp\" (UniqueName: \"kubernetes.io/projected/babf1523-fa08-4397-b0ff-946af7e07278-kube-api-access-9mgbp\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.415947 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfmgm\" (UniqueName: \"kubernetes.io/projected/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-kube-api-access-cfmgm\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.416070 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zmv4\" (UniqueName: \"kubernetes.io/projected/67a04d53-e707-4b0b-9e32-d84c31e413e5-kube-api-access-6zmv4\") pod \"ingress-canary-8h628\" (UID: \"67a04d53-e707-4b0b-9e32-d84c31e413e5\") " pod="openshift-ingress-canary/ingress-canary-8h628" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.416395 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9g7j\" (UniqueName: \"kubernetes.io/projected/e354a328-13db-422e-b0e7-0cc0a5af5a90-kube-api-access-c9g7j\") pod \"node-resolver-278vv\" (UID: \"e354a328-13db-422e-b0e7-0cc0a5af5a90\") " pod="openshift-dns/node-resolver-278vv" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.416470 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlxdl\" (UniqueName: \"kubernetes.io/projected/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-kube-api-access-qlxdl\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.416505 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tszcx\" (UniqueName: \"kubernetes.io/projected/558e70e5-ac1c-4205-af45-7ef40a464aed-kube-api-access-tszcx\") pod \"network-check-target-lnbcn\" (UID: \"558e70e5-ac1c-4205-af45-7ef40a464aed\") " pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.416508 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhqlv\" (UniqueName: \"kubernetes.io/projected/c893dfcd-6f72-45bd-96ab-521e97584a40-kube-api-access-bhqlv\") pod \"prometheus-operator-9cd89b74c-5994g\" (UID: \"c893dfcd-6f72-45bd-96ab-521e97584a40\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5994g" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.416585 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8kcn\" (UniqueName: \"kubernetes.io/projected/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-kube-api-access-h8kcn\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.420169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.417119 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhglv\" (UniqueName: \"kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-kube-api-access-bhglv\") pod \"alertmanager-main-0\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:33:49.505710 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.505681 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5997464558-6x2v8"] Nov 02 17:33:49.506261 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506238 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.506311 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506270 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-node-log\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.506311 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506286 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-registration-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.506383 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506330 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-node-log\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.506383 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506335 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.506383 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506370 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-var-lib-kubelet\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.506501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506385 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-registration-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.506501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506394 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-wtmp\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.506501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506411 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f-host\") pod \"node-ca-dxmdj\" (UID: \"ffdc716d-d67b-41da-aa12-c3ede5dd6b1f\") " pod="openshift-image-registry/node-ca-dxmdj" Nov 02 17:33:49.506501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506418 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-var-lib-kubelet\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.506501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506432 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-var-lib-cni-multus\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.506501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506449 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-systemd\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.506501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506454 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21691b74-73cc-41de-9c08-0f5e27528dba-node-exporter-wtmp\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.506501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506466 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.506501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506480 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-var-lib-cni-multus\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.506501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506488 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffdc716d-d67b-41da-aa12-c3ede5dd6b1f-host\") pod \"node-ca-dxmdj\" (UID: \"ffdc716d-d67b-41da-aa12-c3ede5dd6b1f\") " pod="openshift-image-registry/node-ca-dxmdj" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506481 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-var-lib-openvswitch\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506521 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506513 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-systemd\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506538 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-var-lib-openvswitch\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506547 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-var-lib-kubelet\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506563 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-sysctl-conf\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506584 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-run-netns\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506602 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-cnibin\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506615 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/babf1523-fa08-4397-b0ff-946af7e07278-crio-socket\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506614 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-var-lib-kubelet\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506636 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-run-netns\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506638 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-run-ovn\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506670 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-run-ovn\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506675 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/babf1523-fa08-4397-b0ff-946af7e07278-crio-socket\") pod \"insights-runtime-extractor-vr49c\" (UID: \"babf1523-fa08-4397-b0ff-946af7e07278\") " pod="openshift-insights/insights-runtime-extractor-vr49c" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506675 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-run-k8s-cni-cncf-io\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506681 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-cnibin\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506702 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-modprobe-d\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506705 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-run-k8s-cni-cncf-io\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506738 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-cni-bin\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506748 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-sysctl-conf\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506789 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-cni-bin\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506795 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-var-lib-cni-bin\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506812 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-modprobe-d\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506826 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-etc-kubernetes\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506860 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-run-ovn-kubernetes\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506862 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-var-lib-cni-bin\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506885 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-lib-modules\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506899 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-etc-kubernetes\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506910 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506934 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-sys\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.506926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506905 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-run-ovn-kubernetes\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507001 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-lib-modules\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507011 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-sys\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507016 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-systemd-units\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507045 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-kubernetes\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.506941 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507057 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-systemd-units\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507070 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-socket-dir-parent\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507092 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-host\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507099 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-kubernetes\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507131 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-hostroot\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507142 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-socket-dir-parent\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507156 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e354a328-13db-422e-b0e7-0cc0a5af5a90-hosts-file\") pod \"node-resolver-278vv\" (UID: \"e354a328-13db-422e-b0e7-0cc0a5af5a90\") " pod="openshift-dns/node-resolver-278vv" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507179 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-hostroot\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507182 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-cnibin\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507145 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-host\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507195 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e354a328-13db-422e-b0e7-0cc0a5af5a90-hosts-file\") pod \"node-resolver-278vv\" (UID: \"e354a328-13db-422e-b0e7-0cc0a5af5a90\") " pod="openshift-dns/node-resolver-278vv" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507216 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-cnibin\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507223 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62013dcf-af15-4041-bdc2-f5437c000082-dbus\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507250 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62013dcf-af15-4041-bdc2-f5437c000082-dbus\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507249 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-slash\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507274 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-run-systemd\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507279 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-slash\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507292 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/491761ca-d114-411c-9196-3bd0065ae73a-host-slash\") pod \"iptables-alerter-nx2vb\" (UID: \"491761ca-d114-411c-9196-3bd0065ae73a\") " pod="openshift-network-operator/iptables-alerter-nx2vb" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507313 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-system-cni-dir\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507327 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-run-multus-certs\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507342 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-sysconfig\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507355 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-sysctl-d\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507366 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/491761ca-d114-411c-9196-3bd0065ae73a-host-slash\") pod \"iptables-alerter-nx2vb\" (UID: \"491761ca-d114-411c-9196-3bd0065ae73a\") " pod="openshift-network-operator/iptables-alerter-nx2vb" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507376 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-system-cni-dir\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507386 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-run-systemd\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507429 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-run-multus-certs\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507391 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-etc-openvswitch\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507466 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-etc-openvswitch\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507479 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-cni-netd\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507486 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-sysconfig\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507499 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-etc-sysctl-d\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507503 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507534 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-cni-netd\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507547 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/636356f6-e4d8-475e-8c82-dbbeb5fee0c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z4ptm\" (UID: \"636356f6-e4d8-475e-8c82-dbbeb5fee0c3\") " pod="openshift-multus/multus-additional-cni-plugins-z4ptm" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507548 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-run\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507588 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-system-cni-dir\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507612 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-kubelet\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507634 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-sys-fs\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507642 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3d709e2-f49d-4825-8beb-056b4582f358-run\") pod \"tuned-m78lg\" (UID: \"d3d709e2-f49d-4825-8beb-056b4582f358\") " pod="openshift-cluster-node-tuning-operator/tuned-m78lg" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507655 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-host-kubelet\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507637 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-system-cni-dir\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507655 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62013dcf-af15-4041-bdc2-f5437c000082-kubelet-config\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507687 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-log-socket\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507692 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-sys-fs\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507712 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-run-openvswitch\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507748 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-log-socket\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507733 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-socket-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507772 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-device-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507782 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b6949d0-0006-4488-89ad-296077fcf045-run-openvswitch\") pod \"ovnkube-node-956tn\" (UID: \"2b6949d0-0006-4488-89ad-296077fcf045\") " pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507787 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21691b74-73cc-41de-9c08-0f5e27528dba-sys\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507808 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21691b74-73cc-41de-9c08-0f5e27528dba-root\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507823 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-socket-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507831 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-cni-dir\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507849 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae6e2a1d-9467-4ef5-b493-6e788c0917b6-device-dir\") pod \"aws-ebs-csi-driver-node-fz6j8\" (UID: \"ae6e2a1d-9467-4ef5-b493-6e788c0917b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz6j8" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507880 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62013dcf-af15-4041-bdc2-f5437c000082-kubelet-config\") pod \"global-pull-secret-syncer-drvjl\" (UID: \"62013dcf-af15-4041-bdc2-f5437c000082\") " pod="kube-system/global-pull-secret-syncer-drvjl" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507886 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21691b74-73cc-41de-9c08-0f5e27528dba-root\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507900 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-run-netns\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507920 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-conf-dir\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507929 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21691b74-73cc-41de-9c08-0f5e27528dba-sys\") pod \"node-exporter-47799\" (UID: \"21691b74-73cc-41de-9c08-0f5e27528dba\") " pod="openshift-monitoring/node-exporter-47799" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507982 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-cni-dir\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.507993 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-host-run-netns\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:49.508093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:49.508007 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23e84589-7092-4074-bfc6-d5dd9ea2d4cc-multus-conf-dir\") pod \"multus-jw64s\" (UID: \"23e84589-7092-4074-bfc6-d5dd9ea2d4cc\") " pod="openshift-multus/multus-jw64s" Nov 02 17:33:50.390537 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:50.390508 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:50.395350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:50.395326 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:50.410382 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:50.410357 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-48wvf" Nov 02 17:33:51.416427 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:51.416393 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:51.513889 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:51.513857 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:33:51.560357 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:51.560310 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:53.229307 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:53.229269 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:53.397390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:53.397357 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:55.929090 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:55.929053 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:33:56.715921 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:56.715893 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:57.044186 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:57.044085 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:33:57.108377 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:57.108343 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:57.280346 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:57.280315 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:57.424467 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:57.424377 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:58.658177 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:58.658152 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:33:58.658555 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:33:58.658349 5549 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:34:05.404921 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:05.404837 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:34:05.551143 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:05.551105 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:34:07.049323 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:07.049295 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:34:07.275323 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:07.275290 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:34:07.417848 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:07.417762 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:34:08.148072 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.148034 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:34:08.285016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.284973 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:34:08.562526 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.562492 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-9456bdf8d-wxx6s"] Nov 02 17:34:08.567641 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.567624 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.577026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.577003 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9456bdf8d-wxx6s"] Nov 02 17:34:08.681987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.681921 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-console-serving-cert\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.681987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.681978 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-console-config\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.682176 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.682084 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-console-oauth-config\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.682176 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.682109 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpfns\" (UniqueName: \"kubernetes.io/projected/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-kube-api-access-tpfns\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.682176 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.682156 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-tmp\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.682176 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.682174 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-trusted-ca-bundle\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.682292 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.682209 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-oauth-serving-cert\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.682292 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.682273 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-service-ca\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.691626 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.691602 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:34:08.783187 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.783153 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-oauth-serving-cert\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.783273 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.783192 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-service-ca\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.783273 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.783235 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-console-serving-cert\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.783273 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.783260 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-console-config\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.783405 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.783288 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-console-oauth-config\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.783405 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.783311 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpfns\" (UniqueName: \"kubernetes.io/projected/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-kube-api-access-tpfns\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.783405 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.783337 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-tmp\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.783405 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.783362 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-trusted-ca-bundle\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.783698 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.783669 5549 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 17:34:08.784050 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.784022 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-tmp\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.784203 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.784050 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-service-ca\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.784203 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.784051 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-oauth-serving-cert\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.784337 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.784099 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-console-config\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.784463 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.784446 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-trusted-ca-bundle\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.785921 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.785902 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-console-serving-cert\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.786011 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.785922 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-console-oauth-config\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.793210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.791845 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpfns\" (UniqueName: \"kubernetes.io/projected/48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47-kube-api-access-tpfns\") pod \"console-9456bdf8d-wxx6s\" (UID: \"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47\") " pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.827791 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.827734 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:34:08.877763 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.877745 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:08.903290 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:08.903267 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8s6rw" Nov 02 17:34:09.146571 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:09.146501 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9456bdf8d-wxx6s"] Nov 02 17:34:09.149879 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:34:09.149852 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cf59b2_8d8a_45f7_a81c_71bdf3ba1e47.slice/crio-52ba0c992a22a741497c499da840be9b449427a52b767be24052595a227d7689 WatchSource:0}: Error finding container 52ba0c992a22a741497c499da840be9b449427a52b767be24052595a227d7689: Status 404 returned error can't find the container with id 52ba0c992a22a741497c499da840be9b449427a52b767be24052595a227d7689 Nov 02 17:34:09.196549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:09.196524 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-956tn" Nov 02 17:34:09.450620 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:09.450589 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9456bdf8d-wxx6s" event={"ID":"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47","Type":"ContainerStarted","Data":"2756420ceeabb14315ed285f0b3b5b4016aa7b1b4b24603be66b9cacb95fed42"} Nov 02 17:34:09.450792 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:09.450642 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9456bdf8d-wxx6s" event={"ID":"48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47","Type":"ContainerStarted","Data":"52ba0c992a22a741497c499da840be9b449427a52b767be24052595a227d7689"} Nov 02 17:34:09.468652 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:09.468604 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-9456bdf8d-wxx6s" podStartSLOduration=1.468593015 podStartE2EDuration="1.468593015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:09.468137874 +0000 UTC m=+21.256643402" watchObservedRunningTime="2026-11-02 17:34:09.468593015 +0000 UTC m=+21.257098536" Nov 02 17:34:09.929660 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:09.929578 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:34:10.066158 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:10.066124 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:34:12.319183 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:12.319147 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:34:12.319595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:12.319465 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="alertmanager" containerID="cri-o://32100d40e27e116fa94ce3cd4f88a18c31d2f73730b501aae36e15387d8bd2fb" gracePeriod=120 Nov 02 17:34:12.319783 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:12.319758 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="prom-label-proxy" containerID="cri-o://a48c41142da65c840ebace4f02e42440fbc361d6df950be23f4b4220dcc5890f" gracePeriod=120 Nov 02 17:34:12.319865 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:12.319852 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="kube-rbac-proxy-metric" containerID="cri-o://e7b7ed98d053e64b85b97cea7ff349f2132a9a6eb40a4cd485466ae34a81fc3c" gracePeriod=120 Nov 02 17:34:12.319915 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:12.319904 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="kube-rbac-proxy" containerID="cri-o://51ff36c3b5f28fc2eef6f476ee40097f0be3b41a5506e631d42efc3c56da1b64" gracePeriod=120 Nov 02 17:34:12.320020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:12.319996 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="kube-rbac-proxy-web" containerID="cri-o://d1a1531f37c2c54e81a18a913cfdd1b61dc5031fa128c3c420e27d5e058cef36" gracePeriod=120 Nov 02 17:34:12.320020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:12.320021 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="config-reloader" containerID="cri-o://7f8bee42ef2baea4c975aa3066e5bbcc94be4a7e5bf918e440d9fe573893f266" gracePeriod=120 Nov 02 17:34:12.463822 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:12.463795 5549 generic.go:356] "Generic (PLEG): container finished" podID="1b083135-c135-437c-9446-5267ab9cb71f" containerID="a48c41142da65c840ebace4f02e42440fbc361d6df950be23f4b4220dcc5890f" exitCode=0 Nov 02 17:34:12.463822 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:12.463818 5549 generic.go:356] "Generic (PLEG): container finished" podID="1b083135-c135-437c-9446-5267ab9cb71f" containerID="e7b7ed98d053e64b85b97cea7ff349f2132a9a6eb40a4cd485466ae34a81fc3c" exitCode=0 Nov 02 17:34:12.463822 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:12.463825 5549 generic.go:356] "Generic (PLEG): container finished" podID="1b083135-c135-437c-9446-5267ab9cb71f" containerID="51ff36c3b5f28fc2eef6f476ee40097f0be3b41a5506e631d42efc3c56da1b64" exitCode=0 Nov 02 17:34:12.463822 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:12.463830 5549 generic.go:356] "Generic (PLEG): container finished" podID="1b083135-c135-437c-9446-5267ab9cb71f" containerID="7f8bee42ef2baea4c975aa3066e5bbcc94be4a7e5bf918e440d9fe573893f266" exitCode=0 Nov 02 17:34:12.464078 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:12.463836 5549 generic.go:356] "Generic (PLEG): container finished" podID="1b083135-c135-437c-9446-5267ab9cb71f" containerID="32100d40e27e116fa94ce3cd4f88a18c31d2f73730b501aae36e15387d8bd2fb" exitCode=0 Nov 02 17:34:12.464078 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:12.463863 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b083135-c135-437c-9446-5267ab9cb71f","Type":"ContainerDied","Data":"a48c41142da65c840ebace4f02e42440fbc361d6df950be23f4b4220dcc5890f"} Nov 02 17:34:12.464078 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:12.463919 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b083135-c135-437c-9446-5267ab9cb71f","Type":"ContainerDied","Data":"e7b7ed98d053e64b85b97cea7ff349f2132a9a6eb40a4cd485466ae34a81fc3c"} Nov 02 17:34:12.464078 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:12.463945 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b083135-c135-437c-9446-5267ab9cb71f","Type":"ContainerDied","Data":"51ff36c3b5f28fc2eef6f476ee40097f0be3b41a5506e631d42efc3c56da1b64"} Nov 02 17:34:12.464078 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:12.463989 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b083135-c135-437c-9446-5267ab9cb71f","Type":"ContainerDied","Data":"7f8bee42ef2baea4c975aa3066e5bbcc94be4a7e5bf918e440d9fe573893f266"} Nov 02 17:34:12.464078 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:12.464015 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b083135-c135-437c-9446-5267ab9cb71f","Type":"ContainerDied","Data":"32100d40e27e116fa94ce3cd4f88a18c31d2f73730b501aae36e15387d8bd2fb"} Nov 02 17:34:13.047540 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.047512 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:34:13.190254 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.190221 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:34:13.471527 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.471494 5549 generic.go:356] "Generic (PLEG): container finished" podID="1b083135-c135-437c-9446-5267ab9cb71f" containerID="d1a1531f37c2c54e81a18a913cfdd1b61dc5031fa128c3c420e27d5e058cef36" exitCode=0 Nov 02 17:34:13.471890 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.471538 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b083135-c135-437c-9446-5267ab9cb71f","Type":"ContainerDied","Data":"d1a1531f37c2c54e81a18a913cfdd1b61dc5031fa128c3c420e27d5e058cef36"} Nov 02 17:34:13.563206 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.563178 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:13.729210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.729174 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1b083135-c135-437c-9446-5267ab9cb71f\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " Nov 02 17:34:13.729375 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.729219 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy\") pod \"1b083135-c135-437c-9446-5267ab9cb71f\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " Nov 02 17:34:13.729375 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.729254 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-main-tls\") pod \"1b083135-c135-437c-9446-5267ab9cb71f\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " Nov 02 17:34:13.729375 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.729278 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-cluster-tls-config\") pod \"1b083135-c135-437c-9446-5267ab9cb71f\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " Nov 02 17:34:13.729375 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.729313 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-config-out\") pod \"1b083135-c135-437c-9446-5267ab9cb71f\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " Nov 02 17:34:13.729375 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.729360 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-web-config\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-web-config\") pod \"1b083135-c135-437c-9446-5267ab9cb71f\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " Nov 02 17:34:13.729657 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.729406 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-main-db\") pod \"1b083135-c135-437c-9446-5267ab9cb71f\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " Nov 02 17:34:13.729657 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.729439 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-trusted-ca-bundle\") pod \"1b083135-c135-437c-9446-5267ab9cb71f\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " Nov 02 17:34:13.729657 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.729482 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-tls-assets\" (UniqueName: \"kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-tls-assets\") pod \"1b083135-c135-437c-9446-5267ab9cb71f\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " Nov 02 17:34:13.729657 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.729515 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-kube-api-access-bhglv\" (UniqueName: \"kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-kube-api-access-bhglv\") pod \"1b083135-c135-437c-9446-5267ab9cb71f\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " Nov 02 17:34:13.729657 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.729546 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-metrics-client-ca\") pod \"1b083135-c135-437c-9446-5267ab9cb71f\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " Nov 02 17:34:13.729657 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.729591 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-config-volume\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-config-volume\") pod \"1b083135-c135-437c-9446-5267ab9cb71f\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " Nov 02 17:34:13.729657 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.729634 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-web\") pod \"1b083135-c135-437c-9446-5267ab9cb71f\" (UID: \"1b083135-c135-437c-9446-5267ab9cb71f\") " Nov 02 17:34:13.732650 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.732488 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-metrics-client-ca" pod "1b083135-c135-437c-9446-5267ab9cb71f" (UID: "1b083135-c135-437c-9446-5267ab9cb71f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:13.732768 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.732704 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-main-db" pod "1b083135-c135-437c-9446-5267ab9cb71f" (UID: "1b083135-c135-437c-9446-5267ab9cb71f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:34:13.732969 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.732851 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-trusted-ca-bundle" pod "1b083135-c135-437c-9446-5267ab9cb71f" (UID: "1b083135-c135-437c-9446-5267ab9cb71f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:13.733152 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.733128 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-config-out" pod "1b083135-c135-437c-9446-5267ab9cb71f" (UID: "1b083135-c135-437c-9446-5267ab9cb71f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:34:13.733219 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.733135 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-tls-assets" pod "1b083135-c135-437c-9446-5267ab9cb71f" (UID: "1b083135-c135-437c-9446-5267ab9cb71f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:13.733219 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.733180 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy" pod "1b083135-c135-437c-9446-5267ab9cb71f" (UID: "1b083135-c135-437c-9446-5267ab9cb71f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:13.733327 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.733252 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-metric" pod "1b083135-c135-437c-9446-5267ab9cb71f" (UID: "1b083135-c135-437c-9446-5267ab9cb71f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:13.733372 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.733337 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-main-tls" pod "1b083135-c135-437c-9446-5267ab9cb71f" (UID: "1b083135-c135-437c-9446-5267ab9cb71f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:13.733652 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.733636 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-web" pod "1b083135-c135-437c-9446-5267ab9cb71f" (UID: "1b083135-c135-437c-9446-5267ab9cb71f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:13.734480 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.734459 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-kube-api-access-bhglv" pod "1b083135-c135-437c-9446-5267ab9cb71f" (UID: "1b083135-c135-437c-9446-5267ab9cb71f"). InnerVolumeSpecName "kube-api-access-bhglv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:13.734634 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.734608 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-config-volume" pod "1b083135-c135-437c-9446-5267ab9cb71f" (UID: "1b083135-c135-437c-9446-5267ab9cb71f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:13.737452 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.737433 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-cluster-tls-config" pod "1b083135-c135-437c-9446-5267ab9cb71f" (UID: "1b083135-c135-437c-9446-5267ab9cb71f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:13.743095 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.743072 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-web-config" pod "1b083135-c135-437c-9446-5267ab9cb71f" (UID: "1b083135-c135-437c-9446-5267ab9cb71f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:13.830725 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.830679 5549 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:13.830725 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.830722 5549 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:13.830725 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.830734 5549 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:13.830923 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.830744 5549 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-secret-alertmanager-main-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:13.830923 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.830754 5549 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-cluster-tls-config\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:13.830923 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.830765 5549 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-config-out\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:13.830923 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.830774 5549 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-web-config\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:13.830923 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.830783 5549 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-main-db\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:13.830923 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.830792 5549 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:13.830923 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.830800 5549 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-tls-assets\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:13.830923 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.830809 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhglv\" (UniqueName: \"kubernetes.io/projected/1b083135-c135-437c-9446-5267ab9cb71f-kube-api-access-bhglv\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:13.830923 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.830817 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b083135-c135-437c-9446-5267ab9cb71f-metrics-client-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:13.830923 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:13.830825 5549 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b083135-c135-437c-9446-5267ab9cb71f-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:14.323738 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.323706 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-558b6fbcdf-n4st4" podUID="b69d99f5-8b53-47cf-8a76-917f606488b9" containerName="console" containerID="cri-o://5bb18eb75a166b592fceed1afd3b2495f1761ce94e331d60827b2098f94eeecf" gracePeriod=15 Nov 02 17:34:14.478101 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.478069 5549 generic.go:356] "Generic (PLEG): container finished" podID="b69d99f5-8b53-47cf-8a76-917f606488b9" containerID="5bb18eb75a166b592fceed1afd3b2495f1761ce94e331d60827b2098f94eeecf" exitCode=0 Nov 02 17:34:14.478508 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.478138 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-558b6fbcdf-n4st4" event={"ID":"b69d99f5-8b53-47cf-8a76-917f606488b9","Type":"ContainerDied","Data":"5bb18eb75a166b592fceed1afd3b2495f1761ce94e331d60827b2098f94eeecf"} Nov 02 17:34:14.481388 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.481365 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b083135-c135-437c-9446-5267ab9cb71f","Type":"ContainerDied","Data":"33cc3c3636522c210a2c55e06391e20ffab4fd130b5a0973557180aa5e1650fb"} Nov 02 17:34:14.481498 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.481403 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.481498 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.481428 5549 scope.go:122] "RemoveContainer" containerID="a48c41142da65c840ebace4f02e42440fbc361d6df950be23f4b4220dcc5890f" Nov 02 17:34:14.489926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.489904 5549 scope.go:122] "RemoveContainer" containerID="e7b7ed98d053e64b85b97cea7ff349f2132a9a6eb40a4cd485466ae34a81fc3c" Nov 02 17:34:14.498089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.498071 5549 scope.go:122] "RemoveContainer" containerID="51ff36c3b5f28fc2eef6f476ee40097f0be3b41a5506e631d42efc3c56da1b64" Nov 02 17:34:14.503773 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.503748 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:34:14.506391 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.506371 5549 scope.go:122] "RemoveContainer" containerID="d1a1531f37c2c54e81a18a913cfdd1b61dc5031fa128c3c420e27d5e058cef36" Nov 02 17:34:14.508517 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.508499 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:34:14.513280 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.513250 5549 scope.go:122] "RemoveContainer" containerID="7f8bee42ef2baea4c975aa3066e5bbcc94be4a7e5bf918e440d9fe573893f266" Nov 02 17:34:14.520420 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.520400 5549 scope.go:122] "RemoveContainer" containerID="32100d40e27e116fa94ce3cd4f88a18c31d2f73730b501aae36e15387d8bd2fb" Nov 02 17:34:14.527413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.527394 5549 scope.go:122] "RemoveContainer" containerID="1e32330efff8fe8589858662cbf717d812c6465e667dcc1f671e29ea86034ab7" Nov 02 17:34:14.538427 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.538406 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:34:14.538720 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.538675 5549 cpu_manager.go:414] "containerMap: removing container" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="alertmanager" Nov 02 17:34:14.538720 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.538692 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="alertmanager" Nov 02 17:34:14.538720 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.538718 5549 cpu_manager.go:414] "containerMap: removing container" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="kube-rbac-proxy-metric" Nov 02 17:34:14.538933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.538729 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="kube-rbac-proxy-metric" Nov 02 17:34:14.538933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.538754 5549 cpu_manager.go:414] "containerMap: removing container" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="init-config-reloader" Nov 02 17:34:14.538933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.538761 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="init-config-reloader" Nov 02 17:34:14.538933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.538797 5549 cpu_manager.go:414] "containerMap: removing container" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="config-reloader" Nov 02 17:34:14.538933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.538806 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="config-reloader" Nov 02 17:34:14.538933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.538820 5549 cpu_manager.go:414] "containerMap: removing container" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="kube-rbac-proxy" Nov 02 17:34:14.538933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.538828 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="kube-rbac-proxy" Nov 02 17:34:14.538933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.538841 5549 cpu_manager.go:414] "containerMap: removing container" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="prom-label-proxy" Nov 02 17:34:14.538933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.538849 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="prom-label-proxy" Nov 02 17:34:14.538933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.538860 5549 cpu_manager.go:414] "containerMap: removing container" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="kube-rbac-proxy-web" Nov 02 17:34:14.538933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.538868 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="kube-rbac-proxy-web" Nov 02 17:34:14.539491 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.538996 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="kube-rbac-proxy-metric" Nov 02 17:34:14.539491 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.539017 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="init-config-reloader" Nov 02 17:34:14.539491 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.539026 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="config-reloader" Nov 02 17:34:14.539491 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.539036 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="kube-rbac-proxy" Nov 02 17:34:14.539491 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.539046 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="prom-label-proxy" Nov 02 17:34:14.539491 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.539054 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="kube-rbac-proxy-web" Nov 02 17:34:14.539491 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.539063 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b083135-c135-437c-9446-5267ab9cb71f" containerName="alertmanager" Nov 02 17:34:14.544573 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.544540 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.549064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.549006 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 17:34:14.549064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.549018 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 17:34:14.549064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.549004 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 17:34:14.549064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.549012 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 17:34:14.549064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.549032 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 17:34:14.549064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.549061 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 17:34:14.549415 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.549075 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 17:34:14.549415 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.549100 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gql28\"" Nov 02 17:34:14.549809 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.549781 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 17:34:14.553566 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.553545 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 17:34:14.560562 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.560542 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:34:14.574438 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.574392 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:34:14.739413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.739375 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-service-ca\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-service-ca\") pod \"b69d99f5-8b53-47cf-8a76-917f606488b9\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " Nov 02 17:34:14.739585 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.739426 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-oauth-config\") pod \"b69d99f5-8b53-47cf-8a76-917f606488b9\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " Nov 02 17:34:14.739585 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.739456 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-console-config\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-console-config\") pod \"b69d99f5-8b53-47cf-8a76-917f606488b9\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " Nov 02 17:34:14.739585 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.739493 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b69d99f5-8b53-47cf-8a76-917f606488b9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b69d99f5-8b53-47cf-8a76-917f606488b9-tmp\") pod \"b69d99f5-8b53-47cf-8a76-917f606488b9\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " Nov 02 17:34:14.739585 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.739527 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-trusted-ca-bundle\") pod \"b69d99f5-8b53-47cf-8a76-917f606488b9\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " Nov 02 17:34:14.739786 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.739612 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b69d99f5-8b53-47cf-8a76-917f606488b9-kube-api-access-pjxms\" (UniqueName: \"kubernetes.io/projected/b69d99f5-8b53-47cf-8a76-917f606488b9-kube-api-access-pjxms\") pod \"b69d99f5-8b53-47cf-8a76-917f606488b9\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " Nov 02 17:34:14.739786 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.739660 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-oauth-serving-cert\") pod \"b69d99f5-8b53-47cf-8a76-917f606488b9\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " Nov 02 17:34:14.739786 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.739689 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-serving-cert\") pod \"b69d99f5-8b53-47cf-8a76-917f606488b9\" (UID: \"b69d99f5-8b53-47cf-8a76-917f606488b9\") " Nov 02 17:34:14.739933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.739820 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.739933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.739846 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-config-out\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.739933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.739861 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b69d99f5-8b53-47cf-8a76-917f606488b9-tmp" pod "b69d99f5-8b53-47cf-8a76-917f606488b9" (UID: "b69d99f5-8b53-47cf-8a76-917f606488b9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:34:14.739933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.739878 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-service-ca" pod "b69d99f5-8b53-47cf-8a76-917f606488b9" (UID: "b69d99f5-8b53-47cf-8a76-917f606488b9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:14.739933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.739892 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.740210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.739948 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-trusted-ca-bundle" pod "b69d99f5-8b53-47cf-8a76-917f606488b9" (UID: "b69d99f5-8b53-47cf-8a76-917f606488b9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:14.740210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.739985 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-console-config" pod "b69d99f5-8b53-47cf-8a76-917f606488b9" (UID: "b69d99f5-8b53-47cf-8a76-917f606488b9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:14.740210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.740098 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.740210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.740123 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.740210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.740157 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.740210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.740197 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.740502 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.740254 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.740502 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.740330 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-oauth-serving-cert" pod "b69d99f5-8b53-47cf-8a76-917f606488b9" (UID: "b69d99f5-8b53-47cf-8a76-917f606488b9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:14.740502 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.740354 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.740502 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.740411 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-web-config\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.740502 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.740462 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-config-volume\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.740502 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.740478 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.740502 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.740495 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfgnw\" (UniqueName: \"kubernetes.io/projected/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-kube-api-access-kfgnw\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.740853 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.740551 5549 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b69d99f5-8b53-47cf-8a76-917f606488b9-tmp\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:14.740853 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.740569 5549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-trusted-ca-bundle\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:14.740853 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.740580 5549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-oauth-serving-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:14.740853 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.740589 5549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:14.740853 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.740598 5549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b69d99f5-8b53-47cf-8a76-917f606488b9-console-config\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:14.742253 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.742226 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-oauth-config" pod "b69d99f5-8b53-47cf-8a76-917f606488b9" (UID: "b69d99f5-8b53-47cf-8a76-917f606488b9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:14.742350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.742296 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-serving-cert" pod "b69d99f5-8b53-47cf-8a76-917f606488b9" (UID: "b69d99f5-8b53-47cf-8a76-917f606488b9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:14.742425 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.742402 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69d99f5-8b53-47cf-8a76-917f606488b9-kube-api-access-pjxms" pod "b69d99f5-8b53-47cf-8a76-917f606488b9" (UID: "b69d99f5-8b53-47cf-8a76-917f606488b9"). InnerVolumeSpecName "kube-api-access-pjxms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:14.841767 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.841675 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.841767 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.841715 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.841767 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.841738 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.841767 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.841766 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.842119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.841805 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-web-config\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.842119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.841943 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-config-volume\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.842119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.841998 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.842119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.842024 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfgnw\" (UniqueName: \"kubernetes.io/projected/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-kube-api-access-kfgnw\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.842119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.842066 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.842119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.842089 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-config-out\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.842409 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.842130 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.842409 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.842186 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.842409 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.842214 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.842409 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.842277 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjxms\" (UniqueName: \"kubernetes.io/projected/b69d99f5-8b53-47cf-8a76-917f606488b9-kube-api-access-pjxms\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:14.842409 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.842296 5549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-serving-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:14.842409 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.842311 5549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b69d99f5-8b53-47cf-8a76-917f606488b9-console-oauth-config\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:14.842701 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.842531 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.842701 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.842658 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.843059 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.842997 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.844936 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.844905 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-web-config\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.845081 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.845045 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.845172 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.845059 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.845172 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.845119 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-config-volume\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.845172 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.845159 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.845328 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.845243 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-config-out\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.845552 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.845532 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.845642 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.845624 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.846755 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.846739 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.851207 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.851191 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfgnw\" (UniqueName: \"kubernetes.io/projected/cad2499d-dbf4-443c-ae65-4da5b6bba6f1-kube-api-access-kfgnw\") pod \"alertmanager-main-0\" (UID: \"cad2499d-dbf4-443c-ae65-4da5b6bba6f1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:14.882045 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:14.882020 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:34:15.162553 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.162526 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:34:15.164612 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:34:15.164587 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad2499d_dbf4_443c_ae65_4da5b6bba6f1.slice/crio-b919dd8fde1d32658f7ed73ab178cda5bac4389d702459224422e40e265b4936 WatchSource:0}: Error finding container b919dd8fde1d32658f7ed73ab178cda5bac4389d702459224422e40e265b4936: Status 404 returned error can't find the container with id b919dd8fde1d32658f7ed73ab178cda5bac4389d702459224422e40e265b4936 Nov 02 17:34:15.399336 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.399295 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5997464558-6x2v8" podUID="53eac522-fee5-4865-955b-a14da456d693" containerName="console" containerID="cri-o://598fc09e0cec98cce0d54ddd1af221d7293a81a36e99723e74fbc32c33e6de51" gracePeriod=15 Nov 02 17:34:15.486076 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.486040 5549 generic.go:356] "Generic (PLEG): container finished" podID="cad2499d-dbf4-443c-ae65-4da5b6bba6f1" containerID="ff39628a5e9f4bb9c003a387de92b8e5b3c329e874cb33da647c83288f6cdc5a" exitCode=0 Nov 02 17:34:15.486480 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.486129 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cad2499d-dbf4-443c-ae65-4da5b6bba6f1","Type":"ContainerDied","Data":"ff39628a5e9f4bb9c003a387de92b8e5b3c329e874cb33da647c83288f6cdc5a"} Nov 02 17:34:15.486480 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.486214 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cad2499d-dbf4-443c-ae65-4da5b6bba6f1","Type":"ContainerStarted","Data":"b919dd8fde1d32658f7ed73ab178cda5bac4389d702459224422e40e265b4936"} Nov 02 17:34:15.487605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.487578 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-558b6fbcdf-n4st4" Nov 02 17:34:15.487706 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.487604 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-558b6fbcdf-n4st4" event={"ID":"b69d99f5-8b53-47cf-8a76-917f606488b9","Type":"ContainerDied","Data":"2ce00f5670141c9170eeeaf58029aaa782c2afbb5545da08b06be9b391dbb4e7"} Nov 02 17:34:15.487706 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.487658 5549 scope.go:122] "RemoveContainer" containerID="5bb18eb75a166b592fceed1afd3b2495f1761ce94e331d60827b2098f94eeecf" Nov 02 17:34:15.525060 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.525035 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-558b6fbcdf-n4st4"] Nov 02 17:34:15.531478 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.531445 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-558b6fbcdf-n4st4"] Nov 02 17:34:15.647861 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.647838 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:34:15.748981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.748505 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-console-config\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-console-config\") pod \"53eac522-fee5-4865-955b-a14da456d693\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " Nov 02 17:34:15.748981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.748567 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-service-ca\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-service-ca\") pod \"53eac522-fee5-4865-955b-a14da456d693\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " Nov 02 17:34:15.748981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.748602 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53eac522-fee5-4865-955b-a14da456d693-kube-api-access-hqx5s\" (UniqueName: \"kubernetes.io/projected/53eac522-fee5-4865-955b-a14da456d693-kube-api-access-hqx5s\") pod \"53eac522-fee5-4865-955b-a14da456d693\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " Nov 02 17:34:15.748981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.748643 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53eac522-fee5-4865-955b-a14da456d693-tmp\" (UniqueName: \"kubernetes.io/empty-dir/53eac522-fee5-4865-955b-a14da456d693-tmp\") pod \"53eac522-fee5-4865-955b-a14da456d693\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " Nov 02 17:34:15.748981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.748664 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-serving-cert\") pod \"53eac522-fee5-4865-955b-a14da456d693\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " Nov 02 17:34:15.748981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.748686 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-trusted-ca-bundle\") pod \"53eac522-fee5-4865-955b-a14da456d693\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " Nov 02 17:34:15.748981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.748708 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-oauth-config\") pod \"53eac522-fee5-4865-955b-a14da456d693\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " Nov 02 17:34:15.748981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.748727 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-oauth-serving-cert\") pod \"53eac522-fee5-4865-955b-a14da456d693\" (UID: \"53eac522-fee5-4865-955b-a14da456d693\") " Nov 02 17:34:15.748981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.748912 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-console-config" pod "53eac522-fee5-4865-955b-a14da456d693" (UID: "53eac522-fee5-4865-955b-a14da456d693"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:15.749459 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.749310 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53eac522-fee5-4865-955b-a14da456d693-tmp" pod "53eac522-fee5-4865-955b-a14da456d693" (UID: "53eac522-fee5-4865-955b-a14da456d693"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:34:15.749459 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.749394 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-service-ca" pod "53eac522-fee5-4865-955b-a14da456d693" (UID: "53eac522-fee5-4865-955b-a14da456d693"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:15.749459 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.749347 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-trusted-ca-bundle" pod "53eac522-fee5-4865-955b-a14da456d693" (UID: "53eac522-fee5-4865-955b-a14da456d693"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:15.749687 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.749522 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-oauth-serving-cert" pod "53eac522-fee5-4865-955b-a14da456d693" (UID: "53eac522-fee5-4865-955b-a14da456d693"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:15.751217 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.751190 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-oauth-config" pod "53eac522-fee5-4865-955b-a14da456d693" (UID: "53eac522-fee5-4865-955b-a14da456d693"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:15.751328 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.751302 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-serving-cert" pod "53eac522-fee5-4865-955b-a14da456d693" (UID: "53eac522-fee5-4865-955b-a14da456d693"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:15.751677 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.751656 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53eac522-fee5-4865-955b-a14da456d693-kube-api-access-hqx5s" pod "53eac522-fee5-4865-955b-a14da456d693" (UID: "53eac522-fee5-4865-955b-a14da456d693"). InnerVolumeSpecName "kube-api-access-hqx5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:15.849899 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.849875 5549 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53eac522-fee5-4865-955b-a14da456d693-tmp\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:15.849899 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.849899 5549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-serving-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:15.850070 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.849911 5549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-trusted-ca-bundle\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:15.850070 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.849919 5549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53eac522-fee5-4865-955b-a14da456d693-console-oauth-config\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:15.850070 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.849929 5549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-oauth-serving-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:15.850070 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.849942 5549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-console-config\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:15.850070 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.849971 5549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53eac522-fee5-4865-955b-a14da456d693-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:15.850070 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:15.849982 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqx5s\" (UniqueName: \"kubernetes.io/projected/53eac522-fee5-4865-955b-a14da456d693-kube-api-access-hqx5s\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:16.317101 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:16.317067 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b083135-c135-437c-9446-5267ab9cb71f" path="/var/lib/kubelet/pods/1b083135-c135-437c-9446-5267ab9cb71f/volumes" Nov 02 17:34:16.317754 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:16.317739 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b69d99f5-8b53-47cf-8a76-917f606488b9" path="/var/lib/kubelet/pods/b69d99f5-8b53-47cf-8a76-917f606488b9/volumes" Nov 02 17:34:16.493145 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:16.493110 5549 generic.go:356] "Generic (PLEG): container finished" podID="53eac522-fee5-4865-955b-a14da456d693" containerID="598fc09e0cec98cce0d54ddd1af221d7293a81a36e99723e74fbc32c33e6de51" exitCode=0 Nov 02 17:34:16.493635 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:16.493171 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5997464558-6x2v8" Nov 02 17:34:16.493635 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:16.493225 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5997464558-6x2v8" event={"ID":"53eac522-fee5-4865-955b-a14da456d693","Type":"ContainerDied","Data":"598fc09e0cec98cce0d54ddd1af221d7293a81a36e99723e74fbc32c33e6de51"} Nov 02 17:34:16.493635 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:16.493274 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5997464558-6x2v8" event={"ID":"53eac522-fee5-4865-955b-a14da456d693","Type":"ContainerDied","Data":"5a5511f9cffc9fac7dd2b70f2be71a0228fca3de17c8f3f98cd11efcbcd93c3e"} Nov 02 17:34:16.493635 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:16.493303 5549 scope.go:122] "RemoveContainer" containerID="598fc09e0cec98cce0d54ddd1af221d7293a81a36e99723e74fbc32c33e6de51" Nov 02 17:34:16.498482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:16.498457 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cad2499d-dbf4-443c-ae65-4da5b6bba6f1","Type":"ContainerStarted","Data":"59a38cdbbba05b645c9d4755d50c59240307e2ba7817f9c864f83de83e9af510"} Nov 02 17:34:16.498622 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:16.498519 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cad2499d-dbf4-443c-ae65-4da5b6bba6f1","Type":"ContainerStarted","Data":"f3ff40565499f61ab99176a88cac003191c47fa282cf4587c167b1cacac42dde"} Nov 02 17:34:16.498622 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:16.498572 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cad2499d-dbf4-443c-ae65-4da5b6bba6f1","Type":"ContainerStarted","Data":"8dc50e00576421940830732e2563e8652fb9676d9d1e52e3f8a947030e3335ec"} Nov 02 17:34:16.498622 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:16.498613 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cad2499d-dbf4-443c-ae65-4da5b6bba6f1","Type":"ContainerStarted","Data":"24aa9c8a57cb677211a17f264568b986925481d46c26a7fb33fb2d71fbd54400"} Nov 02 17:34:16.498783 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:16.498650 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cad2499d-dbf4-443c-ae65-4da5b6bba6f1","Type":"ContainerStarted","Data":"5fc5274503c768e780f23904642788d9051a5aed10c7d594d63614858161a462"} Nov 02 17:34:16.498783 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:16.498693 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cad2499d-dbf4-443c-ae65-4da5b6bba6f1","Type":"ContainerStarted","Data":"f640ae923ff076104c95c51e5d221581f4c2757e9bb8e1885464c074b6eac1be"} Nov 02 17:34:16.502401 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:16.502380 5549 scope.go:122] "RemoveContainer" containerID="598fc09e0cec98cce0d54ddd1af221d7293a81a36e99723e74fbc32c33e6de51" Nov 02 17:34:16.502662 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:34:16.502640 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"598fc09e0cec98cce0d54ddd1af221d7293a81a36e99723e74fbc32c33e6de51\": container with ID starting with 598fc09e0cec98cce0d54ddd1af221d7293a81a36e99723e74fbc32c33e6de51 not found: ID does not exist" containerID="598fc09e0cec98cce0d54ddd1af221d7293a81a36e99723e74fbc32c33e6de51" Nov 02 17:34:16.502743 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:16.502670 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"598fc09e0cec98cce0d54ddd1af221d7293a81a36e99723e74fbc32c33e6de51"} err="failed to get container status \"598fc09e0cec98cce0d54ddd1af221d7293a81a36e99723e74fbc32c33e6de51\": rpc error: code = NotFound desc = could not find container \"598fc09e0cec98cce0d54ddd1af221d7293a81a36e99723e74fbc32c33e6de51\": container with ID starting with 598fc09e0cec98cce0d54ddd1af221d7293a81a36e99723e74fbc32c33e6de51 not found: ID does not exist" Nov 02 17:34:16.513631 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:16.513476 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5997464558-6x2v8"] Nov 02 17:34:16.515726 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:16.515703 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5997464558-6x2v8"] Nov 02 17:34:16.540542 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:16.540492 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.529449307 podStartE2EDuration="2.540481381s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.032074ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:16.539176461 +0000 UTC m=+28.327681990" watchObservedRunningTime="2026-11-02 17:34:16.540481381 +0000 UTC m=+28.328986945" Nov 02 17:34:17.584661 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:34:17.584620 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:34:18.316729 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:18.316693 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53eac522-fee5-4865-955b-a14da456d693" path="/var/lib/kubelet/pods/53eac522-fee5-4865-955b-a14da456d693/volumes" Nov 02 17:34:18.878770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:18.878731 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:18.879259 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:18.878794 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:18.883429 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:18.883409 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:19.512643 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:19.512613 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9456bdf8d-wxx6s" Nov 02 17:34:19.562459 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:19.562429 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7979dc65db-56v4t"] Nov 02 17:34:20.347165 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:20.347131 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:34:20.484324 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:20.484294 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:34:25.932376 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:25.932346 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lnbcn" Nov 02 17:34:29.484010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:29.483971 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:34:29.620429 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:29.620392 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:34:44.581672 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.581610 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7979dc65db-56v4t" podUID="c40f5b0d-64b1-4e58-afec-4dc4b19363d1" containerName="console" containerID="cri-o://71381505bc2c86e03f375a654de061a5de203d0d9af8823982362c1d7a3e01b1" gracePeriod=15 Nov 02 17:34:44.814752 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.814728 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:34:44.858395 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.858340 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-tmp\") pod \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " Nov 02 17:34:44.858497 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.858396 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-oauth-serving-cert\") pod \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " Nov 02 17:34:44.858497 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.858431 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-serving-cert\") pod \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " Nov 02 17:34:44.858497 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.858462 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-oauth-config\") pod \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " Nov 02 17:34:44.858694 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.858502 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-trusted-ca-bundle\") pod \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " Nov 02 17:34:44.858694 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.858604 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-service-ca\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-service-ca\") pod \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " Nov 02 17:34:44.858694 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.858607 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-tmp" pod "c40f5b0d-64b1-4e58-afec-4dc4b19363d1" (UID: "c40f5b0d-64b1-4e58-afec-4dc4b19363d1"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:34:44.858694 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.858669 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-config\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-config\") pod \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " Nov 02 17:34:44.858883 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.858722 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-kube-api-access-k229c\" (UniqueName: \"kubernetes.io/projected/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-kube-api-access-k229c\") pod \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\" (UID: \"c40f5b0d-64b1-4e58-afec-4dc4b19363d1\") " Nov 02 17:34:44.858932 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.858910 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-oauth-serving-cert" pod "c40f5b0d-64b1-4e58-afec-4dc4b19363d1" (UID: "c40f5b0d-64b1-4e58-afec-4dc4b19363d1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:44.859012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.858985 5549 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-tmp\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:44.859012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.858983 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-trusted-ca-bundle" pod "c40f5b0d-64b1-4e58-afec-4dc4b19363d1" (UID: "c40f5b0d-64b1-4e58-afec-4dc4b19363d1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:44.859116 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.859075 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-config" pod "c40f5b0d-64b1-4e58-afec-4dc4b19363d1" (UID: "c40f5b0d-64b1-4e58-afec-4dc4b19363d1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:44.859116 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.859101 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-service-ca" pod "c40f5b0d-64b1-4e58-afec-4dc4b19363d1" (UID: "c40f5b0d-64b1-4e58-afec-4dc4b19363d1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:44.860904 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.860879 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-kube-api-access-k229c" pod "c40f5b0d-64b1-4e58-afec-4dc4b19363d1" (UID: "c40f5b0d-64b1-4e58-afec-4dc4b19363d1"). InnerVolumeSpecName "kube-api-access-k229c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:44.861081 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.861055 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-oauth-config" pod "c40f5b0d-64b1-4e58-afec-4dc4b19363d1" (UID: "c40f5b0d-64b1-4e58-afec-4dc4b19363d1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:44.861180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.861123 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-serving-cert" pod "c40f5b0d-64b1-4e58-afec-4dc4b19363d1" (UID: "c40f5b0d-64b1-4e58-afec-4dc4b19363d1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:44.960185 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.960152 5549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-trusted-ca-bundle\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:44.960185 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.960182 5549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:44.960369 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.960195 5549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-config\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:44.960369 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.960209 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k229c\" (UniqueName: \"kubernetes.io/projected/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-kube-api-access-k229c\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:44.960369 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.960223 5549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-oauth-serving-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:44.960369 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.960236 5549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-serving-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:44.960369 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:44.960248 5549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c40f5b0d-64b1-4e58-afec-4dc4b19363d1-console-oauth-config\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:45.595387 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:45.595352 5549 generic.go:356] "Generic (PLEG): container finished" podID="c40f5b0d-64b1-4e58-afec-4dc4b19363d1" containerID="71381505bc2c86e03f375a654de061a5de203d0d9af8823982362c1d7a3e01b1" exitCode=0 Nov 02 17:34:45.595788 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:45.595402 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7979dc65db-56v4t" Nov 02 17:34:45.595788 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:45.595436 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7979dc65db-56v4t" event={"ID":"c40f5b0d-64b1-4e58-afec-4dc4b19363d1","Type":"ContainerDied","Data":"71381505bc2c86e03f375a654de061a5de203d0d9af8823982362c1d7a3e01b1"} Nov 02 17:34:45.595788 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:45.595475 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7979dc65db-56v4t" event={"ID":"c40f5b0d-64b1-4e58-afec-4dc4b19363d1","Type":"ContainerDied","Data":"f86bb36cade1759273de3d44e951ef8f897a140526e207664028c81fa9128f21"} Nov 02 17:34:45.595788 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:45.595495 5549 scope.go:122] "RemoveContainer" containerID="71381505bc2c86e03f375a654de061a5de203d0d9af8823982362c1d7a3e01b1" Nov 02 17:34:45.603223 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:45.603203 5549 scope.go:122] "RemoveContainer" containerID="71381505bc2c86e03f375a654de061a5de203d0d9af8823982362c1d7a3e01b1" Nov 02 17:34:45.603523 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:34:45.603504 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71381505bc2c86e03f375a654de061a5de203d0d9af8823982362c1d7a3e01b1\": container with ID starting with 71381505bc2c86e03f375a654de061a5de203d0d9af8823982362c1d7a3e01b1 not found: ID does not exist" containerID="71381505bc2c86e03f375a654de061a5de203d0d9af8823982362c1d7a3e01b1" Nov 02 17:34:45.603576 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:45.603532 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71381505bc2c86e03f375a654de061a5de203d0d9af8823982362c1d7a3e01b1"} err="failed to get container status \"71381505bc2c86e03f375a654de061a5de203d0d9af8823982362c1d7a3e01b1\": rpc error: code = NotFound desc = could not find container \"71381505bc2c86e03f375a654de061a5de203d0d9af8823982362c1d7a3e01b1\": container with ID starting with 71381505bc2c86e03f375a654de061a5de203d0d9af8823982362c1d7a3e01b1 not found: ID does not exist" Nov 02 17:34:45.619249 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:45.619228 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7979dc65db-56v4t"] Nov 02 17:34:45.622844 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:45.622826 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7979dc65db-56v4t"] Nov 02 17:34:46.315770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:46.315734 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c40f5b0d-64b1-4e58-afec-4dc4b19363d1" path="/var/lib/kubelet/pods/c40f5b0d-64b1-4e58-afec-4dc4b19363d1/volumes" Nov 02 17:34:47.584256 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:34:47.584219 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:34:57.143235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:34:57.143202 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:35:02.241374 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:02.241337 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:35:17.584330 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:35:17.584284 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:35:44.074344 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:44.074261 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:35:44.279452 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:44.279416 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:35:44.471372 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:44.471337 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:35:45.724708 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.724678 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9"] Nov 02 17:35:45.725086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.724927 5549 cpu_manager.go:414] "containerMap: removing container" podUID="b69d99f5-8b53-47cf-8a76-917f606488b9" containerName="console" Nov 02 17:35:45.725086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.724937 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b69d99f5-8b53-47cf-8a76-917f606488b9" containerName="console" Nov 02 17:35:45.725086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.724982 5549 cpu_manager.go:414] "containerMap: removing container" podUID="c40f5b0d-64b1-4e58-afec-4dc4b19363d1" containerName="console" Nov 02 17:35:45.725086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.724990 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40f5b0d-64b1-4e58-afec-4dc4b19363d1" containerName="console" Nov 02 17:35:45.725086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.725008 5549 cpu_manager.go:414] "containerMap: removing container" podUID="53eac522-fee5-4865-955b-a14da456d693" containerName="console" Nov 02 17:35:45.725086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.725013 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53eac522-fee5-4865-955b-a14da456d693" containerName="console" Nov 02 17:35:45.725086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.725056 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40f5b0d-64b1-4e58-afec-4dc4b19363d1" containerName="console" Nov 02 17:35:45.725086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.725066 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="53eac522-fee5-4865-955b-a14da456d693" containerName="console" Nov 02 17:35:45.725086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.725074 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="b69d99f5-8b53-47cf-8a76-917f606488b9" containerName="console" Nov 02 17:35:45.728031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.728014 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9" Nov 02 17:35:45.733624 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.733600 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-82dm4\"" Nov 02 17:35:45.738466 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.738443 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 17:35:45.738597 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.738582 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 17:35:45.748938 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.748918 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9"] Nov 02 17:35:45.815266 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.815234 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d036d91-8a12-4afd-9898-c78e517facc0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9\" (UID: \"2d036d91-8a12-4afd-9898-c78e517facc0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9" Nov 02 17:35:45.815447 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.815282 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fk76\" (UniqueName: \"kubernetes.io/projected/2d036d91-8a12-4afd-9898-c78e517facc0-kube-api-access-9fk76\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9\" (UID: \"2d036d91-8a12-4afd-9898-c78e517facc0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9" Nov 02 17:35:45.815447 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.815303 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d036d91-8a12-4afd-9898-c78e517facc0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9\" (UID: \"2d036d91-8a12-4afd-9898-c78e517facc0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9" Nov 02 17:35:45.829081 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.829056 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj"] Nov 02 17:35:45.832114 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.832092 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:45.839224 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.839204 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 17:35:45.839319 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.839262 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 17:35:45.839533 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.839514 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 17:35:45.839874 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.839545 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 17:35:45.839967 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.839893 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 17:35:45.840133 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.840111 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-phftv\"" Nov 02 17:35:45.844268 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.844249 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 17:35:45.848195 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.848174 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj"] Nov 02 17:35:45.912998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.912945 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:35:45.916014 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.915994 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fk76\" (UniqueName: \"kubernetes.io/projected/2d036d91-8a12-4afd-9898-c78e517facc0-kube-api-access-9fk76\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9\" (UID: \"2d036d91-8a12-4afd-9898-c78e517facc0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9" Nov 02 17:35:45.916095 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.916025 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d036d91-8a12-4afd-9898-c78e517facc0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9\" (UID: \"2d036d91-8a12-4afd-9898-c78e517facc0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9" Nov 02 17:35:45.916095 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.916049 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:45.916095 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.916075 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:45.916222 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.916100 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:45.916222 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.916145 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0234efe0-dffb-4cae-8086-7ec56cc4e950-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:45.916222 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.916213 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:45.916347 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.916234 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:45.916347 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.916283 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d036d91-8a12-4afd-9898-c78e517facc0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9\" (UID: \"2d036d91-8a12-4afd-9898-c78e517facc0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9" Nov 02 17:35:45.916433 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.916354 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwd4l\" (UniqueName: \"kubernetes.io/projected/0234efe0-dffb-4cae-8086-7ec56cc4e950-kube-api-access-dwd4l\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:45.916433 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.916368 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d036d91-8a12-4afd-9898-c78e517facc0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9\" (UID: \"2d036d91-8a12-4afd-9898-c78e517facc0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9" Nov 02 17:35:45.916582 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.916564 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d036d91-8a12-4afd-9898-c78e517facc0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9\" (UID: \"2d036d91-8a12-4afd-9898-c78e517facc0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9" Nov 02 17:35:45.930346 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:45.930322 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fk76\" (UniqueName: \"kubernetes.io/projected/2d036d91-8a12-4afd-9898-c78e517facc0-kube-api-access-9fk76\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9\" (UID: \"2d036d91-8a12-4afd-9898-c78e517facc0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9" Nov 02 17:35:46.017561 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.017480 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwd4l\" (UniqueName: \"kubernetes.io/projected/0234efe0-dffb-4cae-8086-7ec56cc4e950-kube-api-access-dwd4l\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:46.017561 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.017542 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:46.017561 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.017566 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:46.017808 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.017583 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:46.017808 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.017598 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0234efe0-dffb-4cae-8086-7ec56cc4e950-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:46.017808 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.017617 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:46.017808 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.017643 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:46.018341 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.018313 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:46.020054 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.020031 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0234efe0-dffb-4cae-8086-7ec56cc4e950-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:46.020540 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.020184 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:46.020624 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.020459 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:46.020624 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.020470 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:46.029285 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.029261 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwd4l\" (UniqueName: \"kubernetes.io/projected/0234efe0-dffb-4cae-8086-7ec56cc4e950-kube-api-access-dwd4l\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:46.029435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.029414 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-p2wlj\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:46.043227 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.043205 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9" Nov 02 17:35:46.083678 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.083647 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:35:46.143636 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.143608 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:46.316621 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.316594 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9"] Nov 02 17:35:46.317320 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:35:46.317284 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d036d91_8a12_4afd_9898_c78e517facc0.slice/crio-eb176f602adeb5f272f1ed0623a45bde38bac7a8274c447f29b585bb76d797c7 WatchSource:0}: Error finding container eb176f602adeb5f272f1ed0623a45bde38bac7a8274c447f29b585bb76d797c7: Status 404 returned error can't find the container with id eb176f602adeb5f272f1ed0623a45bde38bac7a8274c447f29b585bb76d797c7 Nov 02 17:35:46.319105 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.319086 5549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:35:46.414542 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.414511 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj"] Nov 02 17:35:46.417581 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:35:46.417552 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0234efe0_dffb_4cae_8086_7ec56cc4e950.slice/crio-0d793a09e99be8db2e94749aa4b3f1e2105cc0c544f80be13af04cd58f0a357f WatchSource:0}: Error finding container 0d793a09e99be8db2e94749aa4b3f1e2105cc0c544f80be13af04cd58f0a357f: Status 404 returned error can't find the container with id 0d793a09e99be8db2e94749aa4b3f1e2105cc0c544f80be13af04cd58f0a357f Nov 02 17:35:46.789762 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.789721 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" event={"ID":"0234efe0-dffb-4cae-8086-7ec56cc4e950","Type":"ContainerStarted","Data":"0d793a09e99be8db2e94749aa4b3f1e2105cc0c544f80be13af04cd58f0a357f"} Nov 02 17:35:46.790751 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:46.790721 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9" event={"ID":"2d036d91-8a12-4afd-9898-c78e517facc0","Type":"ContainerStarted","Data":"eb176f602adeb5f272f1ed0623a45bde38bac7a8274c447f29b585bb76d797c7"} Nov 02 17:35:47.526847 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:47.526806 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:35:47.589719 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:35:47.589681 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:35:50.138881 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:50.138839 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:35:50.139292 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:50.138915 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:35:50.808782 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:50.808740 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" event={"ID":"0234efe0-dffb-4cae-8086-7ec56cc4e950","Type":"ContainerStarted","Data":"f2e998415ad7ea67f4db63488114ff3d59bd6c58f3a54e9c100efc877f472a5d"} Nov 02 17:35:50.808940 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:50.808863 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:50.830156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:50.830102 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:35:51.011314 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:51.011279 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:35:51.721253 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:51.721215 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:35:51.814579 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:51.814539 5549 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-p2wlj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:35:51.814755 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:51.814600 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" podUID="0234efe0-dffb-4cae-8086-7ec56cc4e950" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:35:51.876172 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:51.876136 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:35:52.816900 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:52.816862 5549 generic.go:356] "Generic (PLEG): container finished" podID="2d036d91-8a12-4afd-9898-c78e517facc0" containerID="7f6c75f7640d73f642824f3e8b1fe119c6f04654cde10d19cc4d235e461c0833" exitCode=0 Nov 02 17:35:52.817334 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:52.816966 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9" event={"ID":"2d036d91-8a12-4afd-9898-c78e517facc0","Type":"ContainerDied","Data":"7f6c75f7640d73f642824f3e8b1fe119c6f04654cde10d19cc4d235e461c0833"} Nov 02 17:35:52.834943 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:52.834895 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" podStartSLOduration=4.116362446 podStartE2EDuration="7.834883124s" totalImagesPullingTime="3.718520678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:50.834997533 +0000 UTC m=+122.623503083" watchObservedRunningTime="2026-11-02 17:35:52.834883124 +0000 UTC m=+124.623388651" Nov 02 17:35:54.564126 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:54.564090 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:35:54.814168 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:54.814098 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:35:55.829672 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:55.829631 5549 generic.go:356] "Generic (PLEG): container finished" podID="2d036d91-8a12-4afd-9898-c78e517facc0" containerID="c9a771b0ef0b120232248c102254a6cc01c49d7af1583ff7b102a541bc58f34b" exitCode=0 Nov 02 17:35:55.830094 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:55.829717 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9" event={"ID":"2d036d91-8a12-4afd-9898-c78e517facc0","Type":"ContainerDied","Data":"c9a771b0ef0b120232248c102254a6cc01c49d7af1583ff7b102a541bc58f34b"} Nov 02 17:35:57.239448 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:35:57.239412 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:02.857809 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:02.857771 5549 generic.go:356] "Generic (PLEG): container finished" podID="2d036d91-8a12-4afd-9898-c78e517facc0" containerID="9d1cf867d40242c0768811c4c3c13ab643575e5290f2eec9bb2b9596e965a056" exitCode=0 Nov 02 17:36:02.858289 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:02.857849 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9" event={"ID":"2d036d91-8a12-4afd-9898-c78e517facc0","Type":"ContainerDied","Data":"9d1cf867d40242c0768811c4c3c13ab643575e5290f2eec9bb2b9596e965a056"} Nov 02 17:36:03.981848 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:03.981819 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9" Nov 02 17:36:04.073635 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:04.073601 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d036d91-8a12-4afd-9898-c78e517facc0-kube-api-access-9fk76\" (UniqueName: \"kubernetes.io/projected/2d036d91-8a12-4afd-9898-c78e517facc0-kube-api-access-9fk76\") pod \"2d036d91-8a12-4afd-9898-c78e517facc0\" (UID: \"2d036d91-8a12-4afd-9898-c78e517facc0\") " Nov 02 17:36:04.073812 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:04.073672 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d036d91-8a12-4afd-9898-c78e517facc0-util\" (UniqueName: \"kubernetes.io/empty-dir/2d036d91-8a12-4afd-9898-c78e517facc0-util\") pod \"2d036d91-8a12-4afd-9898-c78e517facc0\" (UID: \"2d036d91-8a12-4afd-9898-c78e517facc0\") " Nov 02 17:36:04.073812 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:04.073710 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d036d91-8a12-4afd-9898-c78e517facc0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d036d91-8a12-4afd-9898-c78e517facc0-bundle\") pod \"2d036d91-8a12-4afd-9898-c78e517facc0\" (UID: \"2d036d91-8a12-4afd-9898-c78e517facc0\") " Nov 02 17:36:04.074319 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:04.074287 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d036d91-8a12-4afd-9898-c78e517facc0-bundle" pod "2d036d91-8a12-4afd-9898-c78e517facc0" (UID: "2d036d91-8a12-4afd-9898-c78e517facc0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:36:04.075849 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:04.075825 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d036d91-8a12-4afd-9898-c78e517facc0-kube-api-access-9fk76" pod "2d036d91-8a12-4afd-9898-c78e517facc0" (UID: "2d036d91-8a12-4afd-9898-c78e517facc0"). InnerVolumeSpecName "kube-api-access-9fk76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:36:04.078020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:04.078000 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d036d91-8a12-4afd-9898-c78e517facc0-util" pod "2d036d91-8a12-4afd-9898-c78e517facc0" (UID: "2d036d91-8a12-4afd-9898-c78e517facc0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:36:04.174915 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:04.174833 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fk76\" (UniqueName: \"kubernetes.io/projected/2d036d91-8a12-4afd-9898-c78e517facc0-kube-api-access-9fk76\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:04.174915 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:04.174865 5549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d036d91-8a12-4afd-9898-c78e517facc0-util\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:04.174915 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:04.174874 5549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d036d91-8a12-4afd-9898-c78e517facc0-bundle\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:04.868099 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:04.868063 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9" event={"ID":"2d036d91-8a12-4afd-9898-c78e517facc0","Type":"ContainerDied","Data":"eb176f602adeb5f272f1ed0623a45bde38bac7a8274c447f29b585bb76d797c7"} Nov 02 17:36:04.868276 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:04.868116 5549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb176f602adeb5f272f1ed0623a45bde38bac7a8274c447f29b585bb76d797c7" Nov 02 17:36:04.868276 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:04.868160 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6x7l9" Nov 02 17:36:08.723406 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:08.723378 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:08.878033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:08.877993 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:13.447328 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:13.447295 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:14.382082 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:14.382047 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:17.584860 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:36:17.584822 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:36:17.839491 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:17.839404 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:18.103434 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:18.103354 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:18.277123 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:18.277089 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:18.651156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:18.651123 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:18.827935 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:18.827905 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:19.067900 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.067860 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:19.163153 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.163120 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9"] Nov 02 17:36:19.163486 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.163472 5549 cpu_manager.go:414] "containerMap: removing container" podUID="2d036d91-8a12-4afd-9898-c78e517facc0" containerName="util" Nov 02 17:36:19.163548 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.163487 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d036d91-8a12-4afd-9898-c78e517facc0" containerName="util" Nov 02 17:36:19.163548 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.163511 5549 cpu_manager.go:414] "containerMap: removing container" podUID="2d036d91-8a12-4afd-9898-c78e517facc0" containerName="extract" Nov 02 17:36:19.163548 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.163516 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d036d91-8a12-4afd-9898-c78e517facc0" containerName="extract" Nov 02 17:36:19.163548 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.163534 5549 cpu_manager.go:414] "containerMap: removing container" podUID="2d036d91-8a12-4afd-9898-c78e517facc0" containerName="pull" Nov 02 17:36:19.163548 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.163541 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d036d91-8a12-4afd-9898-c78e517facc0" containerName="pull" Nov 02 17:36:19.163762 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.163595 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d036d91-8a12-4afd-9898-c78e517facc0" containerName="extract" Nov 02 17:36:19.170040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.170017 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.181059 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.181040 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 17:36:19.181175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.181109 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 17:36:19.182053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.182036 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 17:36:19.182053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.182044 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 17:36:19.182175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.182054 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 17:36:19.182175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.182093 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-z8prg\"" Nov 02 17:36:19.198573 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.198545 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9"] Nov 02 17:36:19.292711 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.292678 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/edf532ca-42d5-45d5-a384-3c3633ad50b2-manifests\") pod \"opendatahub-operator-controller-manager-789486f7c8-qmfz9\" (UID: \"edf532ca-42d5-45d5-a384-3c3633ad50b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.292875 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.292718 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/edf532ca-42d5-45d5-a384-3c3633ad50b2-metrics-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-qmfz9\" (UID: \"edf532ca-42d5-45d5-a384-3c3633ad50b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.292875 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.292758 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/edf532ca-42d5-45d5-a384-3c3633ad50b2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-qmfz9\" (UID: \"edf532ca-42d5-45d5-a384-3c3633ad50b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.292984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.292909 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edf532ca-42d5-45d5-a384-3c3633ad50b2-webhook-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-qmfz9\" (UID: \"edf532ca-42d5-45d5-a384-3c3633ad50b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.293066 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.292980 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edf532ca-42d5-45d5-a384-3c3633ad50b2-tmp\") pod \"opendatahub-operator-controller-manager-789486f7c8-qmfz9\" (UID: \"edf532ca-42d5-45d5-a384-3c3633ad50b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.293123 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.293072 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj2xz\" (UniqueName: \"kubernetes.io/projected/edf532ca-42d5-45d5-a384-3c3633ad50b2-kube-api-access-jj2xz\") pod \"opendatahub-operator-controller-manager-789486f7c8-qmfz9\" (UID: \"edf532ca-42d5-45d5-a384-3c3633ad50b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.393658 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.393571 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edf532ca-42d5-45d5-a384-3c3633ad50b2-tmp\") pod \"opendatahub-operator-controller-manager-789486f7c8-qmfz9\" (UID: \"edf532ca-42d5-45d5-a384-3c3633ad50b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.393658 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.393615 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj2xz\" (UniqueName: \"kubernetes.io/projected/edf532ca-42d5-45d5-a384-3c3633ad50b2-kube-api-access-jj2xz\") pod \"opendatahub-operator-controller-manager-789486f7c8-qmfz9\" (UID: \"edf532ca-42d5-45d5-a384-3c3633ad50b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.393888 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.393746 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/edf532ca-42d5-45d5-a384-3c3633ad50b2-manifests\") pod \"opendatahub-operator-controller-manager-789486f7c8-qmfz9\" (UID: \"edf532ca-42d5-45d5-a384-3c3633ad50b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.393888 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.393795 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/edf532ca-42d5-45d5-a384-3c3633ad50b2-metrics-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-qmfz9\" (UID: \"edf532ca-42d5-45d5-a384-3c3633ad50b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.393888 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.393851 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/edf532ca-42d5-45d5-a384-3c3633ad50b2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-qmfz9\" (UID: \"edf532ca-42d5-45d5-a384-3c3633ad50b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.394080 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.393923 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edf532ca-42d5-45d5-a384-3c3633ad50b2-webhook-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-qmfz9\" (UID: \"edf532ca-42d5-45d5-a384-3c3633ad50b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.394171 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.394148 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/edf532ca-42d5-45d5-a384-3c3633ad50b2-manifests\") pod \"opendatahub-operator-controller-manager-789486f7c8-qmfz9\" (UID: \"edf532ca-42d5-45d5-a384-3c3633ad50b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.395877 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.395847 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edf532ca-42d5-45d5-a384-3c3633ad50b2-tmp\") pod \"opendatahub-operator-controller-manager-789486f7c8-qmfz9\" (UID: \"edf532ca-42d5-45d5-a384-3c3633ad50b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.396264 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.396243 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edf532ca-42d5-45d5-a384-3c3633ad50b2-webhook-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-qmfz9\" (UID: \"edf532ca-42d5-45d5-a384-3c3633ad50b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.396350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.396249 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/edf532ca-42d5-45d5-a384-3c3633ad50b2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-qmfz9\" (UID: \"edf532ca-42d5-45d5-a384-3c3633ad50b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.396390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.396365 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/edf532ca-42d5-45d5-a384-3c3633ad50b2-metrics-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-qmfz9\" (UID: \"edf532ca-42d5-45d5-a384-3c3633ad50b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.403929 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.403908 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj2xz\" (UniqueName: \"kubernetes.io/projected/edf532ca-42d5-45d5-a384-3c3633ad50b2-kube-api-access-jj2xz\") pod \"opendatahub-operator-controller-manager-789486f7c8-qmfz9\" (UID: \"edf532ca-42d5-45d5-a384-3c3633ad50b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.408344 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.408321 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:19.482642 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.482611 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:19.564544 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.564516 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:19.726107 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.726076 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:19.775110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.775080 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9"] Nov 02 17:36:19.779605 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:36:19.779578 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf532ca_42d5_45d5_a384_3c3633ad50b2.slice/crio-1c4902ceef9f4311a174af5fb8b4bd326e3fc410fcbaed95ffcb408c5badadfd WatchSource:0}: Error finding container 1c4902ceef9f4311a174af5fb8b4bd326e3fc410fcbaed95ffcb408c5badadfd: Status 404 returned error can't find the container with id 1c4902ceef9f4311a174af5fb8b4bd326e3fc410fcbaed95ffcb408c5badadfd Nov 02 17:36:19.883655 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.883624 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:19.919408 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:19.919376 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" event={"ID":"edf532ca-42d5-45d5-a384-3c3633ad50b2","Type":"ContainerStarted","Data":"1c4902ceef9f4311a174af5fb8b4bd326e3fc410fcbaed95ffcb408c5badadfd"} Nov 02 17:36:20.041470 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:20.041390 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:20.228328 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:20.228292 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:20.386665 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:20.386582 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:21.128149 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:21.128113 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:21.304291 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:21.304250 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:22.729483 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:22.729451 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:22.894711 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:22.894638 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:22.932674 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:22.932639 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" event={"ID":"edf532ca-42d5-45d5-a384-3c3633ad50b2","Type":"ContainerStarted","Data":"06e92c741a9e1fa29498b2a6b7d19611e0257f5b3a28480f129641c6ece5d801"} Nov 02 17:36:23.457094 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:23.457062 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:23.661824 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:23.661788 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:23.937364 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:23.937281 5549 generic.go:356] "Generic (PLEG): container finished" podID="edf532ca-42d5-45d5-a384-3c3633ad50b2" containerID="06e92c741a9e1fa29498b2a6b7d19611e0257f5b3a28480f129641c6ece5d801" exitCode=0 Nov 02 17:36:23.937364 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:23.937339 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" event={"ID":"edf532ca-42d5-45d5-a384-3c3633ad50b2","Type":"ContainerDied","Data":"06e92c741a9e1fa29498b2a6b7d19611e0257f5b3a28480f129641c6ece5d801"} Nov 02 17:36:24.942458 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:24.942368 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" event={"ID":"edf532ca-42d5-45d5-a384-3c3633ad50b2","Type":"ContainerStarted","Data":"dbadfbcc273aab468b8f8834ded5c583882222d332f1a09bad092f6287d23dd3"} Nov 02 17:36:24.942868 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:24.942521 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:24.983261 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:24.983212 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" podStartSLOduration=2.164570077 podStartE2EDuration="5.983197542s" totalImagesPullingTime="3.319073443s" totalInitContainerRuntime="499.554022ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:24.98186356 +0000 UTC m=+156.770369111" watchObservedRunningTime="2026-11-02 17:36:24.983197542 +0000 UTC m=+156.771703064" Nov 02 17:36:25.859144 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:25.859107 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:26.042550 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:26.042517 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:29.808545 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:29.808506 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:30.789260 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:30.789226 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:30.984518 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:30.984484 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:31.165295 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:31.165206 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:35.646718 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:35.646686 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:35.863656 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:35.863624 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:35.948090 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:35.948063 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-qmfz9" Nov 02 17:36:37.662559 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:37.662523 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:38.609006 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:38.608969 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:38.761279 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:38.761245 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:39.227194 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:39.227157 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:39.665080 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:39.664999 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:42.216310 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:42.216275 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:42.661710 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:42.661626 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:43.262185 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:43.262151 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:44.097092 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:44.097046 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:44.280667 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:44.280609 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:47.090615 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:47.090582 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:47.584120 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:36:47.584078 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:36:48.272461 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:48.272431 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:48.465523 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:48.465491 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:49.031474 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:49.031440 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:49.204111 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:49.204075 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:51.412936 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.412902 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:51.570534 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.570498 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:51.664775 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.664700 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-f74t5"] Nov 02 17:36:51.668477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.668458 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f74t5" Nov 02 17:36:51.677062 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.677036 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-f74t5"] Nov 02 17:36:51.677358 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.677339 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 17:36:51.677456 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.677344 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hhm7t\"" Nov 02 17:36:51.738763 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.738732 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:51.756012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.755980 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/db3845da-3e81-491c-9e2d-09f220ea1b9d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-f74t5\" (UID: \"db3845da-3e81-491c-9e2d-09f220ea1b9d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f74t5" Nov 02 17:36:51.756161 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.756035 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/db3845da-3e81-491c-9e2d-09f220ea1b9d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-f74t5\" (UID: \"db3845da-3e81-491c-9e2d-09f220ea1b9d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f74t5" Nov 02 17:36:51.756161 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.756144 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pztm\" (UniqueName: \"kubernetes.io/projected/db3845da-3e81-491c-9e2d-09f220ea1b9d-kube-api-access-9pztm\") pod \"kuadrant-console-plugin-66c4fb575d-f74t5\" (UID: \"db3845da-3e81-491c-9e2d-09f220ea1b9d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f74t5" Nov 02 17:36:51.767657 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.767628 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 17:36:51.767789 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.767684 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 17:36:51.767789 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.767723 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 17:36:51.856788 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.856752 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/db3845da-3e81-491c-9e2d-09f220ea1b9d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-f74t5\" (UID: \"db3845da-3e81-491c-9e2d-09f220ea1b9d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f74t5" Nov 02 17:36:51.856982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.856795 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/db3845da-3e81-491c-9e2d-09f220ea1b9d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-f74t5\" (UID: \"db3845da-3e81-491c-9e2d-09f220ea1b9d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f74t5" Nov 02 17:36:51.856982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.856843 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pztm\" (UniqueName: \"kubernetes.io/projected/db3845da-3e81-491c-9e2d-09f220ea1b9d-kube-api-access-9pztm\") pod \"kuadrant-console-plugin-66c4fb575d-f74t5\" (UID: \"db3845da-3e81-491c-9e2d-09f220ea1b9d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f74t5" Nov 02 17:36:51.857562 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.857540 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/db3845da-3e81-491c-9e2d-09f220ea1b9d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-f74t5\" (UID: \"db3845da-3e81-491c-9e2d-09f220ea1b9d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f74t5" Nov 02 17:36:51.859252 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.859228 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/db3845da-3e81-491c-9e2d-09f220ea1b9d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-f74t5\" (UID: \"db3845da-3e81-491c-9e2d-09f220ea1b9d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f74t5" Nov 02 17:36:51.867297 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.867271 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pztm\" (UniqueName: \"kubernetes.io/projected/db3845da-3e81-491c-9e2d-09f220ea1b9d-kube-api-access-9pztm\") pod \"kuadrant-console-plugin-66c4fb575d-f74t5\" (UID: \"db3845da-3e81-491c-9e2d-09f220ea1b9d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f74t5" Nov 02 17:36:51.928282 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.928180 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:51.992026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:51.991997 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f74t5" Nov 02 17:36:52.269302 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:52.269239 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-f74t5"] Nov 02 17:36:52.272691 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:36:52.272660 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb3845da_3e81_491c_9e2d_09f220ea1b9d.slice/crio-3218686dbf2cfd622fe1c9eed3065118fccf5f48c0f50cf366e4e2b73e2aac81 WatchSource:0}: Error finding container 3218686dbf2cfd622fe1c9eed3065118fccf5f48c0f50cf366e4e2b73e2aac81: Status 404 returned error can't find the container with id 3218686dbf2cfd622fe1c9eed3065118fccf5f48c0f50cf366e4e2b73e2aac81 Nov 02 17:36:52.751406 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:52.751364 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:52.947844 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:52.947797 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:53.043139 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.043052 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f74t5" event={"ID":"db3845da-3e81-491c-9e2d-09f220ea1b9d","Type":"ContainerStarted","Data":"3218686dbf2cfd622fe1c9eed3065118fccf5f48c0f50cf366e4e2b73e2aac81"} Nov 02 17:36:53.109942 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.109908 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cf9f7b655-p25w5"] Nov 02 17:36:53.128238 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.128203 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-p25w5" Nov 02 17:36:53.129880 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.129851 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cf9f7b655-p25w5"] Nov 02 17:36:53.151135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.151088 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7krc4\"" Nov 02 17:36:53.152537 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.152510 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 17:36:53.152537 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.152515 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 17:36:53.168064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.168041 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9bbd\" (UniqueName: \"kubernetes.io/projected/bfa718f4-ef10-4e8d-b342-e4759b10735a-kube-api-access-n9bbd\") pod \"kube-auth-proxy-5cf9f7b655-p25w5\" (UID: \"bfa718f4-ef10-4e8d-b342-e4759b10735a\") " pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-p25w5" Nov 02 17:36:53.168166 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.168110 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfa718f4-ef10-4e8d-b342-e4759b10735a-tls-certs\") pod \"kube-auth-proxy-5cf9f7b655-p25w5\" (UID: \"bfa718f4-ef10-4e8d-b342-e4759b10735a\") " pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-p25w5" Nov 02 17:36:53.168212 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.168192 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfa718f4-ef10-4e8d-b342-e4759b10735a-tmp\") pod \"kube-auth-proxy-5cf9f7b655-p25w5\" (UID: \"bfa718f4-ef10-4e8d-b342-e4759b10735a\") " pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-p25w5" Nov 02 17:36:53.268889 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.268858 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfa718f4-ef10-4e8d-b342-e4759b10735a-tls-certs\") pod \"kube-auth-proxy-5cf9f7b655-p25w5\" (UID: \"bfa718f4-ef10-4e8d-b342-e4759b10735a\") " pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-p25w5" Nov 02 17:36:53.269073 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.268903 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfa718f4-ef10-4e8d-b342-e4759b10735a-tmp\") pod \"kube-auth-proxy-5cf9f7b655-p25w5\" (UID: \"bfa718f4-ef10-4e8d-b342-e4759b10735a\") " pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-p25w5" Nov 02 17:36:53.269073 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.268927 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9bbd\" (UniqueName: \"kubernetes.io/projected/bfa718f4-ef10-4e8d-b342-e4759b10735a-kube-api-access-n9bbd\") pod \"kube-auth-proxy-5cf9f7b655-p25w5\" (UID: \"bfa718f4-ef10-4e8d-b342-e4759b10735a\") " pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-p25w5" Nov 02 17:36:53.271200 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.271171 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfa718f4-ef10-4e8d-b342-e4759b10735a-tmp\") pod \"kube-auth-proxy-5cf9f7b655-p25w5\" (UID: \"bfa718f4-ef10-4e8d-b342-e4759b10735a\") " pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-p25w5" Nov 02 17:36:53.271428 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.271392 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfa718f4-ef10-4e8d-b342-e4759b10735a-tls-certs\") pod \"kube-auth-proxy-5cf9f7b655-p25w5\" (UID: \"bfa718f4-ef10-4e8d-b342-e4759b10735a\") " pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-p25w5" Nov 02 17:36:53.302277 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.302194 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:53.304544 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.304521 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9bbd\" (UniqueName: \"kubernetes.io/projected/bfa718f4-ef10-4e8d-b342-e4759b10735a-kube-api-access-n9bbd\") pod \"kube-auth-proxy-5cf9f7b655-p25w5\" (UID: \"bfa718f4-ef10-4e8d-b342-e4759b10735a\") " pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-p25w5" Nov 02 17:36:53.446655 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.446617 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-p25w5" Nov 02 17:36:53.683226 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.683151 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:53.727032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.727000 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cf9f7b655-p25w5"] Nov 02 17:36:53.729935 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:36:53.729909 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa718f4_ef10_4e8d_b342_e4759b10735a.slice/crio-92122520a3ab05bb7ef9510d95808af7377ec4b1e157500f27a89a0d3127bf9e WatchSource:0}: Error finding container 92122520a3ab05bb7ef9510d95808af7377ec4b1e157500f27a89a0d3127bf9e: Status 404 returned error can't find the container with id 92122520a3ab05bb7ef9510d95808af7377ec4b1e157500f27a89a0d3127bf9e Nov 02 17:36:53.913596 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:53.913560 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:54.048281 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:54.048245 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-p25w5" event={"ID":"bfa718f4-ef10-4e8d-b342-e4759b10735a","Type":"ContainerStarted","Data":"92122520a3ab05bb7ef9510d95808af7377ec4b1e157500f27a89a0d3127bf9e"} Nov 02 17:36:54.147514 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:54.147473 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:54.350275 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:54.350185 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:54.690254 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:54.690173 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:36:54.692683 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:54.692659 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-99bp7" Nov 02 17:36:54.698341 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:54.698319 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 17:36:54.704012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:54.703988 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:36:54.730101 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:54.730069 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:36:54.781773 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:54.781732 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0de90549-6c86-4ce1-9d45-1f3b136fa6e7-config-file\") pod \"limitador-limitador-7c9b87d574-99bp7\" (UID: \"0de90549-6c86-4ce1-9d45-1f3b136fa6e7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-99bp7" Nov 02 17:36:54.781982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:54.781846 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7vqj\" (UniqueName: \"kubernetes.io/projected/0de90549-6c86-4ce1-9d45-1f3b136fa6e7-kube-api-access-p7vqj\") pod \"limitador-limitador-7c9b87d574-99bp7\" (UID: \"0de90549-6c86-4ce1-9d45-1f3b136fa6e7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-99bp7" Nov 02 17:36:54.882849 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:54.882814 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0de90549-6c86-4ce1-9d45-1f3b136fa6e7-config-file\") pod \"limitador-limitador-7c9b87d574-99bp7\" (UID: \"0de90549-6c86-4ce1-9d45-1f3b136fa6e7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-99bp7" Nov 02 17:36:54.883047 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:54.882908 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7vqj\" (UniqueName: \"kubernetes.io/projected/0de90549-6c86-4ce1-9d45-1f3b136fa6e7-kube-api-access-p7vqj\") pod \"limitador-limitador-7c9b87d574-99bp7\" (UID: \"0de90549-6c86-4ce1-9d45-1f3b136fa6e7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-99bp7" Nov 02 17:36:54.883634 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:54.883607 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0de90549-6c86-4ce1-9d45-1f3b136fa6e7-config-file\") pod \"limitador-limitador-7c9b87d574-99bp7\" (UID: \"0de90549-6c86-4ce1-9d45-1f3b136fa6e7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-99bp7" Nov 02 17:36:54.900076 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:54.900044 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7vqj\" (UniqueName: \"kubernetes.io/projected/0de90549-6c86-4ce1-9d45-1f3b136fa6e7-kube-api-access-p7vqj\") pod \"limitador-limitador-7c9b87d574-99bp7\" (UID: \"0de90549-6c86-4ce1-9d45-1f3b136fa6e7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-99bp7" Nov 02 17:36:55.010176 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:55.010127 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-99bp7" Nov 02 17:36:55.078651 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:55.078583 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:55.270730 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:55.270657 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:55.304309 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:55.304274 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:36:55.309173 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:36:55.309138 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de90549_6c86_4ce1_9d45_1f3b136fa6e7.slice/crio-6e8010724bf3bff0b2c08b1ceb7c842be627326248b411412f814b882bce557d WatchSource:0}: Error finding container 6e8010724bf3bff0b2c08b1ceb7c842be627326248b411412f814b882bce557d: Status 404 returned error can't find the container with id 6e8010724bf3bff0b2c08b1ceb7c842be627326248b411412f814b882bce557d Nov 02 17:36:55.478283 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:55.478240 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:55.658925 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:55.658854 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:55.840557 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:55.840523 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:56.026929 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:56.026894 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:56.059022 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:56.058978 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-99bp7" event={"ID":"0de90549-6c86-4ce1-9d45-1f3b136fa6e7","Type":"ContainerStarted","Data":"6e8010724bf3bff0b2c08b1ceb7c842be627326248b411412f814b882bce557d"} Nov 02 17:36:57.064908 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:57.064873 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-p25w5" event={"ID":"bfa718f4-ef10-4e8d-b342-e4759b10735a","Type":"ContainerStarted","Data":"7b9da56be149f214945b5609e658768322bd0eb9f2121a72e239719f2a19ae65"} Nov 02 17:36:57.085617 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:57.085570 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-p25w5" podStartSLOduration=1.692166185 podStartE2EDuration="4.085555221s" totalImagesPullingTime="2.393389036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:57.083871589 +0000 UTC m=+188.872377118" watchObservedRunningTime="2026-11-02 17:36:57.085555221 +0000 UTC m=+188.874060749" Nov 02 17:36:57.254318 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:57.254279 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:57.557117 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:57.557075 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:57.816489 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:57.816395 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:36:58.070615 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:58.070526 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-99bp7" event={"ID":"0de90549-6c86-4ce1-9d45-1f3b136fa6e7","Type":"ContainerStarted","Data":"bd905f08ec40dc18178d5aeade0703beb345f2874670f91aaf7fc3a440b2b9a2"} Nov 02 17:36:58.094412 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:58.094359 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-99bp7" podStartSLOduration=1.46333458 podStartE2EDuration="4.094345801s" totalImagesPullingTime="2.631011221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:58.093527179 +0000 UTC m=+189.882032706" watchObservedRunningTime="2026-11-02 17:36:58.094345801 +0000 UTC m=+189.882851328" Nov 02 17:36:59.074778 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:59.074748 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-99bp7" Nov 02 17:36:59.955217 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:36:59.955178 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:00.155279 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:00.155235 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:00.409483 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:00.409375 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:00.568578 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:00.568547 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:07.483236 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:07.483193 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:07.643333 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:07.643298 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:09.521065 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:09.521032 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:09.699270 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:09.699225 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:10.079569 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:10.079539 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-99bp7" Nov 02 17:37:10.976823 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:10.976785 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:11.159974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:11.159928 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:15.093377 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:15.093319 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:15.147525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:15.147492 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f74t5" event={"ID":"db3845da-3e81-491c-9e2d-09f220ea1b9d","Type":"ContainerStarted","Data":"68526d346c3f6ffbf9e3b1057b24dc87029d2c263d7345f8d8eaf4a4f41b1df2"} Nov 02 17:37:15.166535 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:15.166488 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f74t5" podStartSLOduration=1.995233792 podStartE2EDuration="24.166475727s" totalImagesPullingTime="22.171241935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:15.164849797 +0000 UTC m=+206.953355340" watchObservedRunningTime="2026-11-02 17:37:15.166475727 +0000 UTC m=+206.954981249" Nov 02 17:37:15.298498 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:15.298428 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:15.465039 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:15.465008 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:16.081554 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:16.081524 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:17.585572 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:37:17.585531 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:37:19.740192 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:19.740159 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:19.955570 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:19.955536 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:36.488602 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:36.488568 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:38.350356 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:38.350325 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:41.889065 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:41.889028 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:42.066688 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:42.066655 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:43.053600 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.053559 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx"] Nov 02 17:37:43.056799 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.056778 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" Nov 02 17:37:43.061655 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.061630 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 17:37:43.061896 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.061881 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 17:37:43.061979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.061904 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 17:37:43.061979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.061922 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 17:37:43.066808 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.066769 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx"] Nov 02 17:37:43.093279 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.093242 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 17:37:43.093449 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.093323 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-4lcxd\"" Nov 02 17:37:43.169659 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.169624 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:43.178282 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.178250 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xswfx\" (UID: \"aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" Nov 02 17:37:43.178441 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.178287 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xswfx\" (UID: \"aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" Nov 02 17:37:43.178441 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.178315 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xswfx\" (UID: \"aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" Nov 02 17:37:43.178525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.178481 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47kks\" (UniqueName: \"kubernetes.io/projected/aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4-kube-api-access-47kks\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xswfx\" (UID: \"aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" Nov 02 17:37:43.279833 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.279799 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xswfx\" (UID: \"aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" Nov 02 17:37:43.280053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.279845 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xswfx\" (UID: \"aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" Nov 02 17:37:43.280053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.279896 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47kks\" (UniqueName: \"kubernetes.io/projected/aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4-kube-api-access-47kks\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xswfx\" (UID: \"aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" Nov 02 17:37:43.280053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.279923 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xswfx\" (UID: \"aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" Nov 02 17:37:43.280644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.280617 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xswfx\" (UID: \"aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" Nov 02 17:37:43.280855 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.280836 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xswfx\" (UID: \"aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" Nov 02 17:37:43.282323 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.282302 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xswfx\" (UID: \"aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" Nov 02 17:37:43.288672 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.288649 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47kks\" (UniqueName: \"kubernetes.io/projected/aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4-kube-api-access-47kks\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xswfx\" (UID: \"aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" Nov 02 17:37:43.371021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.370930 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" Nov 02 17:37:43.397246 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.397214 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:43.655503 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:43.655395 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx"] Nov 02 17:37:43.658378 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:37:43.658351 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa514a60_d4ba_4c81_8aca_c1f8f0ae6ef4.slice/crio-f5e5ef22a453b723bcc65b63d2cce2fb634bafbcb6801c60bb4b7e54b1c055ea WatchSource:0}: Error finding container f5e5ef22a453b723bcc65b63d2cce2fb634bafbcb6801c60bb4b7e54b1c055ea: Status 404 returned error can't find the container with id f5e5ef22a453b723bcc65b63d2cce2fb634bafbcb6801c60bb4b7e54b1c055ea Nov 02 17:37:44.249576 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:44.249538 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" event={"ID":"aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4","Type":"ContainerStarted","Data":"f5e5ef22a453b723bcc65b63d2cce2fb634bafbcb6801c60bb4b7e54b1c055ea"} Nov 02 17:37:47.594358 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:37:47.594316 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:37:49.198160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.198122 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:49.287020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.286986 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4q597"] Nov 02 17:37:49.310020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.309986 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4q597"] Nov 02 17:37:49.310198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.310098 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-4q597" Nov 02 17:37:49.319449 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.319247 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 17:37:49.319449 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.319270 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-vbjf9\"" Nov 02 17:37:49.319915 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.319892 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-2wms8"] Nov 02 17:37:49.327224 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.327201 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-2wms8" Nov 02 17:37:49.335160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.334926 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-2wms8"] Nov 02 17:37:49.348049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.348019 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4kj5b\"" Nov 02 17:37:49.348049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.348026 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 17:37:49.348354 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.348064 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 17:37:49.371457 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.371390 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:49.437769 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.437725 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13885bf3-7cdc-40aa-8e58-01eced7a6249-cert\") pod \"odh-model-controller-769875db56-2wms8\" (UID: \"13885bf3-7cdc-40aa-8e58-01eced7a6249\") " pod="opendatahub/odh-model-controller-769875db56-2wms8" Nov 02 17:37:49.437939 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.437871 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp97z\" (UniqueName: \"kubernetes.io/projected/13885bf3-7cdc-40aa-8e58-01eced7a6249-kube-api-access-xp97z\") pod \"odh-model-controller-769875db56-2wms8\" (UID: \"13885bf3-7cdc-40aa-8e58-01eced7a6249\") " pod="opendatahub/odh-model-controller-769875db56-2wms8" Nov 02 17:37:49.437939 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.437913 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw7bf\" (UniqueName: \"kubernetes.io/projected/585a619b-ddf1-497a-9fa0-40d105881005-kube-api-access-qw7bf\") pod \"model-serving-api-75bd89fdfc-4q597\" (UID: \"585a619b-ddf1-497a-9fa0-40d105881005\") " pod="opendatahub/model-serving-api-75bd89fdfc-4q597" Nov 02 17:37:49.438085 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.437985 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13885bf3-7cdc-40aa-8e58-01eced7a6249-metrics-certs\") pod \"odh-model-controller-769875db56-2wms8\" (UID: \"13885bf3-7cdc-40aa-8e58-01eced7a6249\") " pod="opendatahub/odh-model-controller-769875db56-2wms8" Nov 02 17:37:49.438085 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.438026 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/585a619b-ddf1-497a-9fa0-40d105881005-tls-certs\") pod \"model-serving-api-75bd89fdfc-4q597\" (UID: \"585a619b-ddf1-497a-9fa0-40d105881005\") " pod="opendatahub/model-serving-api-75bd89fdfc-4q597" Nov 02 17:37:49.517377 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.517297 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:49.538808 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.538765 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13885bf3-7cdc-40aa-8e58-01eced7a6249-cert\") pod \"odh-model-controller-769875db56-2wms8\" (UID: \"13885bf3-7cdc-40aa-8e58-01eced7a6249\") " pod="opendatahub/odh-model-controller-769875db56-2wms8" Nov 02 17:37:49.539024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.538832 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp97z\" (UniqueName: \"kubernetes.io/projected/13885bf3-7cdc-40aa-8e58-01eced7a6249-kube-api-access-xp97z\") pod \"odh-model-controller-769875db56-2wms8\" (UID: \"13885bf3-7cdc-40aa-8e58-01eced7a6249\") " pod="opendatahub/odh-model-controller-769875db56-2wms8" Nov 02 17:37:49.539024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.538872 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw7bf\" (UniqueName: \"kubernetes.io/projected/585a619b-ddf1-497a-9fa0-40d105881005-kube-api-access-qw7bf\") pod \"model-serving-api-75bd89fdfc-4q597\" (UID: \"585a619b-ddf1-497a-9fa0-40d105881005\") " pod="opendatahub/model-serving-api-75bd89fdfc-4q597" Nov 02 17:37:49.539024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.538922 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13885bf3-7cdc-40aa-8e58-01eced7a6249-metrics-certs\") pod \"odh-model-controller-769875db56-2wms8\" (UID: \"13885bf3-7cdc-40aa-8e58-01eced7a6249\") " pod="opendatahub/odh-model-controller-769875db56-2wms8" Nov 02 17:37:49.539024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.538987 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/585a619b-ddf1-497a-9fa0-40d105881005-tls-certs\") pod \"model-serving-api-75bd89fdfc-4q597\" (UID: \"585a619b-ddf1-497a-9fa0-40d105881005\") " pod="opendatahub/model-serving-api-75bd89fdfc-4q597" Nov 02 17:37:49.542249 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.542219 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/585a619b-ddf1-497a-9fa0-40d105881005-tls-certs\") pod \"model-serving-api-75bd89fdfc-4q597\" (UID: \"585a619b-ddf1-497a-9fa0-40d105881005\") " pod="opendatahub/model-serving-api-75bd89fdfc-4q597" Nov 02 17:37:49.542443 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.542245 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13885bf3-7cdc-40aa-8e58-01eced7a6249-metrics-certs\") pod \"odh-model-controller-769875db56-2wms8\" (UID: \"13885bf3-7cdc-40aa-8e58-01eced7a6249\") " pod="opendatahub/odh-model-controller-769875db56-2wms8" Nov 02 17:37:49.542443 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.542330 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13885bf3-7cdc-40aa-8e58-01eced7a6249-cert\") pod \"odh-model-controller-769875db56-2wms8\" (UID: \"13885bf3-7cdc-40aa-8e58-01eced7a6249\") " pod="opendatahub/odh-model-controller-769875db56-2wms8" Nov 02 17:37:49.550180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.550150 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw7bf\" (UniqueName: \"kubernetes.io/projected/585a619b-ddf1-497a-9fa0-40d105881005-kube-api-access-qw7bf\") pod \"model-serving-api-75bd89fdfc-4q597\" (UID: \"585a619b-ddf1-497a-9fa0-40d105881005\") " pod="opendatahub/model-serving-api-75bd89fdfc-4q597" Nov 02 17:37:49.550339 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.550316 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp97z\" (UniqueName: \"kubernetes.io/projected/13885bf3-7cdc-40aa-8e58-01eced7a6249-kube-api-access-xp97z\") pod \"odh-model-controller-769875db56-2wms8\" (UID: \"13885bf3-7cdc-40aa-8e58-01eced7a6249\") " pod="opendatahub/odh-model-controller-769875db56-2wms8" Nov 02 17:37:49.566020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.565989 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj"] Nov 02 17:37:49.573201 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.573170 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.586088 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.586023 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj"] Nov 02 17:37:49.629362 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.629318 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-4q597" Nov 02 17:37:49.640107 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.640079 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.640255 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.640127 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz6d5\" (UniqueName: \"kubernetes.io/projected/8cb1b264-9e1e-4838-a719-d962ad2f35a9-kube-api-access-tz6d5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.640255 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.640228 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.640390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.640261 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.640390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.640293 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.640390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.640314 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.640390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.640345 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cb1b264-9e1e-4838-a719-d962ad2f35a9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.648281 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.648252 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-2wms8" Nov 02 17:37:49.664263 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.664200 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:49.743735 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.741786 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.743735 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.741857 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.743735 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.741895 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.743735 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.741922 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.743735 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.741985 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cb1b264-9e1e-4838-a719-d962ad2f35a9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.743735 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.742035 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.743735 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.742078 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz6d5\" (UniqueName: \"kubernetes.io/projected/8cb1b264-9e1e-4838-a719-d962ad2f35a9-kube-api-access-tz6d5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.743735 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.743157 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.747266 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.746758 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.747266 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.747142 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.747266 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.747217 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.747542 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.747399 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cb1b264-9e1e-4838-a719-d962ad2f35a9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.754383 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.754347 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz6d5\" (UniqueName: \"kubernetes.io/projected/8cb1b264-9e1e-4838-a719-d962ad2f35a9-kube-api-access-tz6d5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.754526 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.754502 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tlkmj\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.890457 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.890376 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:49.915901 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.915871 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4q597"] Nov 02 17:37:49.929555 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:37:49.929505 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585a619b_ddf1_497a_9fa0_40d105881005.slice/crio-a49bc8c898dc4adde788363da5145c502eb3e584d6b20f4c06f8a384cd28cb93 WatchSource:0}: Error finding container a49bc8c898dc4adde788363da5145c502eb3e584d6b20f4c06f8a384cd28cb93: Status 404 returned error can't find the container with id a49bc8c898dc4adde788363da5145c502eb3e584d6b20f4c06f8a384cd28cb93 Nov 02 17:37:49.940109 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:49.940082 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-2wms8"] Nov 02 17:37:49.945440 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:37:49.945406 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13885bf3_7cdc_40aa_8e58_01eced7a6249.slice/crio-05de7bcafb2f0c170df28fbd5f6a57a1cb0617294b967e5aec1516c4c5c48ce8 WatchSource:0}: Error finding container 05de7bcafb2f0c170df28fbd5f6a57a1cb0617294b967e5aec1516c4c5c48ce8: Status 404 returned error can't find the container with id 05de7bcafb2f0c170df28fbd5f6a57a1cb0617294b967e5aec1516c4c5c48ce8 Nov 02 17:37:50.180971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:50.180874 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj"] Nov 02 17:37:50.183387 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:37:50.183357 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb1b264_9e1e_4838_a719_d962ad2f35a9.slice/crio-c0ba8e45760af4fea54d727b5fd48a83a497b9303f3c2fd62c9bf5fb295132f1 WatchSource:0}: Error finding container c0ba8e45760af4fea54d727b5fd48a83a497b9303f3c2fd62c9bf5fb295132f1: Status 404 returned error can't find the container with id c0ba8e45760af4fea54d727b5fd48a83a497b9303f3c2fd62c9bf5fb295132f1 Nov 02 17:37:50.186266 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:50.186226 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:37:50.186365 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:50.186308 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:37:50.281839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:50.281584 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" event={"ID":"8cb1b264-9e1e-4838-a719-d962ad2f35a9","Type":"ContainerStarted","Data":"a61b851a8ab4a5c371f9537f663914999bebd1cc2062ccb2f488a726309d48fe"} Nov 02 17:37:50.281839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:50.281646 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" event={"ID":"8cb1b264-9e1e-4838-a719-d962ad2f35a9","Type":"ContainerStarted","Data":"c0ba8e45760af4fea54d727b5fd48a83a497b9303f3c2fd62c9bf5fb295132f1"} Nov 02 17:37:50.281839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:50.281776 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:50.284167 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:50.284143 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-2wms8" event={"ID":"13885bf3-7cdc-40aa-8e58-01eced7a6249","Type":"ContainerStarted","Data":"05de7bcafb2f0c170df28fbd5f6a57a1cb0617294b967e5aec1516c4c5c48ce8"} Nov 02 17:37:50.286372 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:50.286354 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-4q597" event={"ID":"585a619b-ddf1-497a-9fa0-40d105881005","Type":"ContainerStarted","Data":"a49bc8c898dc4adde788363da5145c502eb3e584d6b20f4c06f8a384cd28cb93"} Nov 02 17:37:50.308345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:50.308289 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" podStartSLOduration=1.3082753710000001 podStartE2EDuration="1.308275371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:50.307327118 +0000 UTC m=+242.095832647" watchObservedRunningTime="2026-11-02 17:37:50.308275371 +0000 UTC m=+242.096780899" Nov 02 17:37:50.449914 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:50.449846 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:50.602547 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:50.602509 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:50.758853 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:50.758767 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:51.294873 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.294818 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 17:37:51.347714 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.347229 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj"] Nov 02 17:37:51.347714 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.347542 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" podUID="0234efe0-dffb-4cae-8086-7ec56cc4e950" containerName="discovery" containerID="cri-o://f2e998415ad7ea67f4db63488114ff3d59bd6c58f3a54e9c100efc877f472a5d" gracePeriod=30 Nov 02 17:37:51.663798 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.663770 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:37:51.764806 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.764765 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-csr-ca-configmap\") pod \"0234efe0-dffb-4cae-8086-7ec56cc4e950\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " Nov 02 17:37:51.764992 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.764846 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-token\" (UniqueName: \"kubernetes.io/projected/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-token\") pod \"0234efe0-dffb-4cae-8086-7ec56cc4e950\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " Nov 02 17:37:51.764992 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.764945 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-kubeconfig\") pod \"0234efe0-dffb-4cae-8086-7ec56cc4e950\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " Nov 02 17:37:51.765120 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.765083 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-cacerts\" (UniqueName: \"kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-cacerts\") pod \"0234efe0-dffb-4cae-8086-7ec56cc4e950\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " Nov 02 17:37:51.765180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.765168 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0234efe0-dffb-4cae-8086-7ec56cc4e950-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0234efe0-dffb-4cae-8086-7ec56cc4e950-local-certs\") pod \"0234efe0-dffb-4cae-8086-7ec56cc4e950\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " Nov 02 17:37:51.765232 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.765210 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-csr-dns-cert\") pod \"0234efe0-dffb-4cae-8086-7ec56cc4e950\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " Nov 02 17:37:51.765283 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.765240 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0234efe0-dffb-4cae-8086-7ec56cc4e950-kube-api-access-dwd4l\" (UniqueName: \"kubernetes.io/projected/0234efe0-dffb-4cae-8086-7ec56cc4e950-kube-api-access-dwd4l\") pod \"0234efe0-dffb-4cae-8086-7ec56cc4e950\" (UID: \"0234efe0-dffb-4cae-8086-7ec56cc4e950\") " Nov 02 17:37:51.765283 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.765251 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-csr-ca-configmap" pod "0234efe0-dffb-4cae-8086-7ec56cc4e950" (UID: "0234efe0-dffb-4cae-8086-7ec56cc4e950"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:37:51.766294 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.765494 5549 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-csr-ca-configmap\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:37:51.770013 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.769962 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-kubeconfig" pod "0234efe0-dffb-4cae-8086-7ec56cc4e950" (UID: "0234efe0-dffb-4cae-8086-7ec56cc4e950"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:37:51.770322 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.770289 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-csr-dns-cert" pod "0234efe0-dffb-4cae-8086-7ec56cc4e950" (UID: "0234efe0-dffb-4cae-8086-7ec56cc4e950"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:37:51.770564 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.770544 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-cacerts" pod "0234efe0-dffb-4cae-8086-7ec56cc4e950" (UID: "0234efe0-dffb-4cae-8086-7ec56cc4e950"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:37:51.781125 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.781097 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-token" pod "0234efe0-dffb-4cae-8086-7ec56cc4e950" (UID: "0234efe0-dffb-4cae-8086-7ec56cc4e950"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:37:51.782567 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.782500 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0234efe0-dffb-4cae-8086-7ec56cc4e950-kube-api-access-dwd4l" pod "0234efe0-dffb-4cae-8086-7ec56cc4e950" (UID: "0234efe0-dffb-4cae-8086-7ec56cc4e950"). InnerVolumeSpecName "kube-api-access-dwd4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:37:51.783322 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.783261 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0234efe0-dffb-4cae-8086-7ec56cc4e950-local-certs" pod "0234efe0-dffb-4cae-8086-7ec56cc4e950" (UID: "0234efe0-dffb-4cae-8086-7ec56cc4e950"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:37:51.866525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.866436 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:37:51.866525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.866474 5549 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-kubeconfig\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:37:51.866525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.866489 5549 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-cacerts\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:37:51.866525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.866504 5549 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0234efe0-dffb-4cae-8086-7ec56cc4e950-local-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:37:51.866525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.866518 5549 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0234efe0-dffb-4cae-8086-7ec56cc4e950-istio-csr-dns-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:37:51.866897 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:51.866537 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwd4l\" (UniqueName: \"kubernetes.io/projected/0234efe0-dffb-4cae-8086-7ec56cc4e950-kube-api-access-dwd4l\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:37:52.299750 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:52.299715 5549 generic.go:356] "Generic (PLEG): container finished" podID="0234efe0-dffb-4cae-8086-7ec56cc4e950" containerID="f2e998415ad7ea67f4db63488114ff3d59bd6c58f3a54e9c100efc877f472a5d" exitCode=0 Nov 02 17:37:52.300393 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:52.300374 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" Nov 02 17:37:52.302071 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:52.302039 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" event={"ID":"0234efe0-dffb-4cae-8086-7ec56cc4e950","Type":"ContainerDied","Data":"f2e998415ad7ea67f4db63488114ff3d59bd6c58f3a54e9c100efc877f472a5d"} Nov 02 17:37:52.302186 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:52.302108 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj" event={"ID":"0234efe0-dffb-4cae-8086-7ec56cc4e950","Type":"ContainerDied","Data":"0d793a09e99be8db2e94749aa4b3f1e2105cc0c544f80be13af04cd58f0a357f"} Nov 02 17:37:52.302186 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:52.302139 5549 scope.go:122] "RemoveContainer" containerID="f2e998415ad7ea67f4db63488114ff3d59bd6c58f3a54e9c100efc877f472a5d" Nov 02 17:37:52.318409 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:52.318386 5549 scope.go:122] "RemoveContainer" containerID="f2e998415ad7ea67f4db63488114ff3d59bd6c58f3a54e9c100efc877f472a5d" Nov 02 17:37:52.319274 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:37:52.319250 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e998415ad7ea67f4db63488114ff3d59bd6c58f3a54e9c100efc877f472a5d\": container with ID starting with f2e998415ad7ea67f4db63488114ff3d59bd6c58f3a54e9c100efc877f472a5d not found: ID does not exist" containerID="f2e998415ad7ea67f4db63488114ff3d59bd6c58f3a54e9c100efc877f472a5d" Nov 02 17:37:52.319360 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:52.319284 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e998415ad7ea67f4db63488114ff3d59bd6c58f3a54e9c100efc877f472a5d"} err="failed to get container status \"f2e998415ad7ea67f4db63488114ff3d59bd6c58f3a54e9c100efc877f472a5d\": rpc error: code = NotFound desc = could not find container \"f2e998415ad7ea67f4db63488114ff3d59bd6c58f3a54e9c100efc877f472a5d\": container with ID starting with f2e998415ad7ea67f4db63488114ff3d59bd6c58f3a54e9c100efc877f472a5d not found: ID does not exist" Nov 02 17:37:52.336573 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:52.336536 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj"] Nov 02 17:37:52.338648 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:52.338619 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2wlj"] Nov 02 17:37:53.308654 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:53.308048 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-4q597" event={"ID":"585a619b-ddf1-497a-9fa0-40d105881005","Type":"ContainerStarted","Data":"8580b5c77a71f223b4a407b785dd67702afc9c72d3a768d8a21b439b055f0023"} Nov 02 17:37:53.308654 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:53.308164 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-4q597" Nov 02 17:37:53.334672 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:53.334579 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-4q597" podStartSLOduration=1.820592464 podStartE2EDuration="4.334560464s" totalImagesPullingTime="2.513968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:53.333154798 +0000 UTC m=+245.121660338" watchObservedRunningTime="2026-11-02 17:37:53.334560464 +0000 UTC m=+245.123065991" Nov 02 17:37:54.317523 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:54.317491 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0234efe0-dffb-4cae-8086-7ec56cc4e950" path="/var/lib/kubelet/pods/0234efe0-dffb-4cae-8086-7ec56cc4e950/volumes" Nov 02 17:37:54.318009 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:54.317752 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-2wms8" event={"ID":"13885bf3-7cdc-40aa-8e58-01eced7a6249","Type":"ContainerStarted","Data":"e2e085e52d9e00cee222764c9d86cc55ebf9f63020df2a0231dddd7159de16ce"} Nov 02 17:37:54.335172 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:54.335115 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-2wms8" podStartSLOduration=1.8670947949999999 podStartE2EDuration="5.335097092s" totalImagesPullingTime="3.468002297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:54.33409234 +0000 UTC m=+246.122597869" watchObservedRunningTime="2026-11-02 17:37:54.335097092 +0000 UTC m=+246.123602623" Nov 02 17:37:55.318866 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:55.318835 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-2wms8" Nov 02 17:37:57.326476 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:57.326435 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:37:57.328584 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:57.328559 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" event={"ID":"aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4","Type":"ContainerStarted","Data":"5bb0e8dba87f69d8187e76ae4f1fb0f415ea84790f09e4fd6500303cadf714ad"} Nov 02 17:37:57.328842 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:57.328622 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" Nov 02 17:37:57.351831 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:37:57.351764 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" podStartSLOduration=1.1042791 podStartE2EDuration="14.351746214s" totalImagesPullingTime="13.247467114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:57.3502276 +0000 UTC m=+249.138733130" watchObservedRunningTime="2026-11-02 17:37:57.351746214 +0000 UTC m=+249.140251742" Nov 02 17:38:04.318259 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:04.318232 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-4q597" Nov 02 17:38:06.325217 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:06.325186 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-2wms8" Nov 02 17:38:08.339378 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:08.339343 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xswfx" Nov 02 17:38:13.228671 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:13.228635 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:38:13.376107 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:13.376074 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:38:13.689727 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:13.689636 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:38:13.840316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:13.840280 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:38:14.519036 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:14.518992 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:38:14.661500 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:14.661470 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:38:15.936783 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:15.936748 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:38:17.586331 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:38:17.586292 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:38:26.424554 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:26.424515 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:38:26.592172 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:26.592132 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:38:26.811908 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:26.811869 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc"] Nov 02 17:38:26.812262 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:26.812248 5549 cpu_manager.go:414] "containerMap: removing container" podUID="0234efe0-dffb-4cae-8086-7ec56cc4e950" containerName="discovery" Nov 02 17:38:26.812314 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:26.812263 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0234efe0-dffb-4cae-8086-7ec56cc4e950" containerName="discovery" Nov 02 17:38:26.812350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:26.812321 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="0234efe0-dffb-4cae-8086-7ec56cc4e950" containerName="discovery" Nov 02 17:38:26.878249 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:26.878215 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc"] Nov 02 17:38:26.878422 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:26.878352 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" Nov 02 17:38:26.883426 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:26.883399 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 17:38:26.884660 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:26.884638 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 17:38:26.885014 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:26.884977 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 17:38:26.885074 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:26.885003 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 17:38:26.885074 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:26.885026 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tqgzn\"" Nov 02 17:38:26.973387 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:26.973350 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmb4z\" (UniqueName: \"kubernetes.io/projected/1ae546ff-a649-4895-9aaf-53395a39e5a5-kube-api-access-cmb4z\") pod \"maas-api-84f9c6994b-xjlfc\" (UID: \"1ae546ff-a649-4895-9aaf-53395a39e5a5\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" Nov 02 17:38:26.973387 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:26.973390 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae546ff-a649-4895-9aaf-53395a39e5a5-openshift-service-ca\") pod \"maas-api-84f9c6994b-xjlfc\" (UID: \"1ae546ff-a649-4895-9aaf-53395a39e5a5\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" Nov 02 17:38:26.973878 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:26.973426 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae546ff-a649-4895-9aaf-53395a39e5a5-metrics-certs\") pod \"maas-api-84f9c6994b-xjlfc\" (UID: \"1ae546ff-a649-4895-9aaf-53395a39e5a5\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" Nov 02 17:38:26.974130 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:26.974108 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae546ff-a649-4895-9aaf-53395a39e5a5-maas-api-tls\") pod \"maas-api-84f9c6994b-xjlfc\" (UID: \"1ae546ff-a649-4895-9aaf-53395a39e5a5\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" Nov 02 17:38:27.075787 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:27.075691 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmb4z\" (UniqueName: \"kubernetes.io/projected/1ae546ff-a649-4895-9aaf-53395a39e5a5-kube-api-access-cmb4z\") pod \"maas-api-84f9c6994b-xjlfc\" (UID: \"1ae546ff-a649-4895-9aaf-53395a39e5a5\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" Nov 02 17:38:27.075787 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:27.075744 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae546ff-a649-4895-9aaf-53395a39e5a5-openshift-service-ca\") pod \"maas-api-84f9c6994b-xjlfc\" (UID: \"1ae546ff-a649-4895-9aaf-53395a39e5a5\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" Nov 02 17:38:27.075787 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:27.075778 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae546ff-a649-4895-9aaf-53395a39e5a5-metrics-certs\") pod \"maas-api-84f9c6994b-xjlfc\" (UID: \"1ae546ff-a649-4895-9aaf-53395a39e5a5\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" Nov 02 17:38:27.076042 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:27.075803 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae546ff-a649-4895-9aaf-53395a39e5a5-maas-api-tls\") pod \"maas-api-84f9c6994b-xjlfc\" (UID: \"1ae546ff-a649-4895-9aaf-53395a39e5a5\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" Nov 02 17:38:27.076632 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:27.076604 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae546ff-a649-4895-9aaf-53395a39e5a5-openshift-service-ca\") pod \"maas-api-84f9c6994b-xjlfc\" (UID: \"1ae546ff-a649-4895-9aaf-53395a39e5a5\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" Nov 02 17:38:27.078341 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:27.078314 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae546ff-a649-4895-9aaf-53395a39e5a5-metrics-certs\") pod \"maas-api-84f9c6994b-xjlfc\" (UID: \"1ae546ff-a649-4895-9aaf-53395a39e5a5\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" Nov 02 17:38:27.078509 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:27.078487 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae546ff-a649-4895-9aaf-53395a39e5a5-maas-api-tls\") pod \"maas-api-84f9c6994b-xjlfc\" (UID: \"1ae546ff-a649-4895-9aaf-53395a39e5a5\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" Nov 02 17:38:27.087867 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:27.087838 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmb4z\" (UniqueName: \"kubernetes.io/projected/1ae546ff-a649-4895-9aaf-53395a39e5a5-kube-api-access-cmb4z\") pod \"maas-api-84f9c6994b-xjlfc\" (UID: \"1ae546ff-a649-4895-9aaf-53395a39e5a5\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" Nov 02 17:38:27.193595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:27.193552 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" Nov 02 17:38:27.495635 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:27.495594 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc"] Nov 02 17:38:27.498807 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:38:27.498767 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae546ff_a649_4895_9aaf_53395a39e5a5.slice/crio-9805ff650041d9e8328156dcbacd6a9c54a3ac449e2fe43410fed149587ef30c WatchSource:0}: Error finding container 9805ff650041d9e8328156dcbacd6a9c54a3ac449e2fe43410fed149587ef30c: Status 404 returned error can't find the container with id 9805ff650041d9e8328156dcbacd6a9c54a3ac449e2fe43410fed149587ef30c Nov 02 17:38:28.454025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:28.453982 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" event={"ID":"1ae546ff-a649-4895-9aaf-53395a39e5a5","Type":"ContainerStarted","Data":"9805ff650041d9e8328156dcbacd6a9c54a3ac449e2fe43410fed149587ef30c"} Nov 02 17:38:29.021604 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:29.021564 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:38:29.189154 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:29.189117 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:38:31.469038 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:31.468998 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" event={"ID":"1ae546ff-a649-4895-9aaf-53395a39e5a5","Type":"ContainerStarted","Data":"3239ec55f4865c697c101b822965bb97deaec0315473f3c73a9609fbf0673e1d"} Nov 02 17:38:31.469534 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:31.469118 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" Nov 02 17:38:31.489437 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:31.489391 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" podStartSLOduration=2.501444368 podStartE2EDuration="5.489379178s" totalImagesPullingTime="2.98793481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:31.48880886 +0000 UTC m=+283.277314390" watchObservedRunningTime="2026-11-02 17:38:31.489379178 +0000 UTC m=+283.277884727" Nov 02 17:38:37.478831 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:37.478801 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" Nov 02 17:38:47.585270 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:38:47.585231 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:38:48.353721 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:48.353693 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 17:38:48.353844 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:48.353697 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 17:38:57.366230 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:38:57.366195 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:17.585664 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:39:17.585626 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:39:24.724040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:24.724000 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-7rzb7"] Nov 02 17:39:24.728971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:24.728933 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-7rzb7" Nov 02 17:39:24.739129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:24.739104 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 17:39:24.739249 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:24.739137 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-tmdtv\"" Nov 02 17:39:24.739249 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:24.739151 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 17:39:24.740602 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:24.740576 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-7rzb7"] Nov 02 17:39:24.740697 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:24.740601 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw2t6\" (UniqueName: \"kubernetes.io/projected/05961055-fed0-463a-acd4-58784cfe3d5a-kube-api-access-rw2t6\") pod \"ai-gateway-controller-5d8dddb676-7rzb7\" (UID: \"05961055-fed0-463a-acd4-58784cfe3d5a\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-7rzb7" Nov 02 17:39:24.740697 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:24.740639 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05961055-fed0-463a-acd4-58784cfe3d5a-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-7rzb7\" (UID: \"05961055-fed0-463a-acd4-58784cfe3d5a\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-7rzb7" Nov 02 17:39:24.781176 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:24.781141 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:24.841327 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:24.841293 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw2t6\" (UniqueName: \"kubernetes.io/projected/05961055-fed0-463a-acd4-58784cfe3d5a-kube-api-access-rw2t6\") pod \"ai-gateway-controller-5d8dddb676-7rzb7\" (UID: \"05961055-fed0-463a-acd4-58784cfe3d5a\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-7rzb7" Nov 02 17:39:24.841515 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:24.841352 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05961055-fed0-463a-acd4-58784cfe3d5a-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-7rzb7\" (UID: \"05961055-fed0-463a-acd4-58784cfe3d5a\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-7rzb7" Nov 02 17:39:24.843972 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:24.843913 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05961055-fed0-463a-acd4-58784cfe3d5a-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-7rzb7\" (UID: \"05961055-fed0-463a-acd4-58784cfe3d5a\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-7rzb7" Nov 02 17:39:24.850458 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:24.850436 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw2t6\" (UniqueName: \"kubernetes.io/projected/05961055-fed0-463a-acd4-58784cfe3d5a-kube-api-access-rw2t6\") pod \"ai-gateway-controller-5d8dddb676-7rzb7\" (UID: \"05961055-fed0-463a-acd4-58784cfe3d5a\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-7rzb7" Nov 02 17:39:24.917709 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:24.917676 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:25.041546 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:25.041457 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-7rzb7" Nov 02 17:39:25.314175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:25.314147 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-7rzb7"] Nov 02 17:39:25.316726 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:39:25.316701 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05961055_fed0_463a_acd4_58784cfe3d5a.slice/crio-b11ecc4122c711d97335a2ef95fe9f3118062ebd10a48a3e9dc299117aebbc14 WatchSource:0}: Error finding container b11ecc4122c711d97335a2ef95fe9f3118062ebd10a48a3e9dc299117aebbc14: Status 404 returned error can't find the container with id b11ecc4122c711d97335a2ef95fe9f3118062ebd10a48a3e9dc299117aebbc14 Nov 02 17:39:25.680770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:25.680683 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-7rzb7" event={"ID":"05961055-fed0-463a-acd4-58784cfe3d5a","Type":"ContainerStarted","Data":"b11ecc4122c711d97335a2ef95fe9f3118062ebd10a48a3e9dc299117aebbc14"} Nov 02 17:39:28.697707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.697672 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-7rzb7" event={"ID":"05961055-fed0-463a-acd4-58784cfe3d5a","Type":"ContainerStarted","Data":"19d0e7d3f73655919559a6cee5243852deffddf1a4cd743f3ab39f9933d6120d"} Nov 02 17:39:28.698105 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.697777 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-7rzb7" Nov 02 17:39:28.700432 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.700407 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-5hb2w"] Nov 02 17:39:28.703915 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.703900 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-5hb2w" Nov 02 17:39:28.713344 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.713319 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 17:39:28.714574 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.714551 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-h87nl\"" Nov 02 17:39:28.715277 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.715253 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:28.715469 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.715447 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-5hb2w"] Nov 02 17:39:28.716300 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.716278 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m"] Nov 02 17:39:28.719008 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.718970 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-7rzb7" podStartSLOduration=2.120047014 podStartE2EDuration="4.718940391s" totalImagesPullingTime="2.598893377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:28.71788565 +0000 UTC m=+340.506391240" watchObservedRunningTime="2026-11-02 17:39:28.718940391 +0000 UTC m=+340.507445955" Nov 02 17:39:28.719802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.719784 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m" Nov 02 17:39:28.729236 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.729215 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m"] Nov 02 17:39:28.869701 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.869663 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9cvz\" (UniqueName: \"kubernetes.io/projected/b693ebb5-876a-426b-bf1d-6fdf00d3e96a-kube-api-access-h9cvz\") pod \"payload-processing-55fbbbf9dc-5hb2w\" (UID: \"b693ebb5-876a-426b-bf1d-6fdf00d3e96a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-5hb2w" Nov 02 17:39:28.869854 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.869781 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e0ed62-81a5-4917-baef-0942cc4a7aee-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-nnt4m\" (UID: \"53e0ed62-81a5-4917-baef-0942cc4a7aee\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m" Nov 02 17:39:28.869854 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.869819 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b693ebb5-876a-426b-bf1d-6fdf00d3e96a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-5hb2w\" (UID: \"b693ebb5-876a-426b-bf1d-6fdf00d3e96a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-5hb2w" Nov 02 17:39:28.869926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.869863 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7kvv\" (UniqueName: \"kubernetes.io/projected/53e0ed62-81a5-4917-baef-0942cc4a7aee-kube-api-access-d7kvv\") pod \"payload-pre-processing-7b98bb678c-nnt4m\" (UID: \"53e0ed62-81a5-4917-baef-0942cc4a7aee\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m" Nov 02 17:39:28.873026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.872989 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:28.970317 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.970242 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e0ed62-81a5-4917-baef-0942cc4a7aee-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-nnt4m\" (UID: \"53e0ed62-81a5-4917-baef-0942cc4a7aee\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m" Nov 02 17:39:28.970317 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.970277 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b693ebb5-876a-426b-bf1d-6fdf00d3e96a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-5hb2w\" (UID: \"b693ebb5-876a-426b-bf1d-6fdf00d3e96a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-5hb2w" Nov 02 17:39:28.970317 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.970312 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7kvv\" (UniqueName: \"kubernetes.io/projected/53e0ed62-81a5-4917-baef-0942cc4a7aee-kube-api-access-d7kvv\") pod \"payload-pre-processing-7b98bb678c-nnt4m\" (UID: \"53e0ed62-81a5-4917-baef-0942cc4a7aee\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m" Nov 02 17:39:28.970534 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.970353 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9cvz\" (UniqueName: \"kubernetes.io/projected/b693ebb5-876a-426b-bf1d-6fdf00d3e96a-kube-api-access-h9cvz\") pod \"payload-processing-55fbbbf9dc-5hb2w\" (UID: \"b693ebb5-876a-426b-bf1d-6fdf00d3e96a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-5hb2w" Nov 02 17:39:28.970990 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.970943 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e0ed62-81a5-4917-baef-0942cc4a7aee-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-nnt4m\" (UID: \"53e0ed62-81a5-4917-baef-0942cc4a7aee\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m" Nov 02 17:39:28.971064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.971011 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b693ebb5-876a-426b-bf1d-6fdf00d3e96a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-5hb2w\" (UID: \"b693ebb5-876a-426b-bf1d-6fdf00d3e96a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-5hb2w" Nov 02 17:39:28.979370 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.979341 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9cvz\" (UniqueName: \"kubernetes.io/projected/b693ebb5-876a-426b-bf1d-6fdf00d3e96a-kube-api-access-h9cvz\") pod \"payload-processing-55fbbbf9dc-5hb2w\" (UID: \"b693ebb5-876a-426b-bf1d-6fdf00d3e96a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-5hb2w" Nov 02 17:39:28.979370 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:28.979363 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7kvv\" (UniqueName: \"kubernetes.io/projected/53e0ed62-81a5-4917-baef-0942cc4a7aee-kube-api-access-d7kvv\") pod \"payload-pre-processing-7b98bb678c-nnt4m\" (UID: \"53e0ed62-81a5-4917-baef-0942cc4a7aee\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m" Nov 02 17:39:29.016396 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:29.016371 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-5hb2w" Nov 02 17:39:29.035617 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:29.035591 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m" Nov 02 17:39:29.291317 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:29.291287 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-5hb2w"] Nov 02 17:39:29.293704 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:39:29.293670 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb693ebb5_876a_426b_bf1d_6fdf00d3e96a.slice/crio-c6688d00cc84ad97ccf290e8d7e09f543167ccb05d758d1c7160c9f0094563dc WatchSource:0}: Error finding container c6688d00cc84ad97ccf290e8d7e09f543167ccb05d758d1c7160c9f0094563dc: Status 404 returned error can't find the container with id c6688d00cc84ad97ccf290e8d7e09f543167ccb05d758d1c7160c9f0094563dc Nov 02 17:39:29.328614 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:29.328580 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m"] Nov 02 17:39:29.331260 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:39:29.331231 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e0ed62_81a5_4917_baef_0942cc4a7aee.slice/crio-1f62d99dda910f8d0162198ec4ff6dfed34b81ebc79da4acb5f16ba56c15e442 WatchSource:0}: Error finding container 1f62d99dda910f8d0162198ec4ff6dfed34b81ebc79da4acb5f16ba56c15e442: Status 404 returned error can't find the container with id 1f62d99dda910f8d0162198ec4ff6dfed34b81ebc79da4acb5f16ba56c15e442 Nov 02 17:39:29.702928 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:29.702892 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-5hb2w" event={"ID":"b693ebb5-876a-426b-bf1d-6fdf00d3e96a","Type":"ContainerStarted","Data":"c6688d00cc84ad97ccf290e8d7e09f543167ccb05d758d1c7160c9f0094563dc"} Nov 02 17:39:29.704160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:29.704132 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m" event={"ID":"53e0ed62-81a5-4917-baef-0942cc4a7aee","Type":"ContainerStarted","Data":"1f62d99dda910f8d0162198ec4ff6dfed34b81ebc79da4acb5f16ba56c15e442"} Nov 02 17:39:31.715489 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:31.715455 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m" event={"ID":"53e0ed62-81a5-4917-baef-0942cc4a7aee","Type":"ContainerStarted","Data":"75b1461b2e1d1bd992b437817f8a02e77740c8ab0c5aeec85e710c6eda10bfbf"} Nov 02 17:39:31.715990 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:31.715643 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m" Nov 02 17:39:31.717078 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:31.717054 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-5hb2w" event={"ID":"b693ebb5-876a-426b-bf1d-6fdf00d3e96a","Type":"ContainerStarted","Data":"795116a02f6d090dc09a63f9fff16400f64090d41eadd9531880582d94a3cbda"} Nov 02 17:39:31.717213 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:31.717197 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-5hb2w" Nov 02 17:39:31.732326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:31.732277 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m" podStartSLOduration=1.439071348 podStartE2EDuration="3.732265959s" totalImagesPullingTime="2.293194611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:31.731371546 +0000 UTC m=+343.519877089" watchObservedRunningTime="2026-11-02 17:39:31.732265959 +0000 UTC m=+343.520771486" Nov 02 17:39:31.747655 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:31.747606 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-5hb2w" podStartSLOduration=1.386161488 podStartE2EDuration="3.747593691s" totalImagesPullingTime="2.361432203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:31.746584981 +0000 UTC m=+343.535090508" watchObservedRunningTime="2026-11-02 17:39:31.747593691 +0000 UTC m=+343.536099219" Nov 02 17:39:39.706817 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:39.706783 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-7rzb7" Nov 02 17:39:39.967541 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:39.967462 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-56bcd69c95-g7ngq"] Nov 02 17:39:39.971131 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:39.971114 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-56bcd69c95-g7ngq" Nov 02 17:39:39.990223 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:39.990199 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-56bcd69c95-g7ngq"] Nov 02 17:39:40.067097 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:40.067064 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54-metrics-certs\") pod \"ai-gateway-controller-56bcd69c95-g7ngq\" (UID: \"b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54\") " pod="opendatahub/ai-gateway-controller-56bcd69c95-g7ngq" Nov 02 17:39:40.067280 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:40.067188 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qx7v\" (UniqueName: \"kubernetes.io/projected/b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54-kube-api-access-2qx7v\") pod \"ai-gateway-controller-56bcd69c95-g7ngq\" (UID: \"b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54\") " pod="opendatahub/ai-gateway-controller-56bcd69c95-g7ngq" Nov 02 17:39:40.167909 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:40.167878 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54-metrics-certs\") pod \"ai-gateway-controller-56bcd69c95-g7ngq\" (UID: \"b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54\") " pod="opendatahub/ai-gateway-controller-56bcd69c95-g7ngq" Nov 02 17:39:40.168086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:40.167993 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qx7v\" (UniqueName: \"kubernetes.io/projected/b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54-kube-api-access-2qx7v\") pod \"ai-gateway-controller-56bcd69c95-g7ngq\" (UID: \"b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54\") " pod="opendatahub/ai-gateway-controller-56bcd69c95-g7ngq" Nov 02 17:39:40.170345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:40.170324 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54-metrics-certs\") pod \"ai-gateway-controller-56bcd69c95-g7ngq\" (UID: \"b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54\") " pod="opendatahub/ai-gateway-controller-56bcd69c95-g7ngq" Nov 02 17:39:40.177348 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:40.177320 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qx7v\" (UniqueName: \"kubernetes.io/projected/b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54-kube-api-access-2qx7v\") pod \"ai-gateway-controller-56bcd69c95-g7ngq\" (UID: \"b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54\") " pod="opendatahub/ai-gateway-controller-56bcd69c95-g7ngq" Nov 02 17:39:40.283784 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:40.283701 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-56bcd69c95-g7ngq" Nov 02 17:39:40.552418 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:40.552363 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-56bcd69c95-g7ngq"] Nov 02 17:39:40.555060 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:39:40.555027 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b0f604_ac8f_4ea4_8fbd_28f4789c5c54.slice/crio-09abcb5b04aaadc2211deedb21c836124f9adcbb961d391013def0d51ad609df WatchSource:0}: Error finding container 09abcb5b04aaadc2211deedb21c836124f9adcbb961d391013def0d51ad609df: Status 404 returned error can't find the container with id 09abcb5b04aaadc2211deedb21c836124f9adcbb961d391013def0d51ad609df Nov 02 17:39:40.755220 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:40.755185 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-56bcd69c95-g7ngq" event={"ID":"b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54","Type":"ContainerStarted","Data":"09abcb5b04aaadc2211deedb21c836124f9adcbb961d391013def0d51ad609df"} Nov 02 17:39:41.760554 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:41.760516 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-56bcd69c95-g7ngq" event={"ID":"b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54","Type":"ContainerStarted","Data":"7211501b643f63800a6a419e295a72c4c1dafb0bacc024bb09a8506f6399ea8e"} Nov 02 17:39:41.760980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:41.760596 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-56bcd69c95-g7ngq" Nov 02 17:39:41.780660 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:41.780615 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-56bcd69c95-g7ngq" podStartSLOduration=2.3750979660000002 podStartE2EDuration="2.780603104s" totalImagesPullingTime="405.505138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:41.779404887 +0000 UTC m=+353.567910416" watchObservedRunningTime="2026-11-02 17:39:41.780603104 +0000 UTC m=+353.569108655" Nov 02 17:39:42.724075 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:42.724049 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-5hb2w" Nov 02 17:39:42.724217 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:42.724128 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m" Nov 02 17:39:47.585168 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:39:47.585131 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:39:52.767609 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:52.767575 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-56bcd69c95-g7ngq" Nov 02 17:39:52.817767 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:52.817731 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-7rzb7"] Nov 02 17:39:52.817991 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:52.817944 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-7rzb7" podUID="05961055-fed0-463a-acd4-58784cfe3d5a" containerName="manager" containerID="cri-o://19d0e7d3f73655919559a6cee5243852deffddf1a4cd743f3ab39f9933d6120d" gracePeriod=30 Nov 02 17:39:53.056242 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:53.056216 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-7rzb7" Nov 02 17:39:53.185267 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:53.185224 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05961055-fed0-463a-acd4-58784cfe3d5a-kube-api-access-rw2t6\" (UniqueName: \"kubernetes.io/projected/05961055-fed0-463a-acd4-58784cfe3d5a-kube-api-access-rw2t6\") pod \"05961055-fed0-463a-acd4-58784cfe3d5a\" (UID: \"05961055-fed0-463a-acd4-58784cfe3d5a\") " Nov 02 17:39:53.185432 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:53.185295 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05961055-fed0-463a-acd4-58784cfe3d5a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/05961055-fed0-463a-acd4-58784cfe3d5a-metrics-certs\") pod \"05961055-fed0-463a-acd4-58784cfe3d5a\" (UID: \"05961055-fed0-463a-acd4-58784cfe3d5a\") " Nov 02 17:39:53.187432 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:53.187399 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05961055-fed0-463a-acd4-58784cfe3d5a-kube-api-access-rw2t6" pod "05961055-fed0-463a-acd4-58784cfe3d5a" (UID: "05961055-fed0-463a-acd4-58784cfe3d5a"). InnerVolumeSpecName "kube-api-access-rw2t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:39:53.187535 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:53.187431 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05961055-fed0-463a-acd4-58784cfe3d5a-metrics-certs" pod "05961055-fed0-463a-acd4-58784cfe3d5a" (UID: "05961055-fed0-463a-acd4-58784cfe3d5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:39:53.286458 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:53.286423 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05961055-fed0-463a-acd4-58784cfe3d5a-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:39:53.286458 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:53.286452 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw2t6\" (UniqueName: \"kubernetes.io/projected/05961055-fed0-463a-acd4-58784cfe3d5a-kube-api-access-rw2t6\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:39:53.813331 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:53.813295 5549 generic.go:356] "Generic (PLEG): container finished" podID="05961055-fed0-463a-acd4-58784cfe3d5a" containerID="19d0e7d3f73655919559a6cee5243852deffddf1a4cd743f3ab39f9933d6120d" exitCode=0 Nov 02 17:39:53.813770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:53.813337 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-7rzb7" Nov 02 17:39:53.813770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:53.813400 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-7rzb7" event={"ID":"05961055-fed0-463a-acd4-58784cfe3d5a","Type":"ContainerDied","Data":"19d0e7d3f73655919559a6cee5243852deffddf1a4cd743f3ab39f9933d6120d"} Nov 02 17:39:53.813770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:53.813450 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-7rzb7" event={"ID":"05961055-fed0-463a-acd4-58784cfe3d5a","Type":"ContainerDied","Data":"b11ecc4122c711d97335a2ef95fe9f3118062ebd10a48a3e9dc299117aebbc14"} Nov 02 17:39:53.813770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:53.813475 5549 scope.go:122] "RemoveContainer" containerID="19d0e7d3f73655919559a6cee5243852deffddf1a4cd743f3ab39f9933d6120d" Nov 02 17:39:53.823108 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:53.822840 5549 scope.go:122] "RemoveContainer" containerID="19d0e7d3f73655919559a6cee5243852deffddf1a4cd743f3ab39f9933d6120d" Nov 02 17:39:53.823167 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:39:53.823135 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19d0e7d3f73655919559a6cee5243852deffddf1a4cd743f3ab39f9933d6120d\": container with ID starting with 19d0e7d3f73655919559a6cee5243852deffddf1a4cd743f3ab39f9933d6120d not found: ID does not exist" containerID="19d0e7d3f73655919559a6cee5243852deffddf1a4cd743f3ab39f9933d6120d" Nov 02 17:39:53.823204 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:53.823157 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19d0e7d3f73655919559a6cee5243852deffddf1a4cd743f3ab39f9933d6120d"} err="failed to get container status \"19d0e7d3f73655919559a6cee5243852deffddf1a4cd743f3ab39f9933d6120d\": rpc error: code = NotFound desc = could not find container \"19d0e7d3f73655919559a6cee5243852deffddf1a4cd743f3ab39f9933d6120d\": container with ID starting with 19d0e7d3f73655919559a6cee5243852deffddf1a4cd743f3ab39f9933d6120d not found: ID does not exist" Nov 02 17:39:53.835113 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:53.835092 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-7rzb7"] Nov 02 17:39:53.839911 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:53.839887 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-7rzb7"] Nov 02 17:39:54.316801 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:54.316766 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05961055-fed0-463a-acd4-58784cfe3d5a" path="/var/lib/kubelet/pods/05961055-fed0-463a-acd4-58784cfe3d5a/volumes" Nov 02 17:39:54.499715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:54.499679 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:54.988231 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:54.988166 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:56.215025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:56.214990 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:56.357328 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:56.357289 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:57.444186 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:57.444150 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:59.383622 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.383591 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg"] Nov 02 17:39:59.384032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.383944 5549 cpu_manager.go:414] "containerMap: removing container" podUID="05961055-fed0-463a-acd4-58784cfe3d5a" containerName="manager" Nov 02 17:39:59.384032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.383971 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05961055-fed0-463a-acd4-58784cfe3d5a" containerName="manager" Nov 02 17:39:59.384110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.384050 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="05961055-fed0-463a-acd4-58784cfe3d5a" containerName="manager" Nov 02 17:39:59.388562 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.388544 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.397345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.397324 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 02 17:39:59.403833 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.403788 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 17:39:59.403833 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.403815 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 17:39:59.404886 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.404868 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg"] Nov 02 17:39:59.495640 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.495605 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dnkjz\"" Nov 02 17:39:59.500075 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.500046 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:59.541891 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.541859 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/310f7705-ba7a-44d3-9c98-6842c0c64c24-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg\" (UID: \"310f7705-ba7a-44d3-9c98-6842c0c64c24\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.542059 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.541915 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/310f7705-ba7a-44d3-9c98-6842c0c64c24-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg\" (UID: \"310f7705-ba7a-44d3-9c98-6842c0c64c24\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.542059 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.541997 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/310f7705-ba7a-44d3-9c98-6842c0c64c24-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg\" (UID: \"310f7705-ba7a-44d3-9c98-6842c0c64c24\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.542059 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.542037 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/310f7705-ba7a-44d3-9c98-6842c0c64c24-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg\" (UID: \"310f7705-ba7a-44d3-9c98-6842c0c64c24\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.542059 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.542059 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/310f7705-ba7a-44d3-9c98-6842c0c64c24-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg\" (UID: \"310f7705-ba7a-44d3-9c98-6842c0c64c24\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.542199 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.542152 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-972ml\" (UniqueName: \"kubernetes.io/projected/310f7705-ba7a-44d3-9c98-6842c0c64c24-kube-api-access-972ml\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg\" (UID: \"310f7705-ba7a-44d3-9c98-6842c0c64c24\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.642609 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.642536 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/310f7705-ba7a-44d3-9c98-6842c0c64c24-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg\" (UID: \"310f7705-ba7a-44d3-9c98-6842c0c64c24\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.642609 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.642572 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/310f7705-ba7a-44d3-9c98-6842c0c64c24-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg\" (UID: \"310f7705-ba7a-44d3-9c98-6842c0c64c24\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.642609 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.642599 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/310f7705-ba7a-44d3-9c98-6842c0c64c24-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg\" (UID: \"310f7705-ba7a-44d3-9c98-6842c0c64c24\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.642836 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.642621 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/310f7705-ba7a-44d3-9c98-6842c0c64c24-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg\" (UID: \"310f7705-ba7a-44d3-9c98-6842c0c64c24\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.642836 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.642644 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-972ml\" (UniqueName: \"kubernetes.io/projected/310f7705-ba7a-44d3-9c98-6842c0c64c24-kube-api-access-972ml\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg\" (UID: \"310f7705-ba7a-44d3-9c98-6842c0c64c24\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.642836 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.642681 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/310f7705-ba7a-44d3-9c98-6842c0c64c24-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg\" (UID: \"310f7705-ba7a-44d3-9c98-6842c0c64c24\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.643060 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.643036 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/310f7705-ba7a-44d3-9c98-6842c0c64c24-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg\" (UID: \"310f7705-ba7a-44d3-9c98-6842c0c64c24\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.643106 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.643064 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/310f7705-ba7a-44d3-9c98-6842c0c64c24-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg\" (UID: \"310f7705-ba7a-44d3-9c98-6842c0c64c24\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.643106 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.643072 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/310f7705-ba7a-44d3-9c98-6842c0c64c24-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg\" (UID: \"310f7705-ba7a-44d3-9c98-6842c0c64c24\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.644977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.644934 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/310f7705-ba7a-44d3-9c98-6842c0c64c24-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg\" (UID: \"310f7705-ba7a-44d3-9c98-6842c0c64c24\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.645287 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.645269 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/310f7705-ba7a-44d3-9c98-6842c0c64c24-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg\" (UID: \"310f7705-ba7a-44d3-9c98-6842c0c64c24\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.651020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.651000 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-972ml\" (UniqueName: \"kubernetes.io/projected/310f7705-ba7a-44d3-9c98-6842c0c64c24-kube-api-access-972ml\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg\" (UID: \"310f7705-ba7a-44d3-9c98-6842c0c64c24\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.701601 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.701567 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:39:59.841275 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.841243 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:39:59.975044 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:39:59.975003 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg"] Nov 02 17:39:59.977709 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:39:59.977682 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310f7705_ba7a_44d3_9c98_6842c0c64c24.slice/crio-22bee31afeea0d72e4cab521d7b8f30c2509bcb04a63d8dfff4b145d4b281baa WatchSource:0}: Error finding container 22bee31afeea0d72e4cab521d7b8f30c2509bcb04a63d8dfff4b145d4b281baa: Status 404 returned error can't find the container with id 22bee31afeea0d72e4cab521d7b8f30c2509bcb04a63d8dfff4b145d4b281baa Nov 02 17:40:00.846199 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:00.846157 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" event={"ID":"310f7705-ba7a-44d3-9c98-6842c0c64c24","Type":"ContainerStarted","Data":"22bee31afeea0d72e4cab521d7b8f30c2509bcb04a63d8dfff4b145d4b281baa"} Nov 02 17:40:01.741147 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.741073 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl"] Nov 02 17:40:01.744402 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.744379 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:01.751881 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.751859 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 02 17:40:01.755768 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.755745 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl"] Nov 02 17:40:01.831161 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.831129 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:01.851969 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.851925 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" event={"ID":"310f7705-ba7a-44d3-9c98-6842c0c64c24","Type":"ContainerStarted","Data":"e5853d4cfa72432e1b3032ba293d3e19de0ebc052b0df69d622a79082c378ab2"} Nov 02 17:40:01.862489 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.862461 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c4743d-7414-4aa5-b190-fecec685ae2c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r7qcl\" (UID: \"e4c4743d-7414-4aa5-b190-fecec685ae2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:01.862653 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.862624 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c4743d-7414-4aa5-b190-fecec685ae2c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r7qcl\" (UID: \"e4c4743d-7414-4aa5-b190-fecec685ae2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:01.862742 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.862710 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c4743d-7414-4aa5-b190-fecec685ae2c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r7qcl\" (UID: \"e4c4743d-7414-4aa5-b190-fecec685ae2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:01.862852 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.862834 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b829q\" (UniqueName: \"kubernetes.io/projected/e4c4743d-7414-4aa5-b190-fecec685ae2c-kube-api-access-b829q\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r7qcl\" (UID: \"e4c4743d-7414-4aa5-b190-fecec685ae2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:01.862925 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.862908 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c4743d-7414-4aa5-b190-fecec685ae2c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r7qcl\" (UID: \"e4c4743d-7414-4aa5-b190-fecec685ae2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:01.863002 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.862945 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c4743d-7414-4aa5-b190-fecec685ae2c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r7qcl\" (UID: \"e4c4743d-7414-4aa5-b190-fecec685ae2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:01.872370 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.872327 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" podStartSLOduration=1.397914133 podStartE2EDuration="2.872314655s" totalImagesPullingTime="1.474400522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:01.87103041 +0000 UTC m=+373.659535938" watchObservedRunningTime="2026-11-02 17:40:01.872314655 +0000 UTC m=+373.660820183" Nov 02 17:40:01.963857 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.963831 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c4743d-7414-4aa5-b190-fecec685ae2c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r7qcl\" (UID: \"e4c4743d-7414-4aa5-b190-fecec685ae2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:01.964009 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.963900 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b829q\" (UniqueName: \"kubernetes.io/projected/e4c4743d-7414-4aa5-b190-fecec685ae2c-kube-api-access-b829q\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r7qcl\" (UID: \"e4c4743d-7414-4aa5-b190-fecec685ae2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:01.964009 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.963936 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c4743d-7414-4aa5-b190-fecec685ae2c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r7qcl\" (UID: \"e4c4743d-7414-4aa5-b190-fecec685ae2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:01.964009 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.963972 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c4743d-7414-4aa5-b190-fecec685ae2c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r7qcl\" (UID: \"e4c4743d-7414-4aa5-b190-fecec685ae2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:01.964175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.964012 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c4743d-7414-4aa5-b190-fecec685ae2c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r7qcl\" (UID: \"e4c4743d-7414-4aa5-b190-fecec685ae2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:01.964175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.964029 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c4743d-7414-4aa5-b190-fecec685ae2c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r7qcl\" (UID: \"e4c4743d-7414-4aa5-b190-fecec685ae2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:01.964417 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.964390 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c4743d-7414-4aa5-b190-fecec685ae2c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r7qcl\" (UID: \"e4c4743d-7414-4aa5-b190-fecec685ae2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:01.964546 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.964449 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c4743d-7414-4aa5-b190-fecec685ae2c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r7qcl\" (UID: \"e4c4743d-7414-4aa5-b190-fecec685ae2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:01.964668 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.964651 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c4743d-7414-4aa5-b190-fecec685ae2c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r7qcl\" (UID: \"e4c4743d-7414-4aa5-b190-fecec685ae2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:01.966321 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.966298 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c4743d-7414-4aa5-b190-fecec685ae2c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r7qcl\" (UID: \"e4c4743d-7414-4aa5-b190-fecec685ae2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:01.966474 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.966457 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c4743d-7414-4aa5-b190-fecec685ae2c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r7qcl\" (UID: \"e4c4743d-7414-4aa5-b190-fecec685ae2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:01.973480 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:01.973458 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b829q\" (UniqueName: \"kubernetes.io/projected/e4c4743d-7414-4aa5-b190-fecec685ae2c-kube-api-access-b829q\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r7qcl\" (UID: \"e4c4743d-7414-4aa5-b190-fecec685ae2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:02.057386 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:02.057317 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:02.323501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:02.323427 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:40:02.333862 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:02.333840 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl"] Nov 02 17:40:02.336907 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:40:02.336873 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c4743d_7414_4aa5_b190_fecec685ae2c.slice/crio-afe9229a32c25b43aa20a5a19f8cde7817582c27598d4b2e805d28eed7c47526 WatchSource:0}: Error finding container afe9229a32c25b43aa20a5a19f8cde7817582c27598d4b2e805d28eed7c47526: Status 404 returned error can't find the container with id afe9229a32c25b43aa20a5a19f8cde7817582c27598d4b2e805d28eed7c47526 Nov 02 17:40:02.701967 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:02.701933 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:40:02.857937 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:02.857895 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" event={"ID":"e4c4743d-7414-4aa5-b190-fecec685ae2c","Type":"ContainerStarted","Data":"7fc7e9c5e146c80dfb455c116ea70dd8d9f115dfbc833a52aae876d00298a193"} Nov 02 17:40:02.858353 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:02.857978 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" event={"ID":"e4c4743d-7414-4aa5-b190-fecec685ae2c","Type":"ContainerStarted","Data":"afe9229a32c25b43aa20a5a19f8cde7817582c27598d4b2e805d28eed7c47526"} Nov 02 17:40:02.878865 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:02.878818 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" podStartSLOduration=1.56179439 podStartE2EDuration="1.878803916s" totalImagesPullingTime="317.009526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:02.877473845 +0000 UTC m=+374.665979372" watchObservedRunningTime="2026-11-02 17:40:02.878803916 +0000 UTC m=+374.667309443" Nov 02 17:40:03.057579 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:03.057547 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:03.405044 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:03.404967 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:04.159293 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.159258 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:40:04.331459 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.331422 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9"] Nov 02 17:40:04.334843 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.334821 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.343279 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.342963 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 02 17:40:04.351450 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.351425 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9"] Nov 02 17:40:04.436307 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.436227 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:04.483394 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.483362 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1bb0ca2-6b43-4068-be3b-f44f74fe31fa-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-flgk9\" (UID: \"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.483394 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.483397 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1bb0ca2-6b43-4068-be3b-f44f74fe31fa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-flgk9\" (UID: \"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.483583 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.483426 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44nxf\" (UniqueName: \"kubernetes.io/projected/c1bb0ca2-6b43-4068-be3b-f44f74fe31fa-kube-api-access-44nxf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-flgk9\" (UID: \"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.483583 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.483524 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1bb0ca2-6b43-4068-be3b-f44f74fe31fa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-flgk9\" (UID: \"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.483583 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.483550 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1bb0ca2-6b43-4068-be3b-f44f74fe31fa-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-flgk9\" (UID: \"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.483583 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.483571 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1bb0ca2-6b43-4068-be3b-f44f74fe31fa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-flgk9\" (UID: \"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.585018 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.584987 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1bb0ca2-6b43-4068-be3b-f44f74fe31fa-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-flgk9\" (UID: \"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.585186 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.585024 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1bb0ca2-6b43-4068-be3b-f44f74fe31fa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-flgk9\" (UID: \"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.585186 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.585054 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44nxf\" (UniqueName: \"kubernetes.io/projected/c1bb0ca2-6b43-4068-be3b-f44f74fe31fa-kube-api-access-44nxf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-flgk9\" (UID: \"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.585186 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.585081 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1bb0ca2-6b43-4068-be3b-f44f74fe31fa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-flgk9\" (UID: \"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.585186 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.585105 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1bb0ca2-6b43-4068-be3b-f44f74fe31fa-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-flgk9\" (UID: \"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.585186 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.585126 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1bb0ca2-6b43-4068-be3b-f44f74fe31fa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-flgk9\" (UID: \"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.585556 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.585514 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1bb0ca2-6b43-4068-be3b-f44f74fe31fa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-flgk9\" (UID: \"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.585659 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.585529 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1bb0ca2-6b43-4068-be3b-f44f74fe31fa-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-flgk9\" (UID: \"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.585659 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.585567 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1bb0ca2-6b43-4068-be3b-f44f74fe31fa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-flgk9\" (UID: \"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.587389 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.587363 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1bb0ca2-6b43-4068-be3b-f44f74fe31fa-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-flgk9\" (UID: \"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.587632 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.587616 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1bb0ca2-6b43-4068-be3b-f44f74fe31fa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-flgk9\" (UID: \"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.595053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.595028 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44nxf\" (UniqueName: \"kubernetes.io/projected/c1bb0ca2-6b43-4068-be3b-f44f74fe31fa-kube-api-access-44nxf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-flgk9\" (UID: \"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.649041 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.649014 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:04.924925 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.924900 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9"] Nov 02 17:40:04.927457 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:40:04.927430 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bb0ca2_6b43_4068_be3b_f44f74fe31fa.slice/crio-1d79bb9103063bd7e82357527cc6dc02daabc3c6052c7a9187264923b464847d WatchSource:0}: Error finding container 1d79bb9103063bd7e82357527cc6dc02daabc3c6052c7a9187264923b464847d: Status 404 returned error can't find the container with id 1d79bb9103063bd7e82357527cc6dc02daabc3c6052c7a9187264923b464847d Nov 02 17:40:04.966523 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:04.966450 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:40:05.258907 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:05.258877 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:05.876576 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:05.876539 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" event={"ID":"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa","Type":"ContainerStarted","Data":"ae35d32dbee00f5d33958fd2cf38a05fc3974c3e579af96cdf0f81ac54273be8"} Nov 02 17:40:05.876724 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:05.876597 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" event={"ID":"c1bb0ca2-6b43-4068-be3b-f44f74fe31fa","Type":"ContainerStarted","Data":"1d79bb9103063bd7e82357527cc6dc02daabc3c6052c7a9187264923b464847d"} Nov 02 17:40:05.894917 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:05.894861 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" podStartSLOduration=1.6080834080000002 podStartE2EDuration="1.894844432s" totalImagesPullingTime="286.761024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:05.894456391 +0000 UTC m=+377.682961918" watchObservedRunningTime="2026-11-02 17:40:05.894844432 +0000 UTC m=+377.683349961" Nov 02 17:40:06.056708 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:06.056674 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:06.649697 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:06.649643 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:07.516345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.516314 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l"] Nov 02 17:40:07.519909 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.519887 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.531184 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.531163 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 02 17:40:07.534909 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.534885 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l"] Nov 02 17:40:07.612813 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.612780 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scr99\" (UniqueName: \"kubernetes.io/projected/f171b02f-9c9a-489c-a7cd-75cfdff945e5-kube-api-access-scr99\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l\" (UID: \"f171b02f-9c9a-489c-a7cd-75cfdff945e5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.612947 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.612859 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f171b02f-9c9a-489c-a7cd-75cfdff945e5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l\" (UID: \"f171b02f-9c9a-489c-a7cd-75cfdff945e5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.612947 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.612934 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f171b02f-9c9a-489c-a7cd-75cfdff945e5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l\" (UID: \"f171b02f-9c9a-489c-a7cd-75cfdff945e5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.613063 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.612986 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f171b02f-9c9a-489c-a7cd-75cfdff945e5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l\" (UID: \"f171b02f-9c9a-489c-a7cd-75cfdff945e5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.613063 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.613050 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f171b02f-9c9a-489c-a7cd-75cfdff945e5-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l\" (UID: \"f171b02f-9c9a-489c-a7cd-75cfdff945e5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.613138 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.613078 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f171b02f-9c9a-489c-a7cd-75cfdff945e5-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l\" (UID: \"f171b02f-9c9a-489c-a7cd-75cfdff945e5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.618774 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.618731 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:07.714535 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.714502 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f171b02f-9c9a-489c-a7cd-75cfdff945e5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l\" (UID: \"f171b02f-9c9a-489c-a7cd-75cfdff945e5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.714535 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.714535 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f171b02f-9c9a-489c-a7cd-75cfdff945e5-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l\" (UID: \"f171b02f-9c9a-489c-a7cd-75cfdff945e5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.715062 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.714558 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f171b02f-9c9a-489c-a7cd-75cfdff945e5-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l\" (UID: \"f171b02f-9c9a-489c-a7cd-75cfdff945e5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.715062 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.714578 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scr99\" (UniqueName: \"kubernetes.io/projected/f171b02f-9c9a-489c-a7cd-75cfdff945e5-kube-api-access-scr99\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l\" (UID: \"f171b02f-9c9a-489c-a7cd-75cfdff945e5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.715062 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.714623 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f171b02f-9c9a-489c-a7cd-75cfdff945e5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l\" (UID: \"f171b02f-9c9a-489c-a7cd-75cfdff945e5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.715062 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.714662 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f171b02f-9c9a-489c-a7cd-75cfdff945e5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l\" (UID: \"f171b02f-9c9a-489c-a7cd-75cfdff945e5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.715062 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.715005 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f171b02f-9c9a-489c-a7cd-75cfdff945e5-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l\" (UID: \"f171b02f-9c9a-489c-a7cd-75cfdff945e5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.715279 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.715121 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f171b02f-9c9a-489c-a7cd-75cfdff945e5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l\" (UID: \"f171b02f-9c9a-489c-a7cd-75cfdff945e5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.715279 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.715217 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f171b02f-9c9a-489c-a7cd-75cfdff945e5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l\" (UID: \"f171b02f-9c9a-489c-a7cd-75cfdff945e5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.716966 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.716928 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f171b02f-9c9a-489c-a7cd-75cfdff945e5-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l\" (UID: \"f171b02f-9c9a-489c-a7cd-75cfdff945e5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.717140 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.717124 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f171b02f-9c9a-489c-a7cd-75cfdff945e5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l\" (UID: \"f171b02f-9c9a-489c-a7cd-75cfdff945e5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.722884 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.722862 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scr99\" (UniqueName: \"kubernetes.io/projected/f171b02f-9c9a-489c-a7cd-75cfdff945e5-kube-api-access-scr99\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l\" (UID: \"f171b02f-9c9a-489c-a7cd-75cfdff945e5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:07.833989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:07.833901 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:08.108476 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:08.108453 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l"] Nov 02 17:40:08.111092 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:40:08.111063 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf171b02f_9c9a_489c_a7cd_75cfdff945e5.slice/crio-87f3f74d692e5ea5f64cff56684202fbc741c9eca6ee778da035657dc547b830 WatchSource:0}: Error finding container 87f3f74d692e5ea5f64cff56684202fbc741c9eca6ee778da035657dc547b830: Status 404 returned error can't find the container with id 87f3f74d692e5ea5f64cff56684202fbc741c9eca6ee778da035657dc547b830 Nov 02 17:40:08.246019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:08.245984 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:08.893438 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:08.893401 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" event={"ID":"f171b02f-9c9a-489c-a7cd-75cfdff945e5","Type":"ContainerStarted","Data":"5e8a02873a9af937990a7b21f30b13eaeaa94fe0ebbae3c9c0ce3e99e058ca80"} Nov 02 17:40:08.893816 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:08.893455 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" event={"ID":"f171b02f-9c9a-489c-a7cd-75cfdff945e5","Type":"ContainerStarted","Data":"87f3f74d692e5ea5f64cff56684202fbc741c9eca6ee778da035657dc547b830"} Nov 02 17:40:08.911895 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:08.911847 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" podStartSLOduration=1.615653282 podStartE2EDuration="1.911835005s" totalImagesPullingTime="296.181723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:08.910585925 +0000 UTC m=+380.699091469" watchObservedRunningTime="2026-11-02 17:40:08.911835005 +0000 UTC m=+380.700340532" Nov 02 17:40:09.660642 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:09.660610 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:40:09.701761 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:09.701737 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:40:09.713845 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:09.713822 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:40:09.834343 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:09.834319 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:09.915447 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:09.915375 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg" Nov 02 17:40:12.058193 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:12.058158 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:12.070465 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:12.070439 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:12.923510 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:12.923480 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r7qcl" Nov 02 17:40:14.649935 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:14.649894 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:14.662365 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:14.662344 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:14.931728 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:14.931649 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-flgk9" Nov 02 17:40:17.586008 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:40:17.585920 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:40:17.834349 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:17.834314 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:17.846625 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:17.846552 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:17.944942 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:17.944913 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l" Nov 02 17:40:21.952702 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:21.952671 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc"] Nov 02 17:40:21.956650 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:21.956620 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:21.968983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:21.968925 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc"] Nov 02 17:40:22.025493 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.025457 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.025679 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.025500 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.025679 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.025518 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.025679 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.025578 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.025679 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.025633 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.025864 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.025682 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcsnc\" (UniqueName: \"kubernetes.io/projected/5639111b-8dcd-4b4a-a845-f21825bda0d9-kube-api-access-tcsnc\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.025864 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.025735 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.025864 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.025790 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.026024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.025895 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5639111b-8dcd-4b4a-a845-f21825bda0d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.051291 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.051264 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qbv6v\"" Nov 02 17:40:22.127151 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.127110 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.127344 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.127179 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.127344 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.127219 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5639111b-8dcd-4b4a-a845-f21825bda0d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.127344 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.127252 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.127344 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.127276 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.127344 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.127296 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.127616 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.127511 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.127616 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.127573 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.127616 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.127600 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcsnc\" (UniqueName: \"kubernetes.io/projected/5639111b-8dcd-4b4a-a845-f21825bda0d9-kube-api-access-tcsnc\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.127770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.127711 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.127770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.127736 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.127882 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.127803 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.128025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.127987 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.128150 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.128085 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5639111b-8dcd-4b4a-a845-f21825bda0d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.129748 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.129729 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.129902 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.129885 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.135260 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.135235 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.136007 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.135987 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcsnc\" (UniqueName: \"kubernetes.io/projected/5639111b-8dcd-4b4a-a845-f21825bda0d9-kube-api-access-tcsnc\") pod \"maas-default-gateway-openshift-default-65c448b547-vbrfc\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.273208 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.273126 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:22.551269 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.551182 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc"] Nov 02 17:40:22.554609 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:40:22.554578 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5639111b_8dcd_4b4a_a845_f21825bda0d9.slice/crio-0aa719953778a77b6edd72d3ff1cc4113ccb6262b13fd7c93bd23c7b56e0e50d WatchSource:0}: Error finding container 0aa719953778a77b6edd72d3ff1cc4113ccb6262b13fd7c93bd23c7b56e0e50d: Status 404 returned error can't find the container with id 0aa719953778a77b6edd72d3ff1cc4113ccb6262b13fd7c93bd23c7b56e0e50d Nov 02 17:40:22.958926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:22.958885 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" event={"ID":"5639111b-8dcd-4b4a-a845-f21825bda0d9","Type":"ContainerStarted","Data":"0aa719953778a77b6edd72d3ff1cc4113ccb6262b13fd7c93bd23c7b56e0e50d"} Nov 02 17:40:24.996413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:24.996372 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:25.150203 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:25.150160 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:25.386925 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:25.386894 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:40:25.387030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:25.386980 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:40:25.387030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:25.387015 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:40:25.976654 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:25.976616 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" event={"ID":"5639111b-8dcd-4b4a-a845-f21825bda0d9","Type":"ContainerStarted","Data":"7901a099298e95a847238d46d4ffe24d63475a2d9809b3a8fcee3557307afc66"} Nov 02 17:40:26.273949 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:26.273861 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:26.275362 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:26.275335 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-vbrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:40:26.275475 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:26.275384 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" podUID="5639111b-8dcd-4b4a-a845-f21825bda0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 17:40:26.576298 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:26.576207 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:26.719523 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:26.719491 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:27.273449 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:27.273407 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-vbrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:40:27.273661 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:27.273465 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" podUID="5639111b-8dcd-4b4a-a845-f21825bda0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 17:40:28.273926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:28.273884 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-vbrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:40:28.274331 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:28.273933 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" podUID="5639111b-8dcd-4b4a-a845-f21825bda0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 17:40:29.273415 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:29.273377 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-vbrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:40:29.273594 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:29.273435 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" podUID="5639111b-8dcd-4b4a-a845-f21825bda0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 17:40:30.274320 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:30.274282 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-vbrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:40:30.274776 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:30.274341 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" podUID="5639111b-8dcd-4b4a-a845-f21825bda0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 17:40:31.273877 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:31.273841 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-vbrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:40:31.274080 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:31.273893 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" podUID="5639111b-8dcd-4b4a-a845-f21825bda0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 17:40:32.274078 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:32.274030 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-vbrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:40:32.274468 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:32.274106 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" podUID="5639111b-8dcd-4b4a-a845-f21825bda0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 17:40:33.908442 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:33.908406 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:33.909087 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:33.909057 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:33.909787 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:33.909767 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 17:40:33.931280 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:33.931236 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" podStartSLOduration=10.102352984 podStartE2EDuration="12.931223842s" totalImagesPullingTime="2.828870858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:26.005557067 +0000 UTC m=+397.794062595" watchObservedRunningTime="2026-11-02 17:40:33.931223842 +0000 UTC m=+405.719729439" Nov 02 17:40:35.104695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:35.104659 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:36.971788 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:36.971752 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd"] Nov 02 17:40:36.975487 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:36.975463 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:36.985840 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:36.985814 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd"] Nov 02 17:40:37.092024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.091988 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz4xs\" (UniqueName: \"kubernetes.io/projected/781f545f-9659-4d21-9e53-85d4b82a7214-kube-api-access-tz4xs\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.092206 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.092092 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781f545f-9659-4d21-9e53-85d4b82a7214-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.092206 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.092155 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.092206 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.092173 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.092331 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.092252 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.092331 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.092302 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781f545f-9659-4d21-9e53-85d4b82a7214-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.092331 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.092328 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.092435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.092360 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781f545f-9659-4d21-9e53-85d4b82a7214-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.092435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.092395 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.193774 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.193736 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz4xs\" (UniqueName: \"kubernetes.io/projected/781f545f-9659-4d21-9e53-85d4b82a7214-kube-api-access-tz4xs\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.193994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.193795 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781f545f-9659-4d21-9e53-85d4b82a7214-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.193994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.193825 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.193994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.193842 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.193994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.193862 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.193994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.193898 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781f545f-9659-4d21-9e53-85d4b82a7214-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.193994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.193928 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.193994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.193985 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781f545f-9659-4d21-9e53-85d4b82a7214-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.194441 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.194010 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.194441 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.194310 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.194545 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.194506 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.194602 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.194580 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.194650 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.194614 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.194797 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.194773 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781f545f-9659-4d21-9e53-85d4b82a7214-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.196493 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.196465 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.196626 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.196608 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781f545f-9659-4d21-9e53-85d4b82a7214-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.202811 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.202787 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz4xs\" (UniqueName: \"kubernetes.io/projected/781f545f-9659-4d21-9e53-85d4b82a7214-kube-api-access-tz4xs\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.202913 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.202808 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781f545f-9659-4d21-9e53-85d4b82a7214-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-5lphd\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.290752 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.290656 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:37.564075 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.564048 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd"] Nov 02 17:40:37.566616 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:40:37.566589 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781f545f_9659_4d21_9e53_85d4b82a7214.slice/crio-4077a1b79e676a2a1b01504b46d2a9648c7f7abe5243d3f69ce03da25966e97d WatchSource:0}: Error finding container 4077a1b79e676a2a1b01504b46d2a9648c7f7abe5243d3f69ce03da25966e97d: Status 404 returned error can't find the container with id 4077a1b79e676a2a1b01504b46d2a9648c7f7abe5243d3f69ce03da25966e97d Nov 02 17:40:37.569766 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.569726 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:40:37.569845 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.569813 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:40:37.569882 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:37.569857 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:40:38.056881 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:38.056838 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" event={"ID":"781f545f-9659-4d21-9e53-85d4b82a7214","Type":"ContainerStarted","Data":"78366fe1c80b10886716cfa09181443404c28b41f9d40e19e0a91d9ff0a26d3d"} Nov 02 17:40:38.057309 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:38.056931 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" event={"ID":"781f545f-9659-4d21-9e53-85d4b82a7214","Type":"ContainerStarted","Data":"4077a1b79e676a2a1b01504b46d2a9648c7f7abe5243d3f69ce03da25966e97d"} Nov 02 17:40:38.098978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:38.089700 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" podStartSLOduration=2.089681807 podStartE2EDuration="2.089681807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:38.078396993 +0000 UTC m=+409.866902547" watchObservedRunningTime="2026-11-02 17:40:38.089681807 +0000 UTC m=+409.878187337" Nov 02 17:40:38.301978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:38.300930 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5lphd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:40:38.301978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:38.301022 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:38.301978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:38.301031 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" podUID="781f545f-9659-4d21-9e53-85d4b82a7214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 17:40:39.298467 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:39.298423 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5lphd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:40:39.299106 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:39.299068 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" podUID="781f545f-9659-4d21-9e53-85d4b82a7214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 17:40:40.303981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:40.303059 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5lphd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:40:40.303981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:40.303134 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" podUID="781f545f-9659-4d21-9e53-85d4b82a7214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 17:40:41.297511 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:41.297466 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5lphd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:40:41.297804 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:41.297771 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" podUID="781f545f-9659-4d21-9e53-85d4b82a7214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 17:40:42.296411 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:42.296366 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5lphd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:40:42.297032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:42.296996 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" podUID="781f545f-9659-4d21-9e53-85d4b82a7214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 17:40:43.298427 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:43.298380 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5lphd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:40:43.299067 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:43.299030 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" podUID="781f545f-9659-4d21-9e53-85d4b82a7214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 17:40:44.291198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:44.291163 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5lphd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:40:44.291385 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:44.291215 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" podUID="781f545f-9659-4d21-9e53-85d4b82a7214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 17:40:45.736220 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:45.736174 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:45.736697 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:45.736534 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:45.737427 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:45.737406 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 17:40:47.590291 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:40:47.590247 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:40:57.443319 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:57.443283 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:58.261526 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.261495 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:58.428189 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.428158 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:58.570562 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.570461 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:58.773616 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.773577 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs"] Nov 02 17:40:58.782611 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.782585 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:58.787279 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.787251 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs"] Nov 02 17:40:58.792919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.792895 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-868a2ee9-openshift-default-dockercfg-qk7n2\"" Nov 02 17:40:58.833083 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.833015 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:58.905919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.905882 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-envoy\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:58.906119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.905998 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlkkl\" (UniqueName: \"kubernetes.io/projected/a2059966-df3d-42a3-b05f-a00593fb40e6-kube-api-access-nlkkl\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:58.906119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.906101 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-podinfo\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:58.906233 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.906156 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-workload-socket\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:58.906233 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.906202 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2059966-df3d-42a3-b05f-a00593fb40e6-istiod-ca-cert\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:58.906233 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.906228 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-workload-certs\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:58.906379 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.906250 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-token\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:58.906379 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.906281 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-data\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:58.906379 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.906336 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-credential-socket\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:58.944621 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.944590 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k"] Nov 02 17:40:58.949212 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.949187 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:58.964102 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.964076 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k"] Nov 02 17:40:58.985350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:58.985323 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:59.007720 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.007692 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-envoy\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.007829 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.007729 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-podinfo\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.007829 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.007785 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-workload-socket\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.007918 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.007841 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b648d1-eab0-4625-bf8a-d708d790ab14-istiod-ca-cert\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.007918 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.007883 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-workload-socket\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.008090 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.008070 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2059966-df3d-42a3-b05f-a00593fb40e6-istiod-ca-cert\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.008136 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.008124 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-workload-certs\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.008172 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.008151 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-podinfo\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.008229 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.008189 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-token\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.008304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.008274 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-data\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.008364 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.008300 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-workload-socket\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.008364 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.008325 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsqsg\" (UniqueName: \"kubernetes.io/projected/43b648d1-eab0-4625-bf8a-d708d790ab14-kube-api-access-jsqsg\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.008364 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.008358 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-workload-certs\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.008521 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.008383 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-data\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.008521 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.008422 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-credential-socket\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.008521 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.008481 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-workload-certs\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.008635 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.008597 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-credential-socket\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.008672 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.008647 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-envoy\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.008672 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.008648 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-data\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.008735 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.008708 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-credential-socket\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.008767 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.008728 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2059966-df3d-42a3-b05f-a00593fb40e6-istiod-ca-cert\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.008810 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.008772 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlkkl\" (UniqueName: \"kubernetes.io/projected/a2059966-df3d-42a3-b05f-a00593fb40e6-kube-api-access-nlkkl\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.008851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.008817 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-token\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.010360 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.010341 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-podinfo\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.010573 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.010558 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-envoy\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.019716 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.019685 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-token\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.020205 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.020182 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlkkl\" (UniqueName: \"kubernetes.io/projected/a2059966-df3d-42a3-b05f-a00593fb40e6-kube-api-access-nlkkl\") pod \"e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.031387 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.031351 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ad831efe-openshift-default-dockercfg-x4rxs\"" Nov 02 17:40:59.102710 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.102625 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:40:59.109485 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.109455 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b648d1-eab0-4625-bf8a-d708d790ab14-istiod-ca-cert\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.109629 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.109518 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-podinfo\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.109629 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.109578 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsqsg\" (UniqueName: \"kubernetes.io/projected/43b648d1-eab0-4625-bf8a-d708d790ab14-kube-api-access-jsqsg\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.109629 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.109597 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-workload-certs\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.109629 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.109611 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-data\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.109840 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.109660 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-credential-socket\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.109840 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.109695 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-token\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.109840 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.109723 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-envoy\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.109840 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.109751 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-workload-socket\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.110288 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.110209 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-workload-socket\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.110288 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.110263 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-credential-socket\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.110486 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.110460 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b648d1-eab0-4625-bf8a-d708d790ab14-istiod-ca-cert\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.110589 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.110563 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-data\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.110724 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.110704 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-workload-certs\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.112192 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.112167 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-envoy\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.112192 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.112182 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-podinfo\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.119164 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.119138 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-token\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.119260 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.119220 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsqsg\" (UniqueName: \"kubernetes.io/projected/43b648d1-eab0-4625-bf8a-d708d790ab14-kube-api-access-jsqsg\") pod \"e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.141629 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.141593 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:59.262304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.262268 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:40:59.302398 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.302366 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:59.394509 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.394484 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs"] Nov 02 17:40:59.396858 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:40:59.396830 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2059966_df3d_42a3_b05f_a00593fb40e6.slice/crio-b856325b0243d9a4041ff82c7635f61301b102cd9ce0a567aef4aa0913e92b3c WatchSource:0}: Error finding container b856325b0243d9a4041ff82c7635f61301b102cd9ce0a567aef4aa0913e92b3c: Status 404 returned error can't find the container with id b856325b0243d9a4041ff82c7635f61301b102cd9ce0a567aef4aa0913e92b3c Nov 02 17:40:59.399213 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.399180 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:40:59.399285 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.399244 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:40:59.399285 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.399273 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:40:59.442188 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.442163 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:59.543490 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.543454 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k"] Nov 02 17:40:59.547049 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:40:59.547006 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b648d1_eab0_4625_bf8a_d708d790ab14.slice/crio-d013c620e60793fb5078770ab64c6363ea0ed824134ac7e3e1c36d977777d95f WatchSource:0}: Error finding container d013c620e60793fb5078770ab64c6363ea0ed824134ac7e3e1c36d977777d95f: Status 404 returned error can't find the container with id d013c620e60793fb5078770ab64c6363ea0ed824134ac7e3e1c36d977777d95f Nov 02 17:40:59.549487 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.549451 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:40:59.549574 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.549523 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:40:59.549574 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:40:59.549555 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:00.162351 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.162310 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" event={"ID":"43b648d1-eab0-4625-bf8a-d708d790ab14","Type":"ContainerStarted","Data":"1397e74de2801f5ba154ac53ba8d5d0d4c40ad7b1688f24b9f71c09e38dce23a"} Nov 02 17:41:00.162777 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.162414 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" event={"ID":"43b648d1-eab0-4625-bf8a-d708d790ab14","Type":"ContainerStarted","Data":"d013c620e60793fb5078770ab64c6363ea0ed824134ac7e3e1c36d977777d95f"} Nov 02 17:41:00.164003 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.163967 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" event={"ID":"a2059966-df3d-42a3-b05f-a00593fb40e6","Type":"ContainerStarted","Data":"6c7ed4dfb7d51c52cdc02f39e97e3b8fd1e3f2996c6ab3242842473787143b6b"} Nov 02 17:41:00.164147 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.164021 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" event={"ID":"a2059966-df3d-42a3-b05f-a00593fb40e6","Type":"ContainerStarted","Data":"b856325b0243d9a4041ff82c7635f61301b102cd9ce0a567aef4aa0913e92b3c"} Nov 02 17:41:00.183861 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.183814 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" podStartSLOduration=2.183803514 podStartE2EDuration="2.183803514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:00.182007603 +0000 UTC m=+431.970513145" watchObservedRunningTime="2026-11-02 17:41:00.183803514 +0000 UTC m=+431.972309025" Nov 02 17:41:00.204447 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.204395 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" podStartSLOduration=2.204381919 podStartE2EDuration="2.204381919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:00.202938214 +0000 UTC m=+431.991443742" watchObservedRunningTime="2026-11-02 17:41:00.204381919 +0000 UTC m=+431.992887504" Nov 02 17:41:00.262662 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.262628 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:41:00.267857 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.267836 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:41:00.283711 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.283676 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:00.636979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.636934 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s"] Nov 02 17:41:00.641470 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.641442 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.651787 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.651754 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0ae9c41e-openshift-default-dockercfg-krnpn\"" Nov 02 17:41:00.654239 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.654211 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s"] Nov 02 17:41:00.723384 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.723343 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:00.728646 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.728613 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-envoy\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.728780 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.728668 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-workload-socket\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.728780 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.728746 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-credential-socket\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.728780 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.728774 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-data\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.728937 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.728795 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59jmz\" (UniqueName: \"kubernetes.io/projected/72f31e15-2da6-4b7e-9073-4457a2a283ee-kube-api-access-59jmz\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.728937 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.728848 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-workload-certs\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.729031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.728941 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-podinfo\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.729031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.729007 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-token\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.729157 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.729139 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f31e15-2da6-4b7e-9073-4457a2a283ee-istiod-ca-cert\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.832615 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.832568 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-credential-socket\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.832801 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.832634 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-data\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.832801 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.832665 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59jmz\" (UniqueName: \"kubernetes.io/projected/72f31e15-2da6-4b7e-9073-4457a2a283ee-kube-api-access-59jmz\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.832801 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.832696 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-workload-certs\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.832801 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.832733 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-podinfo\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.832801 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.832766 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-token\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.833184 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.832831 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f31e15-2da6-4b7e-9073-4457a2a283ee-istiod-ca-cert\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.833184 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.832915 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-envoy\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.833184 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.832984 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-workload-socket\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.833476 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.833454 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-workload-socket\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.833713 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.833689 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-credential-socket\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.833986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.833947 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-data\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.840426 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.834582 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-workload-certs\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.840426 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.837575 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f31e15-2da6-4b7e-9073-4457a2a283ee-istiod-ca-cert\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.840871 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.840848 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-podinfo\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.850492 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.850463 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-token\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.852243 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.852221 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-envoy\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.853440 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.853422 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59jmz\" (UniqueName: \"kubernetes.io/projected/72f31e15-2da6-4b7e-9073-4457a2a283ee-kube-api-access-59jmz\") pod \"e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:00.858514 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.858490 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:00.957582 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:00.957559 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:01.103591 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:01.103557 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:41:01.108387 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:01.108361 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:41:01.169482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:01.169453 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:41:01.169968 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:01.169806 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:41:01.170258 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:01.170241 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:41:01.170529 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:01.170505 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:41:01.243484 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:01.243452 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s"] Nov 02 17:41:01.247117 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:41:01.247087 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f31e15_2da6_4b7e_9073_4457a2a283ee.slice/crio-d32c5587f706d6700514606885668a0e53d29fc53328efaf355174b9f2211a7b WatchSource:0}: Error finding container d32c5587f706d6700514606885668a0e53d29fc53328efaf355174b9f2211a7b: Status 404 returned error can't find the container with id d32c5587f706d6700514606885668a0e53d29fc53328efaf355174b9f2211a7b Nov 02 17:41:01.249791 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:01.249750 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:01.249884 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:01.249838 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:01.249919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:01.249878 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:01.696127 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:01.696098 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:01.849924 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:01.849885 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:01.991998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:01.991902 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:02.174552 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:02.174514 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" event={"ID":"72f31e15-2da6-4b7e-9073-4457a2a283ee","Type":"ContainerStarted","Data":"05981b10601454fc896609eee6c0fcdf440b15a1e5c1a10bc341f45e918bdc8f"} Nov 02 17:41:02.174975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:02.174570 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" event={"ID":"72f31e15-2da6-4b7e-9073-4457a2a283ee","Type":"ContainerStarted","Data":"d32c5587f706d6700514606885668a0e53d29fc53328efaf355174b9f2211a7b"} Nov 02 17:41:02.206872 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:02.206811 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" podStartSLOduration=2.20678845 podStartE2EDuration="2.20678845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:02.205232181 +0000 UTC m=+433.993737709" watchObservedRunningTime="2026-11-02 17:41:02.20678845 +0000 UTC m=+433.995293978" Nov 02 17:41:02.320600 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:02.320521 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:02.469141 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:02.469110 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:02.640926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:02.640825 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:02.958252 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:02.957922 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:02.973633 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:02.973602 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:03.179939 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.179907 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:03.180869 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.180849 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:03.197289 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.197259 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52"] Nov 02 17:41:03.201092 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.201076 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52" Nov 02 17:41:03.213462 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.212675 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52"] Nov 02 17:41:03.213462 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.213028 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-868a2ee9\"" Nov 02 17:41:03.220546 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.220519 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:03.232616 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.232592 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv"] Nov 02 17:41:03.236161 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.236141 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv" Nov 02 17:41:03.242021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.241998 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-868a2ee9-dockercfg-q6bkj\"" Nov 02 17:41:03.244123 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.244100 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv"] Nov 02 17:41:03.366469 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.366430 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzhz6\" (UniqueName: \"kubernetes.io/projected/8bf05dc6-3567-4d4e-b343-30e04158b39d-kube-api-access-mzhz6\") pod \"payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv\" (UID: \"8bf05dc6-3567-4d4e-b343-30e04158b39d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv" Nov 02 17:41:03.366647 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.366516 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf05dc6-3567-4d4e-b343-30e04158b39d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv\" (UID: \"8bf05dc6-3567-4d4e-b343-30e04158b39d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv" Nov 02 17:41:03.366647 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.366594 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2399aa13-f59f-431f-b775-bd0321ad7728-plugins-config-volume\") pod \"payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52\" (UID: \"2399aa13-f59f-431f-b775-bd0321ad7728\") " pod="openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52" Nov 02 17:41:03.366759 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.366701 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx27n\" (UniqueName: \"kubernetes.io/projected/2399aa13-f59f-431f-b775-bd0321ad7728-kube-api-access-bx27n\") pod \"payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52\" (UID: \"2399aa13-f59f-431f-b775-bd0321ad7728\") " pod="openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52" Nov 02 17:41:03.366985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.366939 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:03.467838 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.467742 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzhz6\" (UniqueName: \"kubernetes.io/projected/8bf05dc6-3567-4d4e-b343-30e04158b39d-kube-api-access-mzhz6\") pod \"payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv\" (UID: \"8bf05dc6-3567-4d4e-b343-30e04158b39d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv" Nov 02 17:41:03.467838 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.467783 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf05dc6-3567-4d4e-b343-30e04158b39d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv\" (UID: \"8bf05dc6-3567-4d4e-b343-30e04158b39d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv" Nov 02 17:41:03.468084 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.467841 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2399aa13-f59f-431f-b775-bd0321ad7728-plugins-config-volume\") pod \"payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52\" (UID: \"2399aa13-f59f-431f-b775-bd0321ad7728\") " pod="openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52" Nov 02 17:41:03.468084 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.467925 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx27n\" (UniqueName: \"kubernetes.io/projected/2399aa13-f59f-431f-b775-bd0321ad7728-kube-api-access-bx27n\") pod \"payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52\" (UID: \"2399aa13-f59f-431f-b775-bd0321ad7728\") " pod="openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52" Nov 02 17:41:03.468693 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.468664 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2399aa13-f59f-431f-b775-bd0321ad7728-plugins-config-volume\") pod \"payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52\" (UID: \"2399aa13-f59f-431f-b775-bd0321ad7728\") " pod="openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52" Nov 02 17:41:03.468693 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.468685 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf05dc6-3567-4d4e-b343-30e04158b39d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv\" (UID: \"8bf05dc6-3567-4d4e-b343-30e04158b39d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv" Nov 02 17:41:03.477034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.477015 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzhz6\" (UniqueName: \"kubernetes.io/projected/8bf05dc6-3567-4d4e-b343-30e04158b39d-kube-api-access-mzhz6\") pod \"payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv\" (UID: \"8bf05dc6-3567-4d4e-b343-30e04158b39d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv" Nov 02 17:41:03.477113 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.477018 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx27n\" (UniqueName: \"kubernetes.io/projected/2399aa13-f59f-431f-b775-bd0321ad7728-kube-api-access-bx27n\") pod \"payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52\" (UID: \"2399aa13-f59f-431f-b775-bd0321ad7728\") " pod="openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52" Nov 02 17:41:03.512891 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.512860 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:03.523048 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.523020 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52" Nov 02 17:41:03.555427 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.555396 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv" Nov 02 17:41:03.809862 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.809775 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52"] Nov 02 17:41:03.813181 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:41:03.813150 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2399aa13_f59f_431f_b775_bd0321ad7728.slice/crio-e1e720f9b3fbfa3bcc1620772792eb551df36ff329ad62f6765daee58f524866 WatchSource:0}: Error finding container e1e720f9b3fbfa3bcc1620772792eb551df36ff329ad62f6765daee58f524866: Status 404 returned error can't find the container with id e1e720f9b3fbfa3bcc1620772792eb551df36ff329ad62f6765daee58f524866 Nov 02 17:41:03.844521 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:41:03.844492 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf05dc6_3567_4d4e_b343_30e04158b39d.slice/crio-3b636334e521d52a1ff58f3554fcf7e8d9622693fcafbb171027ecd96964a00d WatchSource:0}: Error finding container 3b636334e521d52a1ff58f3554fcf7e8d9622693fcafbb171027ecd96964a00d: Status 404 returned error can't find the container with id 3b636334e521d52a1ff58f3554fcf7e8d9622693fcafbb171027ecd96964a00d Nov 02 17:41:03.845362 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.845327 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:03.846384 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.846361 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv"] Nov 02 17:41:03.857572 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.857546 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:41:03.997378 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:03.997341 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:04.134887 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:04.134802 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:04.187330 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:04.187294 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv" event={"ID":"8bf05dc6-3567-4d4e-b343-30e04158b39d","Type":"ContainerStarted","Data":"89a7b0c508f0933be02a725d2078b1d6c98171ed447239949d4a78b54c87205c"} Nov 02 17:41:04.187811 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:04.187359 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv" Nov 02 17:41:04.187811 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:04.187388 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv" event={"ID":"8bf05dc6-3567-4d4e-b343-30e04158b39d","Type":"ContainerStarted","Data":"3b636334e521d52a1ff58f3554fcf7e8d9622693fcafbb171027ecd96964a00d"} Nov 02 17:41:04.188851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:04.188824 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52" event={"ID":"2399aa13-f59f-431f-b775-bd0321ad7728","Type":"ContainerStarted","Data":"6760ed34a51876b62ad77ac9eba7ab67f7be640338d691dc8df7930ef069358e"} Nov 02 17:41:04.188981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:04.188870 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52" event={"ID":"2399aa13-f59f-431f-b775-bd0321ad7728","Type":"ContainerStarted","Data":"e1e720f9b3fbfa3bcc1620772792eb551df36ff329ad62f6765daee58f524866"} Nov 02 17:41:04.189123 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:04.189101 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52" Nov 02 17:41:04.204918 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:04.204859 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv" podStartSLOduration=1.2048432789999999 podStartE2EDuration="1.204843279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:04.204598006 +0000 UTC m=+435.993103557" watchObservedRunningTime="2026-11-02 17:41:04.204843279 +0000 UTC m=+435.993348813" Nov 02 17:41:04.223391 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:04.223345 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52" podStartSLOduration=1.223330702 podStartE2EDuration="1.223330702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:04.222944453 +0000 UTC m=+436.011449998" watchObservedRunningTime="2026-11-02 17:41:04.223330702 +0000 UTC m=+436.011836231" Nov 02 17:41:04.616772 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:04.616736 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:05.023554 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:05.023511 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:05.167965 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:05.167906 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:05.609211 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:05.609182 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:05.753281 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:05.753238 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:06.057471 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:06.057436 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:06.190321 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:06.190268 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:06.381297 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:06.381203 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:06.516844 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:06.516807 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:06.669946 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:06.669879 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:06.824977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:06.822850 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:07.075820 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.075778 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:07.222219 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.222186 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j"] Nov 02 17:41:07.226758 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.226733 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:07.237919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.237891 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j"] Nov 02 17:41:07.238808 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.238787 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ad831efe-serving-cert\"" Nov 02 17:41:07.319588 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.319554 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-metrics-certs\") pod \"maas-api-e2e-ait-ad831efe-68977fb764-sp28j\" (UID: \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:07.319739 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.319619 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-openshift-service-ca\") pod \"maas-api-e2e-ait-ad831efe-68977fb764-sp28j\" (UID: \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:07.319851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.319830 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-maas-api-tls\") pod \"maas-api-e2e-ait-ad831efe-68977fb764-sp28j\" (UID: \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:07.319888 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.319859 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lstg6\" (UniqueName: \"kubernetes.io/projected/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-kube-api-access-lstg6\") pod \"maas-api-e2e-ait-ad831efe-68977fb764-sp28j\" (UID: \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:07.320355 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.320333 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:07.421198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.421107 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-maas-api-tls\") pod \"maas-api-e2e-ait-ad831efe-68977fb764-sp28j\" (UID: \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:07.421198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.421146 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lstg6\" (UniqueName: \"kubernetes.io/projected/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-kube-api-access-lstg6\") pod \"maas-api-e2e-ait-ad831efe-68977fb764-sp28j\" (UID: \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:07.421198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.421193 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-metrics-certs\") pod \"maas-api-e2e-ait-ad831efe-68977fb764-sp28j\" (UID: \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:07.421485 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.421241 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-openshift-service-ca\") pod \"maas-api-e2e-ait-ad831efe-68977fb764-sp28j\" (UID: \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:07.421485 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:41:07.421286 5549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-serving-cert: secret "maas-api-e2e-ait-ad831efe-serving-cert" not found Nov 02 17:41:07.421485 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:41:07.421356 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-maas-api-tls podName:6a98cf7f-b08b-4ce0-9729-35f4884a4a8d nodeName:}" failed. No retries permitted until 2026-11-02 17:41:07.921339515 +0000 UTC m=+439.709845041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-maas-api-tls") pod "maas-api-e2e-ait-ad831efe-68977fb764-sp28j" (UID: "6a98cf7f-b08b-4ce0-9729-35f4884a4a8d") : secret "maas-api-e2e-ait-ad831efe-serving-cert" not found Nov 02 17:41:07.422001 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.421945 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-openshift-service-ca\") pod \"maas-api-e2e-ait-ad831efe-68977fb764-sp28j\" (UID: \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:07.423614 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.423595 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-metrics-certs\") pod \"maas-api-e2e-ait-ad831efe-68977fb764-sp28j\" (UID: \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:07.434194 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.434173 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lstg6\" (UniqueName: \"kubernetes.io/projected/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-kube-api-access-lstg6\") pod \"maas-api-e2e-ait-ad831efe-68977fb764-sp28j\" (UID: \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:07.460326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.460297 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:07.598171 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.598136 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:07.927033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.926995 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-maas-api-tls\") pod \"maas-api-e2e-ait-ad831efe-68977fb764-sp28j\" (UID: \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:07.929431 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:07.929407 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-maas-api-tls\") pod \"maas-api-e2e-ait-ad831efe-68977fb764-sp28j\" (UID: \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:08.144474 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:08.144438 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:08.249808 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:08.249766 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:08.432179 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:08.432152 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j"] Nov 02 17:41:08.435031 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:41:08.434995 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a98cf7f_b08b_4ce0_9729_35f4884a4a8d.slice/crio-73d93589dc0de37e4dd7b85a5f3428af450a26d0bc12c41c563b48c9111579da WatchSource:0}: Error finding container 73d93589dc0de37e4dd7b85a5f3428af450a26d0bc12c41c563b48c9111579da: Status 404 returned error can't find the container with id 73d93589dc0de37e4dd7b85a5f3428af450a26d0bc12c41c563b48c9111579da Nov 02 17:41:08.436566 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:08.436545 5549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:41:09.215977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:09.215909 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" event={"ID":"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d","Type":"ContainerStarted","Data":"ed623290155504a88930906affcfa0fbca5cf3001a19df95e27823d59c09ae3f"} Nov 02 17:41:09.216373 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:09.216021 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" event={"ID":"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d","Type":"ContainerStarted","Data":"73d93589dc0de37e4dd7b85a5f3428af450a26d0bc12c41c563b48c9111579da"} Nov 02 17:41:09.216373 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:09.216062 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:09.234766 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:09.234719 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" podStartSLOduration=1.8458820839999999 podStartE2EDuration="2.234706621s" totalImagesPullingTime="388.824537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:09.23313972 +0000 UTC m=+441.021645251" watchObservedRunningTime="2026-11-02 17:41:09.234706621 +0000 UTC m=+441.023212148" Nov 02 17:41:09.272346 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:09.272314 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:09.800977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:09.797605 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:09.940282 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:09.940248 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:10.111029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:10.110400 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:10.999798 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:10.999766 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:11.168707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:11.168671 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:11.187978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:11.179885 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq"] Nov 02 17:41:11.212461 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:11.210507 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq"] Nov 02 17:41:11.212461 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:11.210642 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq" Nov 02 17:41:11.216909 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:11.215146 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-55f9886f\"" Nov 02 17:41:11.240316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:11.236635 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-55f9886f-dockercfg-s5cpp\"" Nov 02 17:41:11.364793 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:11.364693 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11e31cc-f03c-4d2c-a0e0-5859997fc096-plugins-config-volume\") pod \"payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq\" (UID: \"c11e31cc-f03c-4d2c-a0e0-5859997fc096\") " pod="openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq" Nov 02 17:41:11.364975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:11.364812 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbl6w\" (UniqueName: \"kubernetes.io/projected/c11e31cc-f03c-4d2c-a0e0-5859997fc096-kube-api-access-fbl6w\") pod \"payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq\" (UID: \"c11e31cc-f03c-4d2c-a0e0-5859997fc096\") " pod="openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq" Nov 02 17:41:11.465941 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:11.465905 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11e31cc-f03c-4d2c-a0e0-5859997fc096-plugins-config-volume\") pod \"payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq\" (UID: \"c11e31cc-f03c-4d2c-a0e0-5859997fc096\") " pod="openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq" Nov 02 17:41:11.466173 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:11.465977 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbl6w\" (UniqueName: \"kubernetes.io/projected/c11e31cc-f03c-4d2c-a0e0-5859997fc096-kube-api-access-fbl6w\") pod \"payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq\" (UID: \"c11e31cc-f03c-4d2c-a0e0-5859997fc096\") " pod="openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq" Nov 02 17:41:11.466805 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:11.466781 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11e31cc-f03c-4d2c-a0e0-5859997fc096-plugins-config-volume\") pod \"payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq\" (UID: \"c11e31cc-f03c-4d2c-a0e0-5859997fc096\") " pod="openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq" Nov 02 17:41:11.475085 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:11.475057 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbl6w\" (UniqueName: \"kubernetes.io/projected/c11e31cc-f03c-4d2c-a0e0-5859997fc096-kube-api-access-fbl6w\") pod \"payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq\" (UID: \"c11e31cc-f03c-4d2c-a0e0-5859997fc096\") " pod="openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq" Nov 02 17:41:11.547237 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:11.547198 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq" Nov 02 17:41:11.833926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:11.833895 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:11.848989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:11.848887 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq"] Nov 02 17:41:11.851567 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:41:11.851537 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11e31cc_f03c_4d2c_a0e0_5859997fc096.slice/crio-05710188881e1ac80eccff48d5f3aced3da0c1e5bddcbe297ad559bcce1803a6 WatchSource:0}: Error finding container 05710188881e1ac80eccff48d5f3aced3da0c1e5bddcbe297ad559bcce1803a6: Status 404 returned error can't find the container with id 05710188881e1ac80eccff48d5f3aced3da0c1e5bddcbe297ad559bcce1803a6 Nov 02 17:41:11.949838 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:11.949798 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:41:12.242667 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:12.242628 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq" event={"ID":"c11e31cc-f03c-4d2c-a0e0-5859997fc096","Type":"ContainerStarted","Data":"4f0ff2d1015f9d0a0dc30c92292785d1588b6f89a03666e83f4543955329e3b4"} Nov 02 17:41:12.243146 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:12.242684 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq" event={"ID":"c11e31cc-f03c-4d2c-a0e0-5859997fc096","Type":"ContainerStarted","Data":"05710188881e1ac80eccff48d5f3aced3da0c1e5bddcbe297ad559bcce1803a6"} Nov 02 17:41:12.243146 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:12.242711 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq" Nov 02 17:41:12.260609 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:12.260538 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq" podStartSLOduration=1.260524937 podStartE2EDuration="1.260524937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:12.258913215 +0000 UTC m=+444.047418741" watchObservedRunningTime="2026-11-02 17:41:12.260524937 +0000 UTC m=+444.049030478" Nov 02 17:41:12.650001 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:12.649891 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:13.459661 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:13.459623 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:13.623283 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:13.623248 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:14.228734 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:14.228696 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:14.938987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:14.938293 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:15.220979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:15.213132 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv" Nov 02 17:41:15.220979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:15.217135 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52" Nov 02 17:41:15.236978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:15.236911 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:16.003976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:16.003497 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:16.906975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:16.904820 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:17.633762 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:41:17.633713 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:41:17.845479 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:17.845445 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:18.632974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:18.632055 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:19.227180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:19.227138 5549 patch_prober.go:29] interesting pod/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:41:19.227667 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:19.227211 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" podUID="72f31e15-2da6-4b7e-9073-4457a2a283ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:41:19.328695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:19.328649 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:20.098179 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:20.098142 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:20.387063 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:20.386944 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:41:20.440719 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:20.440678 5549 generic.go:356] "Generic (PLEG): container finished" podID="72f31e15-2da6-4b7e-9073-4457a2a283ee" containerID="05981b10601454fc896609eee6c0fcdf440b15a1e5c1a10bc341f45e918bdc8f" exitCode=0 Nov 02 17:41:20.440888 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:20.440730 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" event={"ID":"72f31e15-2da6-4b7e-9073-4457a2a283ee","Type":"ContainerDied","Data":"05981b10601454fc896609eee6c0fcdf440b15a1e5c1a10bc341f45e918bdc8f"} Nov 02 17:41:20.441301 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:20.441280 5549 scope.go:122] "RemoveContainer" containerID="05981b10601454fc896609eee6c0fcdf440b15a1e5c1a10bc341f45e918bdc8f" Nov 02 17:41:20.442839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:20.442807 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:20.442978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:20.442897 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:20.443054 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:20.442987 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:20.907593 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:20.907487 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:21.448292 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:21.448249 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" event={"ID":"72f31e15-2da6-4b7e-9073-4457a2a283ee","Type":"ContainerStarted","Data":"653dd1d5d1ffaf832f6aaa2bcdd4f82c54de7692d75b9243de80cd5439533e1d"} Nov 02 17:41:21.628113 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:21.628015 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:21.958078 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:21.958006 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:21.960180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:21.958300 5549 patch_prober.go:29] interesting pod/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:41:21.960180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:21.958362 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" podUID="72f31e15-2da6-4b7e-9073-4457a2a283ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 17:41:22.434263 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.434219 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:22.455292 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.455251 5549 generic.go:356] "Generic (PLEG): container finished" podID="a2059966-df3d-42a3-b05f-a00593fb40e6" containerID="6c7ed4dfb7d51c52cdc02f39e97e3b8fd1e3f2996c6ab3242842473787143b6b" exitCode=0 Nov 02 17:41:22.455744 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.455329 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" event={"ID":"a2059966-df3d-42a3-b05f-a00593fb40e6","Type":"ContainerDied","Data":"6c7ed4dfb7d51c52cdc02f39e97e3b8fd1e3f2996c6ab3242842473787143b6b"} Nov 02 17:41:22.455833 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.455812 5549 scope.go:122] "RemoveContainer" containerID="6c7ed4dfb7d51c52cdc02f39e97e3b8fd1e3f2996c6ab3242842473787143b6b" Nov 02 17:41:22.457313 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.457269 5549 generic.go:356] "Generic (PLEG): container finished" podID="43b648d1-eab0-4625-bf8a-d708d790ab14" containerID="1397e74de2801f5ba154ac53ba8d5d0d4c40ad7b1688f24b9f71c09e38dce23a" exitCode=0 Nov 02 17:41:22.457428 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.457328 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" event={"ID":"43b648d1-eab0-4625-bf8a-d708d790ab14","Type":"ContainerDied","Data":"1397e74de2801f5ba154ac53ba8d5d0d4c40ad7b1688f24b9f71c09e38dce23a"} Nov 02 17:41:22.457555 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.457463 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:22.457616 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.457556 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:22.457616 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.457607 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:22.457836 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.457818 5549 scope.go:122] "RemoveContainer" containerID="1397e74de2801f5ba154ac53ba8d5d0d4c40ad7b1688f24b9f71c09e38dce23a" Nov 02 17:41:22.459065 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.459036 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:22.459163 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.459139 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:22.459226 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.459192 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:22.959773 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.958689 5549 patch_prober.go:29] interesting pod/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:41:22.959773 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.958740 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" podUID="72f31e15-2da6-4b7e-9073-4457a2a283ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 17:41:22.966378 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.966345 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp"] Nov 02 17:41:22.969318 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.969298 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp" Nov 02 17:41:22.973578 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.973552 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\"" Nov 02 17:41:22.973687 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.973580 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:41:22.984983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:22.979607 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp"] Nov 02 17:41:23.036858 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:23.032123 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l42s5\" (UniqueName: \"kubernetes.io/projected/0e9f08f2-bea1-4e74-b719-e105d306da88-kube-api-access-l42s5\") pod \"maas-api-revoke-keys-e2e-ait-ad831efe-49svp\" (UID: \"0e9f08f2-bea1-4e74-b719-e105d306da88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp" Nov 02 17:41:23.036858 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:23.032183 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9f08f2-bea1-4e74-b719-e105d306da88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad831efe-49svp\" (UID: \"0e9f08f2-bea1-4e74-b719-e105d306da88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp" Nov 02 17:41:23.036858 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:23.032302 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9f08f2-bea1-4e74-b719-e105d306da88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad831efe-49svp\" (UID: \"0e9f08f2-bea1-4e74-b719-e105d306da88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp" Nov 02 17:41:23.133782 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:23.133741 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l42s5\" (UniqueName: \"kubernetes.io/projected/0e9f08f2-bea1-4e74-b719-e105d306da88-kube-api-access-l42s5\") pod \"maas-api-revoke-keys-e2e-ait-ad831efe-49svp\" (UID: \"0e9f08f2-bea1-4e74-b719-e105d306da88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp" Nov 02 17:41:23.133971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:23.133796 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9f08f2-bea1-4e74-b719-e105d306da88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad831efe-49svp\" (UID: \"0e9f08f2-bea1-4e74-b719-e105d306da88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp" Nov 02 17:41:23.133971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:23.133840 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9f08f2-bea1-4e74-b719-e105d306da88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad831efe-49svp\" (UID: \"0e9f08f2-bea1-4e74-b719-e105d306da88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp" Nov 02 17:41:23.134513 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:23.134481 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9f08f2-bea1-4e74-b719-e105d306da88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad831efe-49svp\" (UID: \"0e9f08f2-bea1-4e74-b719-e105d306da88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp" Nov 02 17:41:23.136242 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:23.136220 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9f08f2-bea1-4e74-b719-e105d306da88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad831efe-49svp\" (UID: \"0e9f08f2-bea1-4e74-b719-e105d306da88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp" Nov 02 17:41:23.142292 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:23.142271 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l42s5\" (UniqueName: \"kubernetes.io/projected/0e9f08f2-bea1-4e74-b719-e105d306da88-kube-api-access-l42s5\") pod \"maas-api-revoke-keys-e2e-ait-ad831efe-49svp\" (UID: \"0e9f08f2-bea1-4e74-b719-e105d306da88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp" Nov 02 17:41:23.255018 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:23.254906 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq" Nov 02 17:41:23.311908 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:23.311859 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp" Nov 02 17:41:23.471783 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:23.471746 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" event={"ID":"a2059966-df3d-42a3-b05f-a00593fb40e6","Type":"ContainerStarted","Data":"a8d001c3cc98c16440e5eb38aa502cc83c30ec836e4ec75939b8e6c6eadb159d"} Nov 02 17:41:23.474126 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:23.474095 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" event={"ID":"43b648d1-eab0-4625-bf8a-d708d790ab14","Type":"ContainerStarted","Data":"1a5eabbb19782a7efbcb811e76d70cfefbfc97b047a714d67b61b1ea4b285433"} Nov 02 17:41:23.609377 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:23.563516 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:23.763710 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:41:23.763155 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9f08f2_bea1_4e74_b719_e105d306da88.slice/crio-f4b09e333bb1b792bfe6b018a06dedd5d7a775602dc2c827c18115b0772203f8 WatchSource:0}: Error finding container f4b09e333bb1b792bfe6b018a06dedd5d7a775602dc2c827c18115b0772203f8: Status 404 returned error can't find the container with id f4b09e333bb1b792bfe6b018a06dedd5d7a775602dc2c827c18115b0772203f8 Nov 02 17:41:23.765547 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:23.765525 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp"] Nov 02 17:41:23.962903 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:23.962388 5549 patch_prober.go:29] interesting pod/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:41:23.962903 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:23.962464 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" podUID="72f31e15-2da6-4b7e-9073-4457a2a283ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 17:41:24.104976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:24.104711 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:41:24.104976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:24.104894 5549 patch_prober.go:29] interesting pod/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:41:24.104976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:24.104941 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" podUID="a2059966-df3d-42a3-b05f-a00593fb40e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:41:24.266066 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:24.265041 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:41:24.266066 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:24.265050 5549 patch_prober.go:29] interesting pod/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 17:41:24.266066 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:24.265162 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" podUID="43b648d1-eab0-4625-bf8a-d708d790ab14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 17:41:24.457980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:24.455271 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:24.549651 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:24.549565 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp" event={"ID":"0e9f08f2-bea1-4e74-b719-e105d306da88","Type":"ContainerStarted","Data":"f4b09e333bb1b792bfe6b018a06dedd5d7a775602dc2c827c18115b0772203f8"} Nov 02 17:41:24.959444 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:24.958980 5549 patch_prober.go:29] interesting pod/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:41:24.959444 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:24.959055 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" podUID="72f31e15-2da6-4b7e-9073-4457a2a283ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 17:41:25.106570 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:25.106502 5549 patch_prober.go:29] interesting pod/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:41:25.106881 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:25.106808 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" podUID="a2059966-df3d-42a3-b05f-a00593fb40e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:41:25.264456 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:25.264366 5549 patch_prober.go:29] interesting pod/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 17:41:25.264735 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:25.264693 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" podUID="43b648d1-eab0-4625-bf8a-d708d790ab14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 17:41:25.278601 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:25.278541 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:25.963025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:25.962482 5549 patch_prober.go:29] interesting pod/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:41:25.963025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:25.962555 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" podUID="72f31e15-2da6-4b7e-9073-4457a2a283ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 17:41:26.029579 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:26.029498 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:26.105206 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:26.104697 5549 patch_prober.go:29] interesting pod/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:41:26.105206 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:26.104764 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" podUID="a2059966-df3d-42a3-b05f-a00593fb40e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:41:26.263974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:26.263538 5549 patch_prober.go:29] interesting pod/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 17:41:26.263974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:26.263603 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" podUID="43b648d1-eab0-4625-bf8a-d708d790ab14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 17:41:26.826993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:26.826876 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:26.966590 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:26.966556 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:26.967013 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:26.966990 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:26.967752 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:26.967732 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:27.103479 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:27.103381 5549 patch_prober.go:29] interesting pod/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:41:27.112198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:27.112141 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" podUID="a2059966-df3d-42a3-b05f-a00593fb40e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:41:27.150522 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:27.150474 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:27.263170 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:27.263132 5549 patch_prober.go:29] interesting pod/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 17:41:27.263344 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:27.263184 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" podUID="43b648d1-eab0-4625-bf8a-d708d790ab14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 17:41:27.336974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:27.336045 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:27.545318 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:27.543925 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:28.108978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:28.106401 5549 patch_prober.go:29] interesting pod/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:41:28.108978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:28.106466 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" podUID="a2059966-df3d-42a3-b05f-a00593fb40e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:41:28.263121 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:28.262734 5549 patch_prober.go:29] interesting pod/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 17:41:28.263121 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:28.262786 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" podUID="43b648d1-eab0-4625-bf8a-d708d790ab14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 17:41:28.340086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:28.339909 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:28.853521 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:28.853481 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:29.009307 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:29.009276 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:29.103044 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:29.103002 5549 patch_prober.go:29] interesting pod/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:41:29.103228 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:29.103073 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" podUID="a2059966-df3d-42a3-b05f-a00593fb40e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:41:29.156742 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:29.156623 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:29.303063 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:29.301892 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:29.366492 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:29.366460 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:41:29.367150 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:29.367091 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:41:29.379619 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:29.379526 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:41:29.446731 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:29.446691 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:29.753375 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:29.753288 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:29.890119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:29.890076 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:30.026694 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:30.026606 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:30.107394 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:30.107366 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:41:30.107680 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:30.107662 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:41:30.108316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:30.108301 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:41:30.215379 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:30.215300 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:30.431349 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:30.431254 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:30.584651 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:30.584619 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:30.652660 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:30.652626 5549 generic.go:356] "Generic (PLEG): container finished" podID="0e9f08f2-bea1-4e74-b719-e105d306da88" containerID="ae699152a1d7ee8c7e723c14745e15b33ed8cc4befce4608e21d98334fd8e131" exitCode=0 Nov 02 17:41:30.652838 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:30.652716 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp" event={"ID":"0e9f08f2-bea1-4e74-b719-e105d306da88","Type":"ContainerDied","Data":"ae699152a1d7ee8c7e723c14745e15b33ed8cc4befce4608e21d98334fd8e131"} Nov 02 17:41:31.067529 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:31.067491 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:31.745296 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:31.745232 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:31.820273 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:31.820247 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp" Nov 02 17:41:31.913309 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:31.912894 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e9f08f2-bea1-4e74-b719-e105d306da88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9f08f2-bea1-4e74-b719-e105d306da88-maas-api-openshift-ca\") pod \"0e9f08f2-bea1-4e74-b719-e105d306da88\" (UID: \"0e9f08f2-bea1-4e74-b719-e105d306da88\") " Nov 02 17:41:31.913478 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:31.913365 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9f08f2-bea1-4e74-b719-e105d306da88-maas-api-openshift-ca" pod "0e9f08f2-bea1-4e74-b719-e105d306da88" (UID: "0e9f08f2-bea1-4e74-b719-e105d306da88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:31.913674 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:31.913648 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e9f08f2-bea1-4e74-b719-e105d306da88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9f08f2-bea1-4e74-b719-e105d306da88-maas-api-xks-ca\") pod \"0e9f08f2-bea1-4e74-b719-e105d306da88\" (UID: \"0e9f08f2-bea1-4e74-b719-e105d306da88\") " Nov 02 17:41:31.913809 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:31.913781 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9f08f2-bea1-4e74-b719-e105d306da88-kube-api-access-l42s5\" (UniqueName: \"kubernetes.io/projected/0e9f08f2-bea1-4e74-b719-e105d306da88-kube-api-access-l42s5\") pod \"0e9f08f2-bea1-4e74-b719-e105d306da88\" (UID: \"0e9f08f2-bea1-4e74-b719-e105d306da88\") " Nov 02 17:41:31.914142 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:31.914122 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9f08f2-bea1-4e74-b719-e105d306da88-maas-api-openshift-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:31.915157 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:31.915132 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:41:31.915909 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:31.915882 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e9f08f2-bea1-4e74-b719-e105d306da88-maas-api-xks-ca" pod "0e9f08f2-bea1-4e74-b719-e105d306da88" (UID: "0e9f08f2-bea1-4e74-b719-e105d306da88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:31.916030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:31.916009 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9f08f2-bea1-4e74-b719-e105d306da88-kube-api-access-l42s5" pod "0e9f08f2-bea1-4e74-b719-e105d306da88" (UID: "0e9f08f2-bea1-4e74-b719-e105d306da88"). InnerVolumeSpecName "kube-api-access-l42s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:32.015412 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:32.015373 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l42s5\" (UniqueName: \"kubernetes.io/projected/0e9f08f2-bea1-4e74-b719-e105d306da88-kube-api-access-l42s5\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:32.015412 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:32.015408 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9f08f2-bea1-4e74-b719-e105d306da88-maas-api-xks-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:32.522506 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:32.522465 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:32.607558 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:32.607471 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52"] Nov 02 17:41:32.620348 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:32.620316 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv"] Nov 02 17:41:32.667411 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:32.667384 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp" Nov 02 17:41:32.667570 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:32.667382 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp" event={"ID":"0e9f08f2-bea1-4e74-b719-e105d306da88","Type":"ContainerDied","Data":"f4b09e333bb1b792bfe6b018a06dedd5d7a775602dc2c827c18115b0772203f8"} Nov 02 17:41:32.667570 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:32.667509 5549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4b09e333bb1b792bfe6b018a06dedd5d7a775602dc2c827c18115b0772203f8" Nov 02 17:41:32.736294 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:32.736262 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:32.889177 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:32.889079 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:33.233753 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.233722 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:33.384021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.383986 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:33.458399 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.458368 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j"] Nov 02 17:41:33.458652 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.458610 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" podUID="6a98cf7f-b08b-4ce0-9729-35f4884a4a8d" containerName="maas-api" containerID="cri-o://ed623290155504a88930906affcfa0fbca5cf3001a19df95e27823d59c09ae3f" gracePeriod=30 Nov 02 17:41:33.550321 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.550218 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:33.675979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.675935 5549 generic.go:356] "Generic (PLEG): container finished" podID="6a98cf7f-b08b-4ce0-9729-35f4884a4a8d" containerID="ed623290155504a88930906affcfa0fbca5cf3001a19df95e27823d59c09ae3f" exitCode=0 Nov 02 17:41:33.676113 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.675996 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" event={"ID":"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d","Type":"ContainerDied","Data":"ed623290155504a88930906affcfa0fbca5cf3001a19df95e27823d59c09ae3f"} Nov 02 17:41:33.708114 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.708083 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:33.724099 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.724078 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:33.821770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.821677 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs"] Nov 02 17:41:33.822030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.821934 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" podUID="a2059966-df3d-42a3-b05f-a00593fb40e6" containerName="istio-proxy" containerID="cri-o://a8d001c3cc98c16440e5eb38aa502cc83c30ec836e4ec75939b8e6c6eadb159d" gracePeriod=30 Nov 02 17:41:33.843198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.842270 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-kube-api-access-lstg6\" (UniqueName: \"kubernetes.io/projected/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-kube-api-access-lstg6\") pod \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\" (UID: \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\") " Nov 02 17:41:33.843198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.842368 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-openshift-service-ca\") pod \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\" (UID: \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\") " Nov 02 17:41:33.843198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.842424 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-maas-api-tls\") pod \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\" (UID: \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\") " Nov 02 17:41:33.843198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.842460 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-metrics-certs\") pod \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\" (UID: \"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d\") " Nov 02 17:41:33.843198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.842968 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-openshift-service-ca" pod "6a98cf7f-b08b-4ce0-9729-35f4884a4a8d" (UID: "6a98cf7f-b08b-4ce0-9729-35f4884a4a8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:33.843578 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.843535 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:33.847593 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.847562 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-kube-api-access-lstg6" pod "6a98cf7f-b08b-4ce0-9729-35f4884a4a8d" (UID: "6a98cf7f-b08b-4ce0-9729-35f4884a4a8d"). InnerVolumeSpecName "kube-api-access-lstg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:33.848634 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.848609 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-maas-api-tls" pod "6a98cf7f-b08b-4ce0-9729-35f4884a4a8d" (UID: "6a98cf7f-b08b-4ce0-9729-35f4884a4a8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:33.852089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.852059 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-metrics-certs" pod "6a98cf7f-b08b-4ce0-9729-35f4884a4a8d" (UID: "6a98cf7f-b08b-4ce0-9729-35f4884a4a8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:33.864412 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.864381 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:33.944551 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.944514 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:33.944551 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.944546 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:33.945051 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.944560 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lstg6\" (UniqueName: \"kubernetes.io/projected/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d-kube-api-access-lstg6\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:33.972174 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.971162 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp"] Nov 02 17:41:33.995975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:33.993168 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad831efe-49svp"] Nov 02 17:41:34.018755 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:34.018288 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:34.112784 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:34.112704 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s"] Nov 02 17:41:34.113006 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:34.112977 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" podUID="72f31e15-2da6-4b7e-9073-4457a2a283ee" containerName="istio-proxy" containerID="cri-o://653dd1d5d1ffaf832f6aaa2bcdd4f82c54de7692d75b9243de80cd5439533e1d" gracePeriod=30 Nov 02 17:41:34.166161 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:34.166124 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:34.305141 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:34.305104 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:34.317995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:34.317937 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e9f08f2-bea1-4e74-b719-e105d306da88" path="/var/lib/kubelet/pods/0e9f08f2-bea1-4e74-b719-e105d306da88/volumes" Nov 02 17:41:34.680497 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:34.680462 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq"] Nov 02 17:41:34.686073 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:34.686044 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" Nov 02 17:41:34.686073 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:34.686062 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j" event={"ID":"6a98cf7f-b08b-4ce0-9729-35f4884a4a8d","Type":"ContainerDied","Data":"73d93589dc0de37e4dd7b85a5f3428af450a26d0bc12c41c563b48c9111579da"} Nov 02 17:41:34.686316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:34.686143 5549 scope.go:122] "RemoveContainer" containerID="ed623290155504a88930906affcfa0fbca5cf3001a19df95e27823d59c09ae3f" Nov 02 17:41:34.721842 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:34.721808 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j"] Nov 02 17:41:34.730519 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:34.730492 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad831efe-68977fb764-sp28j"] Nov 02 17:41:34.740027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:34.739998 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:34.874279 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:34.874243 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:35.031850 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.031817 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:35.221401 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.221341 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:35.365359 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.365261 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:35.536168 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.536133 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:35.561134 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.561102 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k"] Nov 02 17:41:35.561356 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.561329 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" podUID="43b648d1-eab0-4625-bf8a-d708d790ab14" containerName="istio-proxy" containerID="cri-o://1a5eabbb19782a7efbcb811e76d70cfefbfc97b047a714d67b61b1ea4b285433" gracePeriod=30 Nov 02 17:41:35.615978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.615888 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk"] Nov 02 17:41:35.616365 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.616351 5549 cpu_manager.go:414] "containerMap: removing container" podUID="6a98cf7f-b08b-4ce0-9729-35f4884a4a8d" containerName="maas-api" Nov 02 17:41:35.616413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.616367 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a98cf7f-b08b-4ce0-9729-35f4884a4a8d" containerName="maas-api" Nov 02 17:41:35.616413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.616384 5549 cpu_manager.go:414] "containerMap: removing container" podUID="0e9f08f2-bea1-4e74-b719-e105d306da88" containerName="revoke-keys" Nov 02 17:41:35.616413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.616392 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9f08f2-bea1-4e74-b719-e105d306da88" containerName="revoke-keys" Nov 02 17:41:35.616514 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.616461 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a98cf7f-b08b-4ce0-9729-35f4884a4a8d" containerName="maas-api" Nov 02 17:41:35.616514 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.616471 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9f08f2-bea1-4e74-b719-e105d306da88" containerName="revoke-keys" Nov 02 17:41:35.619149 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.619130 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" Nov 02 17:41:35.628747 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.628698 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk"] Nov 02 17:41:35.628908 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.628829 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-a37cf0-dockercfg-mpbmh\"" Nov 02 17:41:35.628908 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.628871 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-a37cf0\"" Nov 02 17:41:35.664498 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.664460 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj4j8\" (UniqueName: \"kubernetes.io/projected/5e400bab-ea24-4848-92ba-c770160096b8-kube-api-access-cj4j8\") pod \"payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk\" (UID: \"5e400bab-ea24-4848-92ba-c770160096b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" Nov 02 17:41:35.664667 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.664649 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e400bab-ea24-4848-92ba-c770160096b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk\" (UID: \"5e400bab-ea24-4848-92ba-c770160096b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" Nov 02 17:41:35.678290 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.678258 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:35.765457 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.765417 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj4j8\" (UniqueName: \"kubernetes.io/projected/5e400bab-ea24-4848-92ba-c770160096b8-kube-api-access-cj4j8\") pod \"payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk\" (UID: \"5e400bab-ea24-4848-92ba-c770160096b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" Nov 02 17:41:35.765693 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.765480 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e400bab-ea24-4848-92ba-c770160096b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk\" (UID: \"5e400bab-ea24-4848-92ba-c770160096b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" Nov 02 17:41:35.766219 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.766197 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e400bab-ea24-4848-92ba-c770160096b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk\" (UID: \"5e400bab-ea24-4848-92ba-c770160096b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" Nov 02 17:41:35.774839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.774808 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj4j8\" (UniqueName: \"kubernetes.io/projected/5e400bab-ea24-4848-92ba-c770160096b8-kube-api-access-cj4j8\") pod \"payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk\" (UID: \"5e400bab-ea24-4848-92ba-c770160096b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" Nov 02 17:41:35.932306 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:35.932228 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" Nov 02 17:41:36.100878 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.100840 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:36.215016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.214992 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk"] Nov 02 17:41:36.217391 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:41:36.217364 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e400bab_ea24_4848_92ba_c770160096b8.slice/crio-07e061412f36946c7bbf0ae48d4ab2dc1d396781687a10fae065bf99393564db WatchSource:0}: Error finding container 07e061412f36946c7bbf0ae48d4ab2dc1d396781687a10fae065bf99393564db: Status 404 returned error can't find the container with id 07e061412f36946c7bbf0ae48d4ab2dc1d396781687a10fae065bf99393564db Nov 02 17:41:36.235081 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.235051 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:36.318492 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.318459 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a98cf7f-b08b-4ce0-9729-35f4884a4a8d" path="/var/lib/kubelet/pods/6a98cf7f-b08b-4ce0-9729-35f4884a4a8d/volumes" Nov 02 17:41:36.375677 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.375635 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:36.520203 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.520079 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:36.700798 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.700762 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" event={"ID":"5e400bab-ea24-4848-92ba-c770160096b8","Type":"ContainerStarted","Data":"b5e4f7d395217d9a7b55ef9ef14ec1537a49d4efbf76b0a3849de375e6489b38"} Nov 02 17:41:36.701047 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.701015 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" event={"ID":"5e400bab-ea24-4848-92ba-c770160096b8","Type":"ContainerStarted","Data":"07e061412f36946c7bbf0ae48d4ab2dc1d396781687a10fae065bf99393564db"} Nov 02 17:41:36.701185 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.701062 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" Nov 02 17:41:36.705244 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.705212 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:36.726073 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.726022 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" podStartSLOduration=1.726003756 podStartE2EDuration="1.726003756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:36.724686386 +0000 UTC m=+468.513191913" watchObservedRunningTime="2026-11-02 17:41:36.726003756 +0000 UTC m=+468.514509285" Nov 02 17:41:36.804553 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.804472 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn"] Nov 02 17:41:36.807445 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.807419 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.812727 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.812684 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-77f103-openshift-default-dockercfg-v9fvh\"" Nov 02 17:41:36.823135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.822662 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn"] Nov 02 17:41:36.866737 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.866675 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:36.876981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.876919 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.877148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.877028 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.877148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.877084 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.877286 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.877237 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.877286 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.877272 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a22d1c3-76ca-445d-ab56-052577f4fcff-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.877386 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.877296 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.877437 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.877416 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk88c\" (UniqueName: \"kubernetes.io/projected/9a22d1c3-76ca-445d-ab56-052577f4fcff-kube-api-access-fk88c\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.877517 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.877497 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.877613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.877591 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.979122 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.979082 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.979122 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.979125 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a22d1c3-76ca-445d-ab56-052577f4fcff-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.979394 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.979156 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.979394 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.979180 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk88c\" (UniqueName: \"kubernetes.io/projected/9a22d1c3-76ca-445d-ab56-052577f4fcff-kube-api-access-fk88c\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.979394 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.979245 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.979394 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.979306 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.979394 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.979348 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.979608 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.979582 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.979937 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.979907 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.980082 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.979908 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.980139 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.980120 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a22d1c3-76ca-445d-ab56-052577f4fcff-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.982344 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.979374 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.982461 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.982425 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.982764 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.982741 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.982844 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.982780 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.985202 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.985176 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.988683 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.988647 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:36.990194 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:36.988865 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk88c\" (UniqueName: \"kubernetes.io/projected/9a22d1c3-76ca-445d-ab56-052577f4fcff-kube-api-access-fk88c\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:37.001457 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.001412 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:37.125620 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.125509 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:37.152020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.148939 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:37.415322 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.415290 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn"] Nov 02 17:41:37.416735 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:41:37.416703 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a22d1c3_76ca_445d_ab56_052577f4fcff.slice/crio-8423912090dc2eeabc4b8214c32b676087baf3f749c2a0c2ae3ddd61c9149117 WatchSource:0}: Error finding container 8423912090dc2eeabc4b8214c32b676087baf3f749c2a0c2ae3ddd61c9149117: Status 404 returned error can't find the container with id 8423912090dc2eeabc4b8214c32b676087baf3f749c2a0c2ae3ddd61c9149117 Nov 02 17:41:37.419321 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.419285 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:37.419420 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.419364 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:37.419420 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.419410 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:37.608578 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.608532 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52" podUID="2399aa13-f59f-431f-b775-bd0321ad7728" containerName="payload-processing" containerID="cri-o://6760ed34a51876b62ad77ac9eba7ab67f7be640338d691dc8df7930ef069358e" gracePeriod=25 Nov 02 17:41:37.621282 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.621245 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv" podUID="8bf05dc6-3567-4d4e-b343-30e04158b39d" containerName="payload-pre-processing" containerID="cri-o://89a7b0c508f0933be02a725d2078b1d6c98171ed447239949d4a78b54c87205c" gracePeriod=25 Nov 02 17:41:37.708591 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.708551 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" event={"ID":"9a22d1c3-76ca-445d-ab56-052577f4fcff","Type":"ContainerStarted","Data":"a52b88620fb74528f2f11b3f3b294a868725e09db95d8ec2a5e9c3a0e17de335"} Nov 02 17:41:37.708763 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.708614 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" event={"ID":"9a22d1c3-76ca-445d-ab56-052577f4fcff","Type":"ContainerStarted","Data":"8423912090dc2eeabc4b8214c32b676087baf3f749c2a0c2ae3ddd61c9149117"} Nov 02 17:41:37.731619 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.731555 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" podStartSLOduration=1.7315358029999999 podStartE2EDuration="1.731535803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:37.729753723 +0000 UTC m=+469.518259253" watchObservedRunningTime="2026-11-02 17:41:37.731535803 +0000 UTC m=+469.520041331" Nov 02 17:41:37.898783 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.898755 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52" Nov 02 17:41:37.935052 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.935022 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv" Nov 02 17:41:37.994311 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.994283 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf05dc6-3567-4d4e-b343-30e04158b39d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf05dc6-3567-4d4e-b343-30e04158b39d-plugins-config-volume\") pod \"8bf05dc6-3567-4d4e-b343-30e04158b39d\" (UID: \"8bf05dc6-3567-4d4e-b343-30e04158b39d\") " Nov 02 17:41:37.994454 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.994376 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf05dc6-3567-4d4e-b343-30e04158b39d-kube-api-access-mzhz6\" (UniqueName: \"kubernetes.io/projected/8bf05dc6-3567-4d4e-b343-30e04158b39d-kube-api-access-mzhz6\") pod \"8bf05dc6-3567-4d4e-b343-30e04158b39d\" (UID: \"8bf05dc6-3567-4d4e-b343-30e04158b39d\") " Nov 02 17:41:37.994454 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.994419 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2399aa13-f59f-431f-b775-bd0321ad7728-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2399aa13-f59f-431f-b775-bd0321ad7728-plugins-config-volume\") pod \"2399aa13-f59f-431f-b775-bd0321ad7728\" (UID: \"2399aa13-f59f-431f-b775-bd0321ad7728\") " Nov 02 17:41:37.994574 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.994495 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2399aa13-f59f-431f-b775-bd0321ad7728-kube-api-access-bx27n\" (UniqueName: \"kubernetes.io/projected/2399aa13-f59f-431f-b775-bd0321ad7728-kube-api-access-bx27n\") pod \"2399aa13-f59f-431f-b775-bd0321ad7728\" (UID: \"2399aa13-f59f-431f-b775-bd0321ad7728\") " Nov 02 17:41:37.994706 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.994673 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf05dc6-3567-4d4e-b343-30e04158b39d-plugins-config-volume" pod "8bf05dc6-3567-4d4e-b343-30e04158b39d" (UID: "8bf05dc6-3567-4d4e-b343-30e04158b39d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:37.995017 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.994926 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2399aa13-f59f-431f-b775-bd0321ad7728-plugins-config-volume" pod "2399aa13-f59f-431f-b775-bd0321ad7728" (UID: "2399aa13-f59f-431f-b775-bd0321ad7728"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:37.995017 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.994976 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf05dc6-3567-4d4e-b343-30e04158b39d-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:37.996708 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.996684 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2399aa13-f59f-431f-b775-bd0321ad7728-kube-api-access-bx27n" pod "2399aa13-f59f-431f-b775-bd0321ad7728" (UID: "2399aa13-f59f-431f-b775-bd0321ad7728"). InnerVolumeSpecName "kube-api-access-bx27n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:37.996708 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:37.996700 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf05dc6-3567-4d4e-b343-30e04158b39d-kube-api-access-mzhz6" pod "8bf05dc6-3567-4d4e-b343-30e04158b39d" (UID: "8bf05dc6-3567-4d4e-b343-30e04158b39d"). InnerVolumeSpecName "kube-api-access-mzhz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:38.095867 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.095826 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzhz6\" (UniqueName: \"kubernetes.io/projected/8bf05dc6-3567-4d4e-b343-30e04158b39d-kube-api-access-mzhz6\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:38.095867 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.095866 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2399aa13-f59f-431f-b775-bd0321ad7728-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:38.096120 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.095881 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx27n\" (UniqueName: \"kubernetes.io/projected/2399aa13-f59f-431f-b775-bd0321ad7728-kube-api-access-bx27n\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:38.126159 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.126119 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:38.131544 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.131516 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:38.166799 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.166763 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:38.344514 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.344481 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:38.473372 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.473340 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:38.694049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.693968 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:38.715448 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.715418 5549 generic.go:356] "Generic (PLEG): container finished" podID="8bf05dc6-3567-4d4e-b343-30e04158b39d" containerID="89a7b0c508f0933be02a725d2078b1d6c98171ed447239949d4a78b54c87205c" exitCode=0 Nov 02 17:41:38.715650 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.715464 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv" Nov 02 17:41:38.715650 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.715500 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv" event={"ID":"8bf05dc6-3567-4d4e-b343-30e04158b39d","Type":"ContainerDied","Data":"89a7b0c508f0933be02a725d2078b1d6c98171ed447239949d4a78b54c87205c"} Nov 02 17:41:38.715650 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.715560 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv" event={"ID":"8bf05dc6-3567-4d4e-b343-30e04158b39d","Type":"ContainerDied","Data":"3b636334e521d52a1ff58f3554fcf7e8d9622693fcafbb171027ecd96964a00d"} Nov 02 17:41:38.715650 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.715589 5549 scope.go:122] "RemoveContainer" containerID="89a7b0c508f0933be02a725d2078b1d6c98171ed447239949d4a78b54c87205c" Nov 02 17:41:38.716988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.716967 5549 generic.go:356] "Generic (PLEG): container finished" podID="2399aa13-f59f-431f-b775-bd0321ad7728" containerID="6760ed34a51876b62ad77ac9eba7ab67f7be640338d691dc8df7930ef069358e" exitCode=0 Nov 02 17:41:38.717083 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.717023 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52" Nov 02 17:41:38.717083 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.717042 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52" event={"ID":"2399aa13-f59f-431f-b775-bd0321ad7728","Type":"ContainerDied","Data":"6760ed34a51876b62ad77ac9eba7ab67f7be640338d691dc8df7930ef069358e"} Nov 02 17:41:38.717165 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.717095 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52" event={"ID":"2399aa13-f59f-431f-b775-bd0321ad7728","Type":"ContainerDied","Data":"e1e720f9b3fbfa3bcc1620772792eb551df36ff329ad62f6765daee58f524866"} Nov 02 17:41:38.717860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.717388 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:38.718572 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.718550 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:38.724812 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.724796 5549 scope.go:122] "RemoveContainer" containerID="89a7b0c508f0933be02a725d2078b1d6c98171ed447239949d4a78b54c87205c" Nov 02 17:41:38.725134 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:41:38.725114 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a7b0c508f0933be02a725d2078b1d6c98171ed447239949d4a78b54c87205c\": container with ID starting with 89a7b0c508f0933be02a725d2078b1d6c98171ed447239949d4a78b54c87205c not found: ID does not exist" containerID="89a7b0c508f0933be02a725d2078b1d6c98171ed447239949d4a78b54c87205c" Nov 02 17:41:38.725179 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.725144 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a7b0c508f0933be02a725d2078b1d6c98171ed447239949d4a78b54c87205c"} err="failed to get container status \"89a7b0c508f0933be02a725d2078b1d6c98171ed447239949d4a78b54c87205c\": rpc error: code = NotFound desc = could not find container \"89a7b0c508f0933be02a725d2078b1d6c98171ed447239949d4a78b54c87205c\": container with ID starting with 89a7b0c508f0933be02a725d2078b1d6c98171ed447239949d4a78b54c87205c not found: ID does not exist" Nov 02 17:41:38.725179 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.725163 5549 scope.go:122] "RemoveContainer" containerID="6760ed34a51876b62ad77ac9eba7ab67f7be640338d691dc8df7930ef069358e" Nov 02 17:41:38.733479 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.733456 5549 scope.go:122] "RemoveContainer" containerID="6760ed34a51876b62ad77ac9eba7ab67f7be640338d691dc8df7930ef069358e" Nov 02 17:41:38.733813 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:41:38.733773 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6760ed34a51876b62ad77ac9eba7ab67f7be640338d691dc8df7930ef069358e\": container with ID starting with 6760ed34a51876b62ad77ac9eba7ab67f7be640338d691dc8df7930ef069358e not found: ID does not exist" containerID="6760ed34a51876b62ad77ac9eba7ab67f7be640338d691dc8df7930ef069358e" Nov 02 17:41:38.733850 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.733821 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6760ed34a51876b62ad77ac9eba7ab67f7be640338d691dc8df7930ef069358e"} err="failed to get container status \"6760ed34a51876b62ad77ac9eba7ab67f7be640338d691dc8df7930ef069358e\": rpc error: code = NotFound desc = could not find container \"6760ed34a51876b62ad77ac9eba7ab67f7be640338d691dc8df7930ef069358e\": container with ID starting with 6760ed34a51876b62ad77ac9eba7ab67f7be640338d691dc8df7930ef069358e not found: ID does not exist" Nov 02 17:41:38.764388 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.764351 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52"] Nov 02 17:41:38.770071 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.770027 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-868a2ee9-74bd97d5b7-s2x52"] Nov 02 17:41:38.782379 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.782344 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv"] Nov 02 17:41:38.786167 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.786139 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-868a2ee9-54664896ff-sndzv"] Nov 02 17:41:38.830269 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:38.830232 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:39.079035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.079008 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:41:39.109767 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.107363 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:39.207137 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.207106 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2059966-df3d-42a3-b05f-a00593fb40e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2059966-df3d-42a3-b05f-a00593fb40e6-istiod-ca-cert\") pod \"a2059966-df3d-42a3-b05f-a00593fb40e6\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " Nov 02 17:41:39.207595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.207153 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-data\") pod \"a2059966-df3d-42a3-b05f-a00593fb40e6\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " Nov 02 17:41:39.207595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.207201 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-token\" (UniqueName: \"kubernetes.io/projected/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-token\") pod \"a2059966-df3d-42a3-b05f-a00593fb40e6\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " Nov 02 17:41:39.207595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.207259 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-podinfo\") pod \"a2059966-df3d-42a3-b05f-a00593fb40e6\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " Nov 02 17:41:39.207595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.207314 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-envoy\") pod \"a2059966-df3d-42a3-b05f-a00593fb40e6\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " Nov 02 17:41:39.207595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.207381 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-workload-socket\") pod \"a2059966-df3d-42a3-b05f-a00593fb40e6\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " Nov 02 17:41:39.207595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.207415 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2059966-df3d-42a3-b05f-a00593fb40e6-kube-api-access-nlkkl\" (UniqueName: \"kubernetes.io/projected/a2059966-df3d-42a3-b05f-a00593fb40e6-kube-api-access-nlkkl\") pod \"a2059966-df3d-42a3-b05f-a00593fb40e6\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " Nov 02 17:41:39.207595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.207450 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-workload-certs\") pod \"a2059966-df3d-42a3-b05f-a00593fb40e6\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " Nov 02 17:41:39.207595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.207505 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-data" pod "a2059966-df3d-42a3-b05f-a00593fb40e6" (UID: "a2059966-df3d-42a3-b05f-a00593fb40e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:39.207595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.207545 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2059966-df3d-42a3-b05f-a00593fb40e6-istiod-ca-cert" pod "a2059966-df3d-42a3-b05f-a00593fb40e6" (UID: "a2059966-df3d-42a3-b05f-a00593fb40e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:39.208067 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.207734 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-workload-socket" pod "a2059966-df3d-42a3-b05f-a00593fb40e6" (UID: "a2059966-df3d-42a3-b05f-a00593fb40e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:39.208067 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.207880 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-credential-socket\") pod \"a2059966-df3d-42a3-b05f-a00593fb40e6\" (UID: \"a2059966-df3d-42a3-b05f-a00593fb40e6\") " Nov 02 17:41:39.208067 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.207894 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-workload-certs" pod "a2059966-df3d-42a3-b05f-a00593fb40e6" (UID: "a2059966-df3d-42a3-b05f-a00593fb40e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:39.208216 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.208179 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-credential-socket" pod "a2059966-df3d-42a3-b05f-a00593fb40e6" (UID: "a2059966-df3d-42a3-b05f-a00593fb40e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:39.208406 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.208384 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.208485 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.208412 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.208485 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.208427 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.208485 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.208443 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2059966-df3d-42a3-b05f-a00593fb40e6-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.208485 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.208457 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.209632 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.209608 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-token" pod "a2059966-df3d-42a3-b05f-a00593fb40e6" (UID: "a2059966-df3d-42a3-b05f-a00593fb40e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:39.209770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.209745 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2059966-df3d-42a3-b05f-a00593fb40e6-kube-api-access-nlkkl" pod "a2059966-df3d-42a3-b05f-a00593fb40e6" (UID: "a2059966-df3d-42a3-b05f-a00593fb40e6"). InnerVolumeSpecName "kube-api-access-nlkkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:39.209899 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.209875 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-envoy" pod "a2059966-df3d-42a3-b05f-a00593fb40e6" (UID: "a2059966-df3d-42a3-b05f-a00593fb40e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:39.209899 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.209883 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-podinfo" pod "a2059966-df3d-42a3-b05f-a00593fb40e6" (UID: "a2059966-df3d-42a3-b05f-a00593fb40e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:41:39.244238 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.244153 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:41:39.322678 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.322254 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.322678 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.322295 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlkkl\" (UniqueName: \"kubernetes.io/projected/a2059966-df3d-42a3-b05f-a00593fb40e6-kube-api-access-nlkkl\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.322678 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.322311 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.322678 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.322325 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2059966-df3d-42a3-b05f-a00593fb40e6-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.367325 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.367296 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:39.423303 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.423270 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f31e15-2da6-4b7e-9073-4457a2a283ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f31e15-2da6-4b7e-9073-4457a2a283ee-istiod-ca-cert\") pod \"72f31e15-2da6-4b7e-9073-4457a2a283ee\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " Nov 02 17:41:39.423477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.423315 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-podinfo\") pod \"72f31e15-2da6-4b7e-9073-4457a2a283ee\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " Nov 02 17:41:39.423477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.423339 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-data\") pod \"72f31e15-2da6-4b7e-9073-4457a2a283ee\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " Nov 02 17:41:39.423477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.423368 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f31e15-2da6-4b7e-9073-4457a2a283ee-kube-api-access-59jmz\" (UniqueName: \"kubernetes.io/projected/72f31e15-2da6-4b7e-9073-4457a2a283ee-kube-api-access-59jmz\") pod \"72f31e15-2da6-4b7e-9073-4457a2a283ee\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " Nov 02 17:41:39.423477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.423398 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-credential-socket\") pod \"72f31e15-2da6-4b7e-9073-4457a2a283ee\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " Nov 02 17:41:39.423477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.423416 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-workload-socket\") pod \"72f31e15-2da6-4b7e-9073-4457a2a283ee\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " Nov 02 17:41:39.423733 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.423491 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-workload-certs\") pod \"72f31e15-2da6-4b7e-9073-4457a2a283ee\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " Nov 02 17:41:39.423733 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.423527 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-envoy\") pod \"72f31e15-2da6-4b7e-9073-4457a2a283ee\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " Nov 02 17:41:39.423858 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.423706 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f31e15-2da6-4b7e-9073-4457a2a283ee-istiod-ca-cert" pod "72f31e15-2da6-4b7e-9073-4457a2a283ee" (UID: "72f31e15-2da6-4b7e-9073-4457a2a283ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:39.423914 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.423806 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-workload-certs" pod "72f31e15-2da6-4b7e-9073-4457a2a283ee" (UID: "72f31e15-2da6-4b7e-9073-4457a2a283ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:39.423914 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.423817 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-workload-socket" pod "72f31e15-2da6-4b7e-9073-4457a2a283ee" (UID: "72f31e15-2da6-4b7e-9073-4457a2a283ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:39.423914 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.423901 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-token\" (UniqueName: \"kubernetes.io/projected/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-token\") pod \"72f31e15-2da6-4b7e-9073-4457a2a283ee\" (UID: \"72f31e15-2da6-4b7e-9073-4457a2a283ee\") " Nov 02 17:41:39.424058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.423907 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-data" pod "72f31e15-2da6-4b7e-9073-4457a2a283ee" (UID: "72f31e15-2da6-4b7e-9073-4457a2a283ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:39.424093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.424050 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-credential-socket" pod "72f31e15-2da6-4b7e-9073-4457a2a283ee" (UID: "72f31e15-2da6-4b7e-9073-4457a2a283ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:39.424380 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.424355 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.424490 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.424385 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f31e15-2da6-4b7e-9073-4457a2a283ee-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.424490 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.424400 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.424490 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.424415 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.424490 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.424428 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.426135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.426106 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-podinfo" pod "72f31e15-2da6-4b7e-9073-4457a2a283ee" (UID: "72f31e15-2da6-4b7e-9073-4457a2a283ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:41:39.426135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.426112 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-envoy" pod "72f31e15-2da6-4b7e-9073-4457a2a283ee" (UID: "72f31e15-2da6-4b7e-9073-4457a2a283ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:39.426296 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.426245 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f31e15-2da6-4b7e-9073-4457a2a283ee-kube-api-access-59jmz" pod "72f31e15-2da6-4b7e-9073-4457a2a283ee" (UID: "72f31e15-2da6-4b7e-9073-4457a2a283ee"). InnerVolumeSpecName "kube-api-access-59jmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:39.426492 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.426473 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-token" pod "72f31e15-2da6-4b7e-9073-4457a2a283ee" (UID: "72f31e15-2da6-4b7e-9073-4457a2a283ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:39.525614 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.525527 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.525614 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.525568 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.525614 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.525583 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f31e15-2da6-4b7e-9073-4457a2a283ee-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.525614 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.525599 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59jmz\" (UniqueName: \"kubernetes.io/projected/72f31e15-2da6-4b7e-9073-4457a2a283ee-kube-api-access-59jmz\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.593946 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.593905 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:39.609257 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.609226 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97"] Nov 02 17:41:39.609839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.609773 5549 cpu_manager.go:414] "containerMap: removing container" podUID="a2059966-df3d-42a3-b05f-a00593fb40e6" containerName="istio-proxy" Nov 02 17:41:39.609839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.609799 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2059966-df3d-42a3-b05f-a00593fb40e6" containerName="istio-proxy" Nov 02 17:41:39.609839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.609815 5549 cpu_manager.go:414] "containerMap: removing container" podUID="72f31e15-2da6-4b7e-9073-4457a2a283ee" containerName="istio-proxy" Nov 02 17:41:39.609839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.609825 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f31e15-2da6-4b7e-9073-4457a2a283ee" containerName="istio-proxy" Nov 02 17:41:39.609839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.609839 5549 cpu_manager.go:414] "containerMap: removing container" podUID="8bf05dc6-3567-4d4e-b343-30e04158b39d" containerName="payload-pre-processing" Nov 02 17:41:39.609839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.609846 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf05dc6-3567-4d4e-b343-30e04158b39d" containerName="payload-pre-processing" Nov 02 17:41:39.610318 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.609860 5549 cpu_manager.go:414] "containerMap: removing container" podUID="72f31e15-2da6-4b7e-9073-4457a2a283ee" containerName="istio-proxy" Nov 02 17:41:39.610318 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.609868 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f31e15-2da6-4b7e-9073-4457a2a283ee" containerName="istio-proxy" Nov 02 17:41:39.610318 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.609883 5549 cpu_manager.go:414] "containerMap: removing container" podUID="2399aa13-f59f-431f-b775-bd0321ad7728" containerName="payload-processing" Nov 02 17:41:39.610318 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.609890 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2399aa13-f59f-431f-b775-bd0321ad7728" containerName="payload-processing" Nov 02 17:41:39.610318 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.610052 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2059966-df3d-42a3-b05f-a00593fb40e6" containerName="istio-proxy" Nov 02 17:41:39.610318 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.610072 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f31e15-2da6-4b7e-9073-4457a2a283ee" containerName="istio-proxy" Nov 02 17:41:39.610318 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.610083 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf05dc6-3567-4d4e-b343-30e04158b39d" containerName="payload-pre-processing" Nov 02 17:41:39.610318 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.610093 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f31e15-2da6-4b7e-9073-4457a2a283ee" containerName="istio-proxy" Nov 02 17:41:39.610318 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.610103 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="2399aa13-f59f-431f-b775-bd0321ad7728" containerName="payload-processing" Nov 02 17:41:39.614872 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.614841 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" Nov 02 17:41:39.620981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.620939 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-66eed1-dockercfg-cmzmp\"" Nov 02 17:41:39.622437 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.621439 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-66eed1\"" Nov 02 17:41:39.624248 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.624225 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97"] Nov 02 17:41:39.626235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.626203 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97\" (UID: \"fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" Nov 02 17:41:39.626339 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.626270 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j9dq\" (UniqueName: \"kubernetes.io/projected/fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb-kube-api-access-7j9dq\") pod \"payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97\" (UID: \"fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" Nov 02 17:41:39.670782 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.670747 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6"] Nov 02 17:41:39.671252 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.671236 5549 cpu_manager.go:414] "containerMap: removing container" podUID="a2059966-df3d-42a3-b05f-a00593fb40e6" containerName="istio-proxy" Nov 02 17:41:39.671310 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.671254 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2059966-df3d-42a3-b05f-a00593fb40e6" containerName="istio-proxy" Nov 02 17:41:39.671347 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.671321 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2059966-df3d-42a3-b05f-a00593fb40e6" containerName="istio-proxy" Nov 02 17:41:39.674424 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.674403 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" Nov 02 17:41:39.681357 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.681303 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq" podUID="c11e31cc-f03c-4d2c-a0e0-5859997fc096" containerName="payload-processing" containerID="cri-o://4f0ff2d1015f9d0a0dc30c92292785d1588b6f89a03666e83f4543955329e3b4" gracePeriod=25 Nov 02 17:41:39.688889 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.688860 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6"] Nov 02 17:41:39.724748 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.724714 5549 generic.go:356] "Generic (PLEG): container finished" podID="72f31e15-2da6-4b7e-9073-4457a2a283ee" containerID="653dd1d5d1ffaf832f6aaa2bcdd4f82c54de7692d75b9243de80cd5439533e1d" exitCode=0 Nov 02 17:41:39.724947 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.724784 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" Nov 02 17:41:39.724947 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.724797 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" event={"ID":"72f31e15-2da6-4b7e-9073-4457a2a283ee","Type":"ContainerDied","Data":"653dd1d5d1ffaf832f6aaa2bcdd4f82c54de7692d75b9243de80cd5439533e1d"} Nov 02 17:41:39.724947 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.724859 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s" event={"ID":"72f31e15-2da6-4b7e-9073-4457a2a283ee","Type":"ContainerDied","Data":"d32c5587f706d6700514606885668a0e53d29fc53328efaf355174b9f2211a7b"} Nov 02 17:41:39.724947 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.724885 5549 scope.go:122] "RemoveContainer" containerID="653dd1d5d1ffaf832f6aaa2bcdd4f82c54de7692d75b9243de80cd5439533e1d" Nov 02 17:41:39.726838 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.726811 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j9dq\" (UniqueName: \"kubernetes.io/projected/fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb-kube-api-access-7j9dq\") pod \"payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97\" (UID: \"fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" Nov 02 17:41:39.727034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.727011 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj5m4\" (UniqueName: \"kubernetes.io/projected/f5c74aae-fd45-4ba6-b30d-99fb59e22c11-kube-api-access-qj5m4\") pod \"payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6\" (UID: \"f5c74aae-fd45-4ba6-b30d-99fb59e22c11\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" Nov 02 17:41:39.727135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.727052 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97\" (UID: \"fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" Nov 02 17:41:39.727135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.727075 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c74aae-fd45-4ba6-b30d-99fb59e22c11-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6\" (UID: \"f5c74aae-fd45-4ba6-b30d-99fb59e22c11\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" Nov 02 17:41:39.727697 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.727680 5549 generic.go:356] "Generic (PLEG): container finished" podID="a2059966-df3d-42a3-b05f-a00593fb40e6" containerID="a8d001c3cc98c16440e5eb38aa502cc83c30ec836e4ec75939b8e6c6eadb159d" exitCode=0 Nov 02 17:41:39.727928 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.727883 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97\" (UID: \"fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" Nov 02 17:41:39.727998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.727990 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" Nov 02 17:41:39.728099 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.728047 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" event={"ID":"a2059966-df3d-42a3-b05f-a00593fb40e6","Type":"ContainerDied","Data":"a8d001c3cc98c16440e5eb38aa502cc83c30ec836e4ec75939b8e6c6eadb159d"} Nov 02 17:41:39.728173 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.728105 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs" event={"ID":"a2059966-df3d-42a3-b05f-a00593fb40e6","Type":"ContainerDied","Data":"b856325b0243d9a4041ff82c7635f61301b102cd9ce0a567aef4aa0913e92b3c"} Nov 02 17:41:39.734828 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.734808 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:39.737345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.737323 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j9dq\" (UniqueName: \"kubernetes.io/projected/fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb-kube-api-access-7j9dq\") pod \"payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97\" (UID: \"fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" Nov 02 17:41:39.756652 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.756620 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs"] Nov 02 17:41:39.759426 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.759397 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-868a2ee9-openshift-default-65ddd584cc-7xjhs"] Nov 02 17:41:39.771445 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.771409 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s"] Nov 02 17:41:39.775684 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.775620 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0ae9c41e-openshift-default-6c6bfd575c-bwc7s"] Nov 02 17:41:39.823180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.817988 5549 scope.go:122] "RemoveContainer" containerID="05981b10601454fc896609eee6c0fcdf440b15a1e5c1a10bc341f45e918bdc8f" Nov 02 17:41:39.835227 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.833197 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj5m4\" (UniqueName: \"kubernetes.io/projected/f5c74aae-fd45-4ba6-b30d-99fb59e22c11-kube-api-access-qj5m4\") pod \"payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6\" (UID: \"f5c74aae-fd45-4ba6-b30d-99fb59e22c11\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" Nov 02 17:41:39.835227 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.833242 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c74aae-fd45-4ba6-b30d-99fb59e22c11-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6\" (UID: \"f5c74aae-fd45-4ba6-b30d-99fb59e22c11\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" Nov 02 17:41:39.835227 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.834497 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c74aae-fd45-4ba6-b30d-99fb59e22c11-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6\" (UID: \"f5c74aae-fd45-4ba6-b30d-99fb59e22c11\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" Nov 02 17:41:39.845164 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.844872 5549 scope.go:122] "RemoveContainer" containerID="653dd1d5d1ffaf832f6aaa2bcdd4f82c54de7692d75b9243de80cd5439533e1d" Nov 02 17:41:39.845371 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:41:39.845339 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653dd1d5d1ffaf832f6aaa2bcdd4f82c54de7692d75b9243de80cd5439533e1d\": container with ID starting with 653dd1d5d1ffaf832f6aaa2bcdd4f82c54de7692d75b9243de80cd5439533e1d not found: ID does not exist" containerID="653dd1d5d1ffaf832f6aaa2bcdd4f82c54de7692d75b9243de80cd5439533e1d" Nov 02 17:41:39.845474 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.845377 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653dd1d5d1ffaf832f6aaa2bcdd4f82c54de7692d75b9243de80cd5439533e1d"} err="failed to get container status \"653dd1d5d1ffaf832f6aaa2bcdd4f82c54de7692d75b9243de80cd5439533e1d\": rpc error: code = NotFound desc = could not find container \"653dd1d5d1ffaf832f6aaa2bcdd4f82c54de7692d75b9243de80cd5439533e1d\": container with ID starting with 653dd1d5d1ffaf832f6aaa2bcdd4f82c54de7692d75b9243de80cd5439533e1d not found: ID does not exist" Nov 02 17:41:39.845474 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.845402 5549 scope.go:122] "RemoveContainer" containerID="05981b10601454fc896609eee6c0fcdf440b15a1e5c1a10bc341f45e918bdc8f" Nov 02 17:41:39.845733 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:41:39.845707 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05981b10601454fc896609eee6c0fcdf440b15a1e5c1a10bc341f45e918bdc8f\": container with ID starting with 05981b10601454fc896609eee6c0fcdf440b15a1e5c1a10bc341f45e918bdc8f not found: ID does not exist" containerID="05981b10601454fc896609eee6c0fcdf440b15a1e5c1a10bc341f45e918bdc8f" Nov 02 17:41:39.845786 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.845743 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05981b10601454fc896609eee6c0fcdf440b15a1e5c1a10bc341f45e918bdc8f"} err="failed to get container status \"05981b10601454fc896609eee6c0fcdf440b15a1e5c1a10bc341f45e918bdc8f\": rpc error: code = NotFound desc = could not find container \"05981b10601454fc896609eee6c0fcdf440b15a1e5c1a10bc341f45e918bdc8f\": container with ID starting with 05981b10601454fc896609eee6c0fcdf440b15a1e5c1a10bc341f45e918bdc8f not found: ID does not exist" Nov 02 17:41:39.845786 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.845767 5549 scope.go:122] "RemoveContainer" containerID="a8d001c3cc98c16440e5eb38aa502cc83c30ec836e4ec75939b8e6c6eadb159d" Nov 02 17:41:39.849403 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.849378 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj5m4\" (UniqueName: \"kubernetes.io/projected/f5c74aae-fd45-4ba6-b30d-99fb59e22c11-kube-api-access-qj5m4\") pod \"payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6\" (UID: \"f5c74aae-fd45-4ba6-b30d-99fb59e22c11\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" Nov 02 17:41:39.858551 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.858525 5549 scope.go:122] "RemoveContainer" containerID="6c7ed4dfb7d51c52cdc02f39e97e3b8fd1e3f2996c6ab3242842473787143b6b" Nov 02 17:41:39.868497 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.867825 5549 scope.go:122] "RemoveContainer" containerID="a8d001c3cc98c16440e5eb38aa502cc83c30ec836e4ec75939b8e6c6eadb159d" Nov 02 17:41:39.868497 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:41:39.868210 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d001c3cc98c16440e5eb38aa502cc83c30ec836e4ec75939b8e6c6eadb159d\": container with ID starting with a8d001c3cc98c16440e5eb38aa502cc83c30ec836e4ec75939b8e6c6eadb159d not found: ID does not exist" containerID="a8d001c3cc98c16440e5eb38aa502cc83c30ec836e4ec75939b8e6c6eadb159d" Nov 02 17:41:39.868497 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.868244 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d001c3cc98c16440e5eb38aa502cc83c30ec836e4ec75939b8e6c6eadb159d"} err="failed to get container status \"a8d001c3cc98c16440e5eb38aa502cc83c30ec836e4ec75939b8e6c6eadb159d\": rpc error: code = NotFound desc = could not find container \"a8d001c3cc98c16440e5eb38aa502cc83c30ec836e4ec75939b8e6c6eadb159d\": container with ID starting with a8d001c3cc98c16440e5eb38aa502cc83c30ec836e4ec75939b8e6c6eadb159d not found: ID does not exist" Nov 02 17:41:39.868497 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.868273 5549 scope.go:122] "RemoveContainer" containerID="6c7ed4dfb7d51c52cdc02f39e97e3b8fd1e3f2996c6ab3242842473787143b6b" Nov 02 17:41:39.868654 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:41:39.868519 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c7ed4dfb7d51c52cdc02f39e97e3b8fd1e3f2996c6ab3242842473787143b6b\": container with ID starting with 6c7ed4dfb7d51c52cdc02f39e97e3b8fd1e3f2996c6ab3242842473787143b6b not found: ID does not exist" containerID="6c7ed4dfb7d51c52cdc02f39e97e3b8fd1e3f2996c6ab3242842473787143b6b" Nov 02 17:41:39.868654 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.868538 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c7ed4dfb7d51c52cdc02f39e97e3b8fd1e3f2996c6ab3242842473787143b6b"} err="failed to get container status \"6c7ed4dfb7d51c52cdc02f39e97e3b8fd1e3f2996c6ab3242842473787143b6b\": rpc error: code = NotFound desc = could not find container \"6c7ed4dfb7d51c52cdc02f39e97e3b8fd1e3f2996c6ab3242842473787143b6b\": container with ID starting with 6c7ed4dfb7d51c52cdc02f39e97e3b8fd1e3f2996c6ab3242842473787143b6b not found: ID does not exist" Nov 02 17:41:39.891646 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.891604 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:39.930767 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.930659 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" Nov 02 17:41:39.970752 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.970724 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq" Nov 02 17:41:39.995832 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:39.993431 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" Nov 02 17:41:40.034850 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.034758 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c11e31cc-f03c-4d2c-a0e0-5859997fc096-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11e31cc-f03c-4d2c-a0e0-5859997fc096-plugins-config-volume\") pod \"c11e31cc-f03c-4d2c-a0e0-5859997fc096\" (UID: \"c11e31cc-f03c-4d2c-a0e0-5859997fc096\") " Nov 02 17:41:40.034850 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.034837 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c11e31cc-f03c-4d2c-a0e0-5859997fc096-kube-api-access-fbl6w\" (UniqueName: \"kubernetes.io/projected/c11e31cc-f03c-4d2c-a0e0-5859997fc096-kube-api-access-fbl6w\") pod \"c11e31cc-f03c-4d2c-a0e0-5859997fc096\" (UID: \"c11e31cc-f03c-4d2c-a0e0-5859997fc096\") " Nov 02 17:41:40.035395 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.035367 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c11e31cc-f03c-4d2c-a0e0-5859997fc096-plugins-config-volume" pod "c11e31cc-f03c-4d2c-a0e0-5859997fc096" (UID: "c11e31cc-f03c-4d2c-a0e0-5859997fc096"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:40.037661 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.037430 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11e31cc-f03c-4d2c-a0e0-5859997fc096-kube-api-access-fbl6w" pod "c11e31cc-f03c-4d2c-a0e0-5859997fc096" (UID: "c11e31cc-f03c-4d2c-a0e0-5859997fc096"). InnerVolumeSpecName "kube-api-access-fbl6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:40.045244 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.045202 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:40.135728 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.135695 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11e31cc-f03c-4d2c-a0e0-5859997fc096-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:40.135728 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.135725 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbl6w\" (UniqueName: \"kubernetes.io/projected/c11e31cc-f03c-4d2c-a0e0-5859997fc096-kube-api-access-fbl6w\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:40.216581 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.216551 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97"] Nov 02 17:41:40.218665 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:41:40.218633 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd37ac93_1d69_49a9_82b0_e5b77d3ebcfb.slice/crio-971446573f40c5894508a9bbf9bbd9604736158480777bcee6411f8dda26ab97 WatchSource:0}: Error finding container 971446573f40c5894508a9bbf9bbd9604736158480777bcee6411f8dda26ab97: Status 404 returned error can't find the container with id 971446573f40c5894508a9bbf9bbd9604736158480777bcee6411f8dda26ab97 Nov 02 17:41:40.227023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.226997 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6"] Nov 02 17:41:40.227433 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.227421 5549 cpu_manager.go:414] "containerMap: removing container" podUID="c11e31cc-f03c-4d2c-a0e0-5859997fc096" containerName="payload-processing" Nov 02 17:41:40.227479 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.227435 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c11e31cc-f03c-4d2c-a0e0-5859997fc096" containerName="payload-processing" Nov 02 17:41:40.227514 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.227493 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c11e31cc-f03c-4d2c-a0e0-5859997fc096" containerName="payload-processing" Nov 02 17:41:40.231696 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.231678 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 17:41:40.236748 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.236723 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71734cf2-241f-47d9-ba84-c4fededcda62-maas-api-tls\") pod \"maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6\" (UID: \"71734cf2-241f-47d9-ba84-c4fededcda62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 17:41:40.236873 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.236851 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71734cf2-241f-47d9-ba84-c4fededcda62-metrics-certs\") pod \"maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6\" (UID: \"71734cf2-241f-47d9-ba84-c4fededcda62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 17:41:40.236972 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.236933 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71734cf2-241f-47d9-ba84-c4fededcda62-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6\" (UID: \"71734cf2-241f-47d9-ba84-c4fededcda62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 17:41:40.237063 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.237043 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w45zt\" (UniqueName: \"kubernetes.io/projected/71734cf2-241f-47d9-ba84-c4fededcda62-kube-api-access-w45zt\") pod \"maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6\" (UID: \"71734cf2-241f-47d9-ba84-c4fededcda62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 17:41:40.248593 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.248566 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6"] Nov 02 17:41:40.253461 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.253436 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-66eed1-serving-cert\"" Nov 02 17:41:40.287701 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.287677 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6"] Nov 02 17:41:40.293087 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:41:40.293056 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c74aae_fd45_4ba6_b30d_99fb59e22c11.slice/crio-e62748934f70e2f78b7caf29748441c1d3abf0ae35c0089cf1269c1b01c894dd WatchSource:0}: Error finding container e62748934f70e2f78b7caf29748441c1d3abf0ae35c0089cf1269c1b01c894dd: Status 404 returned error can't find the container with id e62748934f70e2f78b7caf29748441c1d3abf0ae35c0089cf1269c1b01c894dd Nov 02 17:41:40.321641 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.321613 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2399aa13-f59f-431f-b775-bd0321ad7728" path="/var/lib/kubelet/pods/2399aa13-f59f-431f-b775-bd0321ad7728/volumes" Nov 02 17:41:40.322103 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.322073 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f31e15-2da6-4b7e-9073-4457a2a283ee" path="/var/lib/kubelet/pods/72f31e15-2da6-4b7e-9073-4457a2a283ee/volumes" Nov 02 17:41:40.322489 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.322474 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf05dc6-3567-4d4e-b343-30e04158b39d" path="/var/lib/kubelet/pods/8bf05dc6-3567-4d4e-b343-30e04158b39d/volumes" Nov 02 17:41:40.323196 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.323181 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2059966-df3d-42a3-b05f-a00593fb40e6" path="/var/lib/kubelet/pods/a2059966-df3d-42a3-b05f-a00593fb40e6/volumes" Nov 02 17:41:40.337983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.337929 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71734cf2-241f-47d9-ba84-c4fededcda62-metrics-certs\") pod \"maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6\" (UID: \"71734cf2-241f-47d9-ba84-c4fededcda62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 17:41:40.338146 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.338026 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71734cf2-241f-47d9-ba84-c4fededcda62-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6\" (UID: \"71734cf2-241f-47d9-ba84-c4fededcda62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 17:41:40.338146 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.338058 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w45zt\" (UniqueName: \"kubernetes.io/projected/71734cf2-241f-47d9-ba84-c4fededcda62-kube-api-access-w45zt\") pod \"maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6\" (UID: \"71734cf2-241f-47d9-ba84-c4fededcda62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 17:41:40.338256 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.338186 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71734cf2-241f-47d9-ba84-c4fededcda62-maas-api-tls\") pod \"maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6\" (UID: \"71734cf2-241f-47d9-ba84-c4fededcda62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 17:41:40.338350 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:41:40.338328 5549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-serving-cert: secret "maas-api-e2e-worker-w2-66eed1-serving-cert" not found Nov 02 17:41:40.338415 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:41:40.338404 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71734cf2-241f-47d9-ba84-c4fededcda62-maas-api-tls podName:71734cf2-241f-47d9-ba84-c4fededcda62 nodeName:}" failed. No retries permitted until 2026-11-02 17:41:40.838386574 +0000 UTC m=+472.626892084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71734cf2-241f-47d9-ba84-c4fededcda62-maas-api-tls") pod "maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" (UID: "71734cf2-241f-47d9-ba84-c4fededcda62") : secret "maas-api-e2e-worker-w2-66eed1-serving-cert" not found Nov 02 17:41:40.338716 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.338697 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71734cf2-241f-47d9-ba84-c4fededcda62-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6\" (UID: \"71734cf2-241f-47d9-ba84-c4fededcda62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 17:41:40.340323 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.340301 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71734cf2-241f-47d9-ba84-c4fededcda62-metrics-certs\") pod \"maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6\" (UID: \"71734cf2-241f-47d9-ba84-c4fededcda62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 17:41:40.351727 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.351702 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w45zt\" (UniqueName: \"kubernetes.io/projected/71734cf2-241f-47d9-ba84-c4fededcda62-kube-api-access-w45zt\") pod \"maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6\" (UID: \"71734cf2-241f-47d9-ba84-c4fededcda62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 17:41:40.401192 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.401158 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:40.568027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.567904 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:40.739303 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.739267 5549 generic.go:356] "Generic (PLEG): container finished" podID="c11e31cc-f03c-4d2c-a0e0-5859997fc096" containerID="4f0ff2d1015f9d0a0dc30c92292785d1588b6f89a03666e83f4543955329e3b4" exitCode=0 Nov 02 17:41:40.739484 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.739326 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq" Nov 02 17:41:40.739484 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.739356 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq" event={"ID":"c11e31cc-f03c-4d2c-a0e0-5859997fc096","Type":"ContainerDied","Data":"4f0ff2d1015f9d0a0dc30c92292785d1588b6f89a03666e83f4543955329e3b4"} Nov 02 17:41:40.739484 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.739438 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq" event={"ID":"c11e31cc-f03c-4d2c-a0e0-5859997fc096","Type":"ContainerDied","Data":"05710188881e1ac80eccff48d5f3aced3da0c1e5bddcbe297ad559bcce1803a6"} Nov 02 17:41:40.739484 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.739466 5549 scope.go:122] "RemoveContainer" containerID="4f0ff2d1015f9d0a0dc30c92292785d1588b6f89a03666e83f4543955329e3b4" Nov 02 17:41:40.741702 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.741681 5549 generic.go:356] "Generic (PLEG): container finished" podID="43b648d1-eab0-4625-bf8a-d708d790ab14" containerID="1a5eabbb19782a7efbcb811e76d70cfefbfc97b047a714d67b61b1ea4b285433" exitCode=0 Nov 02 17:41:40.741806 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.741776 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" event={"ID":"43b648d1-eab0-4625-bf8a-d708d790ab14","Type":"ContainerDied","Data":"1a5eabbb19782a7efbcb811e76d70cfefbfc97b047a714d67b61b1ea4b285433"} Nov 02 17:41:40.743412 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.743384 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" event={"ID":"f5c74aae-fd45-4ba6-b30d-99fb59e22c11","Type":"ContainerStarted","Data":"dda7dc55e039bb6afaebb05a9d41917182af703242d326dae7f73b14001ed63d"} Nov 02 17:41:40.743533 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.743436 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" event={"ID":"f5c74aae-fd45-4ba6-b30d-99fb59e22c11","Type":"ContainerStarted","Data":"e62748934f70e2f78b7caf29748441c1d3abf0ae35c0089cf1269c1b01c894dd"} Nov 02 17:41:40.743533 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.743465 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" Nov 02 17:41:40.745006 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.744983 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" event={"ID":"fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb","Type":"ContainerStarted","Data":"eea96c58c7cc8e36a7c079b01efa5ae509f1f7248782db5214ce113f6e146a5c"} Nov 02 17:41:40.745082 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.745028 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" event={"ID":"fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb","Type":"ContainerStarted","Data":"971446573f40c5894508a9bbf9bbd9604736158480777bcee6411f8dda26ab97"} Nov 02 17:41:40.745227 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.745205 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" Nov 02 17:41:40.748842 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.748826 5549 scope.go:122] "RemoveContainer" containerID="4f0ff2d1015f9d0a0dc30c92292785d1588b6f89a03666e83f4543955329e3b4" Nov 02 17:41:40.749107 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:41:40.749090 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0ff2d1015f9d0a0dc30c92292785d1588b6f89a03666e83f4543955329e3b4\": container with ID starting with 4f0ff2d1015f9d0a0dc30c92292785d1588b6f89a03666e83f4543955329e3b4 not found: ID does not exist" containerID="4f0ff2d1015f9d0a0dc30c92292785d1588b6f89a03666e83f4543955329e3b4" Nov 02 17:41:40.749151 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.749115 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0ff2d1015f9d0a0dc30c92292785d1588b6f89a03666e83f4543955329e3b4"} err="failed to get container status \"4f0ff2d1015f9d0a0dc30c92292785d1588b6f89a03666e83f4543955329e3b4\": rpc error: code = NotFound desc = could not find container \"4f0ff2d1015f9d0a0dc30c92292785d1588b6f89a03666e83f4543955329e3b4\": container with ID starting with 4f0ff2d1015f9d0a0dc30c92292785d1588b6f89a03666e83f4543955329e3b4 not found: ID does not exist" Nov 02 17:41:40.749151 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.749133 5549 scope.go:122] "RemoveContainer" containerID="1397e74de2801f5ba154ac53ba8d5d0d4c40ad7b1688f24b9f71c09e38dce23a" Nov 02 17:41:40.769101 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.769053 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" podStartSLOduration=1.7690387699999999 podStartE2EDuration="1.76903877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:40.767825449 +0000 UTC m=+472.556330978" watchObservedRunningTime="2026-11-02 17:41:40.76903877 +0000 UTC m=+472.557544305" Nov 02 17:41:40.788202 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.788158 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq"] Nov 02 17:41:40.790613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.790575 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-55f9886f-5bd5b6fcd9-c5gdq"] Nov 02 17:41:40.806254 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.806204 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" podStartSLOduration=1.806187647 podStartE2EDuration="1.806187647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:40.805774318 +0000 UTC m=+472.594279846" watchObservedRunningTime="2026-11-02 17:41:40.806187647 +0000 UTC m=+472.594693204" Nov 02 17:41:40.844361 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.844267 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71734cf2-241f-47d9-ba84-c4fededcda62-maas-api-tls\") pod \"maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6\" (UID: \"71734cf2-241f-47d9-ba84-c4fededcda62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 17:41:40.846749 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.846724 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71734cf2-241f-47d9-ba84-c4fededcda62-maas-api-tls\") pod \"maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6\" (UID: \"71734cf2-241f-47d9-ba84-c4fededcda62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 17:41:40.955930 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.955770 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 17:41:40.957753 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.957727 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh"] Nov 02 17:41:40.961303 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.961288 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" Nov 02 17:41:40.969546 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.969519 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-4b97c6\"" Nov 02 17:41:40.970773 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.970751 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-4b97c6-dockercfg-gjb7h\"" Nov 02 17:41:40.971916 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.971890 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh"] Nov 02 17:41:40.982699 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:40.982672 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:41.046802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.046763 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skd2l\" (UniqueName: \"kubernetes.io/projected/55ebe7be-c46e-4fb3-b7df-cf2a522b9f70-kube-api-access-skd2l\") pod \"payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh\" (UID: \"55ebe7be-c46e-4fb3-b7df-cf2a522b9f70\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" Nov 02 17:41:41.047042 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.047016 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55ebe7be-c46e-4fb3-b7df-cf2a522b9f70-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh\" (UID: \"55ebe7be-c46e-4fb3-b7df-cf2a522b9f70\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" Nov 02 17:41:41.127363 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.127282 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:41.150064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.148635 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55ebe7be-c46e-4fb3-b7df-cf2a522b9f70-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh\" (UID: \"55ebe7be-c46e-4fb3-b7df-cf2a522b9f70\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" Nov 02 17:41:41.150064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.148694 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skd2l\" (UniqueName: \"kubernetes.io/projected/55ebe7be-c46e-4fb3-b7df-cf2a522b9f70-kube-api-access-skd2l\") pod \"payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh\" (UID: \"55ebe7be-c46e-4fb3-b7df-cf2a522b9f70\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" Nov 02 17:41:41.150064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.149989 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55ebe7be-c46e-4fb3-b7df-cf2a522b9f70-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh\" (UID: \"55ebe7be-c46e-4fb3-b7df-cf2a522b9f70\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" Nov 02 17:41:41.173467 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.173407 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skd2l\" (UniqueName: \"kubernetes.io/projected/55ebe7be-c46e-4fb3-b7df-cf2a522b9f70-kube-api-access-skd2l\") pod \"payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh\" (UID: \"55ebe7be-c46e-4fb3-b7df-cf2a522b9f70\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" Nov 02 17:41:41.236773 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.236744 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6"] Nov 02 17:41:41.238516 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:41:41.238485 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71734cf2_241f_47d9_ba84_c4fededcda62.slice/crio-ae5eca0cbeb63bf9415f7e46d88e55ede6a041038a9bba855e1d04173f4a92bf WatchSource:0}: Error finding container ae5eca0cbeb63bf9415f7e46d88e55ede6a041038a9bba855e1d04173f4a92bf: Status 404 returned error can't find the container with id ae5eca0cbeb63bf9415f7e46d88e55ede6a041038a9bba855e1d04173f4a92bf Nov 02 17:41:41.256156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.256133 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:41:41.277934 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.277905 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" Nov 02 17:41:41.343727 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.342445 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:41.351495 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.351455 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-envoy\") pod \"43b648d1-eab0-4625-bf8a-d708d790ab14\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " Nov 02 17:41:41.351645 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.351565 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-data\") pod \"43b648d1-eab0-4625-bf8a-d708d790ab14\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " Nov 02 17:41:41.351645 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.351604 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b648d1-eab0-4625-bf8a-d708d790ab14-kube-api-access-jsqsg\" (UniqueName: \"kubernetes.io/projected/43b648d1-eab0-4625-bf8a-d708d790ab14-kube-api-access-jsqsg\") pod \"43b648d1-eab0-4625-bf8a-d708d790ab14\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " Nov 02 17:41:41.351765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.351659 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-podinfo\") pod \"43b648d1-eab0-4625-bf8a-d708d790ab14\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " Nov 02 17:41:41.351765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.351737 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-credential-socket\") pod \"43b648d1-eab0-4625-bf8a-d708d790ab14\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " Nov 02 17:41:41.351865 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.351790 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b648d1-eab0-4625-bf8a-d708d790ab14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b648d1-eab0-4625-bf8a-d708d790ab14-istiod-ca-cert\") pod \"43b648d1-eab0-4625-bf8a-d708d790ab14\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " Nov 02 17:41:41.351865 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.351823 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-workload-certs\") pod \"43b648d1-eab0-4625-bf8a-d708d790ab14\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " Nov 02 17:41:41.351980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.351864 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-token\" (UniqueName: \"kubernetes.io/projected/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-token\") pod \"43b648d1-eab0-4625-bf8a-d708d790ab14\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " Nov 02 17:41:41.351980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.351913 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-workload-socket\") pod \"43b648d1-eab0-4625-bf8a-d708d790ab14\" (UID: \"43b648d1-eab0-4625-bf8a-d708d790ab14\") " Nov 02 17:41:41.352581 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.352551 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-workload-socket" pod "43b648d1-eab0-4625-bf8a-d708d790ab14" (UID: "43b648d1-eab0-4625-bf8a-d708d790ab14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:41.353307 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.353283 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-workload-certs" pod "43b648d1-eab0-4625-bf8a-d708d790ab14" (UID: "43b648d1-eab0-4625-bf8a-d708d790ab14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:41.353614 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.353590 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b648d1-eab0-4625-bf8a-d708d790ab14-istiod-ca-cert" pod "43b648d1-eab0-4625-bf8a-d708d790ab14" (UID: "43b648d1-eab0-4625-bf8a-d708d790ab14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:41.353938 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.353912 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-credential-socket" pod "43b648d1-eab0-4625-bf8a-d708d790ab14" (UID: "43b648d1-eab0-4625-bf8a-d708d790ab14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:41.354225 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.354204 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-data" pod "43b648d1-eab0-4625-bf8a-d708d790ab14" (UID: "43b648d1-eab0-4625-bf8a-d708d790ab14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:41.362327 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.362289 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b648d1-eab0-4625-bf8a-d708d790ab14-kube-api-access-jsqsg" pod "43b648d1-eab0-4625-bf8a-d708d790ab14" (UID: "43b648d1-eab0-4625-bf8a-d708d790ab14"). InnerVolumeSpecName "kube-api-access-jsqsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:41.362505 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.362482 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-token" pod "43b648d1-eab0-4625-bf8a-d708d790ab14" (UID: "43b648d1-eab0-4625-bf8a-d708d790ab14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:41.363172 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.363148 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-envoy" pod "43b648d1-eab0-4625-bf8a-d708d790ab14" (UID: "43b648d1-eab0-4625-bf8a-d708d790ab14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:41.367375 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.367351 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-podinfo" pod "43b648d1-eab0-4625-bf8a-d708d790ab14" (UID: "43b648d1-eab0-4625-bf8a-d708d790ab14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:41:41.453844 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.453806 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:41.453844 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.453840 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:41.454138 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.453854 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsqsg\" (UniqueName: \"kubernetes.io/projected/43b648d1-eab0-4625-bf8a-d708d790ab14-kube-api-access-jsqsg\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:41.454138 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.453867 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:41.454138 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.453878 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:41.454138 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.453892 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b648d1-eab0-4625-bf8a-d708d790ab14-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:41.454138 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.453905 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:41.454138 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.453917 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b648d1-eab0-4625-bf8a-d708d790ab14-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:41.454138 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.453929 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b648d1-eab0-4625-bf8a-d708d790ab14-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:41.568376 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.568339 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh"] Nov 02 17:41:41.572368 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:41:41.572333 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ebe7be_c46e_4fb3_b7df_cf2a522b9f70.slice/crio-c3706c17746f00e66fed82c8a973ee2608d3ea8331e4dd108e66df340410a0f2 WatchSource:0}: Error finding container c3706c17746f00e66fed82c8a973ee2608d3ea8331e4dd108e66df340410a0f2: Status 404 returned error can't find the container with id c3706c17746f00e66fed82c8a973ee2608d3ea8331e4dd108e66df340410a0f2 Nov 02 17:41:41.751913 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.751835 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" Nov 02 17:41:41.751913 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.751844 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k" event={"ID":"43b648d1-eab0-4625-bf8a-d708d790ab14","Type":"ContainerDied","Data":"d013c620e60793fb5078770ab64c6363ea0ed824134ac7e3e1c36d977777d95f"} Nov 02 17:41:41.752152 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.751913 5549 scope.go:122] "RemoveContainer" containerID="1a5eabbb19782a7efbcb811e76d70cfefbfc97b047a714d67b61b1ea4b285433" Nov 02 17:41:41.753787 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.753745 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" event={"ID":"71734cf2-241f-47d9-ba84-c4fededcda62","Type":"ContainerStarted","Data":"ccc7e107a710dea5ccbbe3f376b0300c721ac3ac850eab232508c922f9f95f7c"} Nov 02 17:41:41.753902 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.753798 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 17:41:41.753902 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.753822 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" event={"ID":"71734cf2-241f-47d9-ba84-c4fededcda62","Type":"ContainerStarted","Data":"ae5eca0cbeb63bf9415f7e46d88e55ede6a041038a9bba855e1d04173f4a92bf"} Nov 02 17:41:41.755809 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.755714 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" event={"ID":"55ebe7be-c46e-4fb3-b7df-cf2a522b9f70","Type":"ContainerStarted","Data":"28bab6f9e8a30d1cecd87e2d7dc87bd816b039eed24e027d9b148fbe57f5aad7"} Nov 02 17:41:41.755809 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.755767 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" event={"ID":"55ebe7be-c46e-4fb3-b7df-cf2a522b9f70","Type":"ContainerStarted","Data":"c3706c17746f00e66fed82c8a973ee2608d3ea8331e4dd108e66df340410a0f2"} Nov 02 17:41:41.755995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.755904 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" Nov 02 17:41:41.775653 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.775604 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" podStartSLOduration=1.7755911979999999 podStartE2EDuration="1.775591198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:41.775266852 +0000 UTC m=+473.563772373" watchObservedRunningTime="2026-11-02 17:41:41.775591198 +0000 UTC m=+473.564096725" Nov 02 17:41:41.796459 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.796407 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" podStartSLOduration=1.393788813 podStartE2EDuration="1.796396435s" totalImagesPullingTime="402.607622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:41.794145179 +0000 UTC m=+473.582650704" watchObservedRunningTime="2026-11-02 17:41:41.796396435 +0000 UTC m=+473.584901963" Nov 02 17:41:41.811007 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.810977 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k"] Nov 02 17:41:41.814442 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.814408 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:41.815893 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.815868 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ad831efe-openshift-default-5c8ddf497f-pcz6k"] Nov 02 17:41:41.826253 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.826226 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth"] Nov 02 17:41:41.826690 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.826677 5549 cpu_manager.go:414] "containerMap: removing container" podUID="43b648d1-eab0-4625-bf8a-d708d790ab14" containerName="istio-proxy" Nov 02 17:41:41.826749 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.826692 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b648d1-eab0-4625-bf8a-d708d790ab14" containerName="istio-proxy" Nov 02 17:41:41.826749 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.826701 5549 cpu_manager.go:414] "containerMap: removing container" podUID="43b648d1-eab0-4625-bf8a-d708d790ab14" containerName="istio-proxy" Nov 02 17:41:41.826749 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.826707 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b648d1-eab0-4625-bf8a-d708d790ab14" containerName="istio-proxy" Nov 02 17:41:41.826852 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.826800 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b648d1-eab0-4625-bf8a-d708d790ab14" containerName="istio-proxy" Nov 02 17:41:41.826852 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.826810 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b648d1-eab0-4625-bf8a-d708d790ab14" containerName="istio-proxy" Nov 02 17:41:41.831295 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.831276 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" Nov 02 17:41:41.839042 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.839020 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-24cfb6-dockercfg-hkj5m\"" Nov 02 17:41:41.839331 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.839312 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-24cfb6\"" Nov 02 17:41:41.842115 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.842092 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth"] Nov 02 17:41:41.856769 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.856738 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060e829c-0a72-48a9-b9db-13db919d875f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth\" (UID: \"060e829c-0a72-48a9-b9db-13db919d875f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" Nov 02 17:41:41.857264 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.857236 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9j25\" (UniqueName: \"kubernetes.io/projected/060e829c-0a72-48a9-b9db-13db919d875f-kube-api-access-d9j25\") pod \"payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth\" (UID: \"060e829c-0a72-48a9-b9db-13db919d875f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" Nov 02 17:41:41.864202 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.864171 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f"] Nov 02 17:41:41.867745 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.867724 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" Nov 02 17:41:41.878219 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.878184 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f"] Nov 02 17:41:41.955519 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.955475 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:41.958496 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.958464 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060e829c-0a72-48a9-b9db-13db919d875f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth\" (UID: \"060e829c-0a72-48a9-b9db-13db919d875f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" Nov 02 17:41:41.959470 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.959441 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060e829c-0a72-48a9-b9db-13db919d875f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth\" (UID: \"060e829c-0a72-48a9-b9db-13db919d875f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" Nov 02 17:41:41.959578 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.959560 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f\" (UID: \"3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" Nov 02 17:41:41.959740 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.959713 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9j25\" (UniqueName: \"kubernetes.io/projected/060e829c-0a72-48a9-b9db-13db919d875f-kube-api-access-d9j25\") pod \"payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth\" (UID: \"060e829c-0a72-48a9-b9db-13db919d875f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" Nov 02 17:41:41.959842 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.959820 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmr76\" (UniqueName: \"kubernetes.io/projected/3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0-kube-api-access-rmr76\") pod \"payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f\" (UID: \"3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" Nov 02 17:41:41.968336 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:41.968306 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9j25\" (UniqueName: \"kubernetes.io/projected/060e829c-0a72-48a9-b9db-13db919d875f-kube-api-access-d9j25\") pod \"payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth\" (UID: \"060e829c-0a72-48a9-b9db-13db919d875f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" Nov 02 17:41:42.061854 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.061732 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmr76\" (UniqueName: \"kubernetes.io/projected/3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0-kube-api-access-rmr76\") pod \"payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f\" (UID: \"3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" Nov 02 17:41:42.062088 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.061869 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f\" (UID: \"3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" Nov 02 17:41:42.062803 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.062776 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f\" (UID: \"3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" Nov 02 17:41:42.072455 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.072409 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmr76\" (UniqueName: \"kubernetes.io/projected/3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0-kube-api-access-rmr76\") pod \"payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f\" (UID: \"3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" Nov 02 17:41:42.147371 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.147339 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" Nov 02 17:41:42.174420 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.174390 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:42.182392 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.182355 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" Nov 02 17:41:42.186290 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.186265 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl"] Nov 02 17:41:42.190604 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.190582 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" Nov 02 17:41:42.200066 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.200040 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-77f103\"" Nov 02 17:41:42.202719 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.202677 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl"] Nov 02 17:41:42.228318 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.228284 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8"] Nov 02 17:41:42.233161 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.233129 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" Nov 02 17:41:42.239773 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.239692 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8"] Nov 02 17:41:42.241087 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.241006 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-77f103-dockercfg-wpck6\"" Nov 02 17:41:42.264600 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.264567 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23593f20-8dc6-4ff6-93f4-d9b1fc859341-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl\" (UID: \"23593f20-8dc6-4ff6-93f4-d9b1fc859341\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" Nov 02 17:41:42.264746 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.264678 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqlg4\" (UniqueName: \"kubernetes.io/projected/23593f20-8dc6-4ff6-93f4-d9b1fc859341-kube-api-access-zqlg4\") pod \"payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl\" (UID: \"23593f20-8dc6-4ff6-93f4-d9b1fc859341\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" Nov 02 17:41:42.264746 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.264719 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609d560e-9146-4fbf-9aeb-1ca65e5d3971-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8\" (UID: \"609d560e-9146-4fbf-9aeb-1ca65e5d3971\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" Nov 02 17:41:42.264746 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.264742 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb2w9\" (UniqueName: \"kubernetes.io/projected/609d560e-9146-4fbf-9aeb-1ca65e5d3971-kube-api-access-zb2w9\") pod \"payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8\" (UID: \"609d560e-9146-4fbf-9aeb-1ca65e5d3971\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" Nov 02 17:41:42.327732 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.327072 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:42.332341 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.332272 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b648d1-eab0-4625-bf8a-d708d790ab14" path="/var/lib/kubelet/pods/43b648d1-eab0-4625-bf8a-d708d790ab14/volumes" Nov 02 17:41:42.332858 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.332824 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c11e31cc-f03c-4d2c-a0e0-5859997fc096" path="/var/lib/kubelet/pods/c11e31cc-f03c-4d2c-a0e0-5859997fc096/volumes" Nov 02 17:41:42.366139 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.366101 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609d560e-9146-4fbf-9aeb-1ca65e5d3971-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8\" (UID: \"609d560e-9146-4fbf-9aeb-1ca65e5d3971\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" Nov 02 17:41:42.366139 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.366140 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb2w9\" (UniqueName: \"kubernetes.io/projected/609d560e-9146-4fbf-9aeb-1ca65e5d3971-kube-api-access-zb2w9\") pod \"payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8\" (UID: \"609d560e-9146-4fbf-9aeb-1ca65e5d3971\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" Nov 02 17:41:42.366349 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.366223 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23593f20-8dc6-4ff6-93f4-d9b1fc859341-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl\" (UID: \"23593f20-8dc6-4ff6-93f4-d9b1fc859341\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" Nov 02 17:41:42.366397 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.366373 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqlg4\" (UniqueName: \"kubernetes.io/projected/23593f20-8dc6-4ff6-93f4-d9b1fc859341-kube-api-access-zqlg4\") pod \"payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl\" (UID: \"23593f20-8dc6-4ff6-93f4-d9b1fc859341\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" Nov 02 17:41:42.367098 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.367072 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23593f20-8dc6-4ff6-93f4-d9b1fc859341-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl\" (UID: \"23593f20-8dc6-4ff6-93f4-d9b1fc859341\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" Nov 02 17:41:42.367190 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.367072 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609d560e-9146-4fbf-9aeb-1ca65e5d3971-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8\" (UID: \"609d560e-9146-4fbf-9aeb-1ca65e5d3971\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" Nov 02 17:41:42.374829 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.374804 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqlg4\" (UniqueName: \"kubernetes.io/projected/23593f20-8dc6-4ff6-93f4-d9b1fc859341-kube-api-access-zqlg4\") pod \"payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl\" (UID: \"23593f20-8dc6-4ff6-93f4-d9b1fc859341\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" Nov 02 17:41:42.375385 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.375366 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb2w9\" (UniqueName: \"kubernetes.io/projected/609d560e-9146-4fbf-9aeb-1ca65e5d3971-kube-api-access-zb2w9\") pod \"payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8\" (UID: \"609d560e-9146-4fbf-9aeb-1ca65e5d3971\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" Nov 02 17:41:42.445218 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.445144 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth"] Nov 02 17:41:42.447425 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:41:42.447393 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060e829c_0a72_48a9_b9db_13db919d875f.slice/crio-0cc307e641548f5fd1e4b0c1021166020b8876a3ca5dc01715416e382e134e16 WatchSource:0}: Error finding container 0cc307e641548f5fd1e4b0c1021166020b8876a3ca5dc01715416e382e134e16: Status 404 returned error can't find the container with id 0cc307e641548f5fd1e4b0c1021166020b8876a3ca5dc01715416e382e134e16 Nov 02 17:41:42.480935 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.480909 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f"] Nov 02 17:41:42.483197 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:41:42.483163 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8e6a76_9bdc_47d3_9e0f_cef5d546c3d0.slice/crio-889e8bd5fb04401520c19787d4eeb7b8fd12cba426d2a9b2e8b3fdb672e8b85e WatchSource:0}: Error finding container 889e8bd5fb04401520c19787d4eeb7b8fd12cba426d2a9b2e8b3fdb672e8b85e: Status 404 returned error can't find the container with id 889e8bd5fb04401520c19787d4eeb7b8fd12cba426d2a9b2e8b3fdb672e8b85e Nov 02 17:41:42.527137 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.527107 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" Nov 02 17:41:42.553452 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.553417 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:42.559080 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.559053 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" Nov 02 17:41:42.764647 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.764607 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" event={"ID":"060e829c-0a72-48a9-b9db-13db919d875f","Type":"ContainerStarted","Data":"a068c66873b39a611762edeeb9ed2ffda2d3a620fca0303fd24466d3e5cb89b9"} Nov 02 17:41:42.764828 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.764673 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" event={"ID":"060e829c-0a72-48a9-b9db-13db919d875f","Type":"ContainerStarted","Data":"0cc307e641548f5fd1e4b0c1021166020b8876a3ca5dc01715416e382e134e16"} Nov 02 17:41:42.764828 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.764708 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" Nov 02 17:41:42.766340 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.766304 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" event={"ID":"3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0","Type":"ContainerStarted","Data":"75cf09299eed3f253d2b5e94a79365624903e38044b52689a4cf57ec4fc0368e"} Nov 02 17:41:42.766479 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.766371 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" event={"ID":"3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0","Type":"ContainerStarted","Data":"889e8bd5fb04401520c19787d4eeb7b8fd12cba426d2a9b2e8b3fdb672e8b85e"} Nov 02 17:41:42.766566 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.766549 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" Nov 02 17:41:42.816972 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.806549 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" podStartSLOduration=1.806532052 podStartE2EDuration="1.806532052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:42.784195847 +0000 UTC m=+474.572701376" watchObservedRunningTime="2026-11-02 17:41:42.806532052 +0000 UTC m=+474.595037582" Nov 02 17:41:42.816972 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.807265 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" podStartSLOduration=1.807250018 podStartE2EDuration="1.807250018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:42.804801044 +0000 UTC m=+474.593306572" watchObservedRunningTime="2026-11-02 17:41:42.807250018 +0000 UTC m=+474.595755563" Nov 02 17:41:42.816972 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.815573 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl"] Nov 02 17:41:42.835350 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:41:42.835319 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23593f20_8dc6_4ff6_93f4_d9b1fc859341.slice/crio-c2fa95881d1db0c0a69fe0fb2da77ea7fcc496941295709074d62adcfe05092e WatchSource:0}: Error finding container c2fa95881d1db0c0a69fe0fb2da77ea7fcc496941295709074d62adcfe05092e: Status 404 returned error can't find the container with id c2fa95881d1db0c0a69fe0fb2da77ea7fcc496941295709074d62adcfe05092e Nov 02 17:41:42.857524 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.857499 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8"] Nov 02 17:41:42.863114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:41:42.863084 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609d560e_9146_4fbf_9aeb_1ca65e5d3971.slice/crio-7b1b479c8b63dfce36fcf10c4a37f302f047c6bf6118f9e82e67e797b4d522aa WatchSource:0}: Error finding container 7b1b479c8b63dfce36fcf10c4a37f302f047c6bf6118f9e82e67e797b4d522aa: Status 404 returned error can't find the container with id 7b1b479c8b63dfce36fcf10c4a37f302f047c6bf6118f9e82e67e797b4d522aa Nov 02 17:41:42.908404 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.908375 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj"] Nov 02 17:41:42.916350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.916312 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 17:41:42.929630 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.926520 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-24cfb6-serving-cert\"" Nov 02 17:41:42.936218 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.936187 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj"] Nov 02 17:41:42.976310 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.976274 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-metrics-certs\") pod \"maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj\" (UID: \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 17:41:42.976486 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.976333 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-maas-api-tls\") pod \"maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj\" (UID: \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 17:41:42.976486 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.976443 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj\" (UID: \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 17:41:42.976582 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:42.976566 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z8v2\" (UniqueName: \"kubernetes.io/projected/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-kube-api-access-5z8v2\") pod \"maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj\" (UID: \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 17:41:43.013708 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.013664 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:43.077561 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.077522 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-metrics-certs\") pod \"maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj\" (UID: \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 17:41:43.077716 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.077568 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-maas-api-tls\") pod \"maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj\" (UID: \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 17:41:43.077716 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.077599 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj\" (UID: \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 17:41:43.077716 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.077647 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z8v2\" (UniqueName: \"kubernetes.io/projected/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-kube-api-access-5z8v2\") pod \"maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj\" (UID: \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 17:41:43.077817 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:41:43.077733 5549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-serving-cert: secret "maas-api-e2e-worker-w1-24cfb6-serving-cert" not found Nov 02 17:41:43.077817 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:41:43.077798 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-maas-api-tls podName:2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2 nodeName:}" failed. No retries permitted until 2026-11-02 17:41:43.577782321 +0000 UTC m=+475.366287827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-maas-api-tls") pod "maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" (UID: "2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2") : secret "maas-api-e2e-worker-w1-24cfb6-serving-cert" not found Nov 02 17:41:43.078334 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.078310 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj\" (UID: \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 17:41:43.080144 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.080119 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-metrics-certs\") pod \"maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj\" (UID: \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 17:41:43.086066 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.086016 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z8v2\" (UniqueName: \"kubernetes.io/projected/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-kube-api-access-5z8v2\") pod \"maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj\" (UID: \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 17:41:43.147191 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.147156 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:43.279134 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.279096 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:43.584163 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.584124 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-maas-api-tls\") pod \"maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj\" (UID: \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 17:41:43.586555 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.586525 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-maas-api-tls\") pod \"maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj\" (UID: \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 17:41:43.730746 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.730706 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:43.774130 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.774094 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" event={"ID":"609d560e-9146-4fbf-9aeb-1ca65e5d3971","Type":"ContainerStarted","Data":"047efcc3b279a6bb8b7ba49bba4bea599c56776e417d1abc5ac90de17358d6cc"} Nov 02 17:41:43.774315 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.774148 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" event={"ID":"609d560e-9146-4fbf-9aeb-1ca65e5d3971","Type":"ContainerStarted","Data":"7b1b479c8b63dfce36fcf10c4a37f302f047c6bf6118f9e82e67e797b4d522aa"} Nov 02 17:41:43.774315 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.774191 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" Nov 02 17:41:43.775796 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.775771 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" event={"ID":"23593f20-8dc6-4ff6-93f4-d9b1fc859341","Type":"ContainerStarted","Data":"5f289c32be11f7a00c9333de065ac38f7e617d07ce8b83ef8a784416e9ba1e09"} Nov 02 17:41:43.775920 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.775811 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" event={"ID":"23593f20-8dc6-4ff6-93f4-d9b1fc859341","Type":"ContainerStarted","Data":"c2fa95881d1db0c0a69fe0fb2da77ea7fcc496941295709074d62adcfe05092e"} Nov 02 17:41:43.775920 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.775907 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" Nov 02 17:41:43.791144 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.791101 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" podStartSLOduration=1.791089014 podStartE2EDuration="1.791089014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:43.791073391 +0000 UTC m=+475.579578920" watchObservedRunningTime="2026-11-02 17:41:43.791089014 +0000 UTC m=+475.579594542" Nov 02 17:41:43.814292 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.814243 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" podStartSLOduration=1.814234049 podStartE2EDuration="1.814234049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:43.813054776 +0000 UTC m=+475.601560308" watchObservedRunningTime="2026-11-02 17:41:43.814234049 +0000 UTC m=+475.602739577" Nov 02 17:41:43.851313 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:43.851239 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 17:41:44.126812 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:44.126783 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj"] Nov 02 17:41:44.128602 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:41:44.128573 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d54ac06_5cc0_40c5_a413_4ab95a5bdbe2.slice/crio-4465386e31b59fe7bf9068ce5e1e1bfd1b230ef612a3dd052517d33569e3559b WatchSource:0}: Error finding container 4465386e31b59fe7bf9068ce5e1e1bfd1b230ef612a3dd052517d33569e3559b: Status 404 returned error can't find the container with id 4465386e31b59fe7bf9068ce5e1e1bfd1b230ef612a3dd052517d33569e3559b Nov 02 17:41:44.639292 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:44.639258 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:44.782600 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:44.782514 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" event={"ID":"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2","Type":"ContainerStarted","Data":"43a6ad10bc23b1232d936448a48df8873e6f47dc326a713d69319564cd374350"} Nov 02 17:41:44.782600 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:44.782566 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" event={"ID":"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2","Type":"ContainerStarted","Data":"4465386e31b59fe7bf9068ce5e1e1bfd1b230ef612a3dd052517d33569e3559b"} Nov 02 17:41:44.784675 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:44.784650 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:44.811166 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:44.811107 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" podStartSLOduration=2.415940578 podStartE2EDuration="2.811093554s" totalImagesPullingTime="395.152976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:44.810474804 +0000 UTC m=+476.598980332" watchObservedRunningTime="2026-11-02 17:41:44.811093554 +0000 UTC m=+476.599599081" Nov 02 17:41:44.928638 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:44.928596 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:45.558793 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:45.558634 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:45.788552 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:45.788518 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 17:41:46.514728 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:46.514679 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:47.350632 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:47.350593 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:41:47.524187 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:47.524158 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:47.585714 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:41:47.585683 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:41:47.711895 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:47.711867 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" Nov 02 17:41:47.772818 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:47.772791 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 17:41:48.799622 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:48.799592 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:48.952687 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:48.952643 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:49.409509 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:49.409475 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:49.555178 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:49.555142 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:49.705548 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:49.705512 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:49.897003 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:49.887505 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:50.807567 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:50.807515 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:51.029181 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:51.029150 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:51.704492 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:51.704459 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:51.759717 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:51.759689 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" Nov 02 17:41:51.759903 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:51.759766 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" Nov 02 17:41:51.809662 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:51.809633 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 17:41:52.491868 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:52.491817 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:52.769434 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:52.769347 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" Nov 02 17:41:53.308847 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:53.308412 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:53.388987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:53.387096 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:41:53.716535 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:53.716493 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:53.778608 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:53.778580 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" Nov 02 17:41:53.778762 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:53.778654 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" Nov 02 17:41:53.936165 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:53.936136 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:54.086706 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:54.086527 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:54.239683 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:54.239640 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:54.786330 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:54.786240 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" Nov 02 17:41:54.786330 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:54.786317 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" Nov 02 17:41:54.811590 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:54.811556 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:55.822440 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:55.822391 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:56.644216 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:56.644183 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:56.804338 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:56.804217 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:57.481139 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:57.481109 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:57.854998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:57.854889 5549 generic.go:356] "Generic (PLEG): container finished" podID="9a22d1c3-76ca-445d-ab56-052577f4fcff" containerID="a52b88620fb74528f2f11b3f3b294a868725e09db95d8ec2a5e9c3a0e17de335" exitCode=0 Nov 02 17:41:57.854998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:57.854973 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" event={"ID":"9a22d1c3-76ca-445d-ab56-052577f4fcff","Type":"ContainerDied","Data":"a52b88620fb74528f2f11b3f3b294a868725e09db95d8ec2a5e9c3a0e17de335"} Nov 02 17:41:57.855401 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:57.855381 5549 scope.go:122] "RemoveContainer" containerID="a52b88620fb74528f2f11b3f3b294a868725e09db95d8ec2a5e9c3a0e17de335" Nov 02 17:41:57.856890 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:57.856854 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:57.857019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:57.856926 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:57.857019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:57.856976 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:57.920891 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:57.920848 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:58.835781 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:58.835584 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:58.862112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:58.862076 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" event={"ID":"9a22d1c3-76ca-445d-ab56-052577f4fcff","Type":"ContainerStarted","Data":"d8b83019bdd4a6b996b690c6c81a33e2ce0f869611bba896ccfc8ca5544c4143"} Nov 02 17:41:59.126638 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:59.126544 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:41:59.126804 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:59.126758 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 17:41:59.126851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:59.126797 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" podUID="9a22d1c3-76ca-445d-ab56-052577f4fcff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 17:41:59.523921 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:59.523882 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:59.865639 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:41:59.865544 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:00.034252 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:00.034208 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:00.126975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:00.126861 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 17:42:00.126975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:00.126913 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" podUID="9a22d1c3-76ca-445d-ab56-052577f4fcff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 17:42:00.208244 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:00.208200 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:00.351477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:00.351439 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:00.664901 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:00.664859 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:00.838621 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:00.838581 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:00.992110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:00.992074 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:01.127469 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:01.127433 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 17:42:01.127637 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:01.127499 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" podUID="9a22d1c3-76ca-445d-ab56-052577f4fcff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 17:42:01.163313 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:01.163270 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:01.315155 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:01.307334 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:01.805977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:01.805619 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:02.126639 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:02.126544 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 17:42:02.126639 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:02.126616 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" podUID="9a22d1c3-76ca-445d-ab56-052577f4fcff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 17:42:02.574431 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:02.574379 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:03.126107 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:03.126067 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 17:42:03.126294 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:03.126138 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" podUID="9a22d1c3-76ca-445d-ab56-052577f4fcff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 17:42:03.611976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:03.611237 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:04.130308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:04.130271 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:42:04.130574 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:04.130548 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:42:04.131171 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:04.131151 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 17:42:04.157784 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:04.157755 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:42:04.314763 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:04.314730 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:05.218672 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:05.218634 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:05.977906 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:05.976680 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:06.708506 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:06.708473 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:06.842992 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:06.842938 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:07.512366 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:07.512317 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:08.182908 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:08.182875 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:08.922111 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:08.922081 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:09.343487 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:09.343451 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:09.705206 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:09.705110 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:10.396565 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:10.396525 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:11.108212 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:11.108178 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:11.241222 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:11.241190 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:11.825595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:11.825555 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:12.635830 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:12.635782 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:12.783407 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:12.783376 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:13.329631 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:13.329590 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:13.888802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:13.888772 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:42:14.030374 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:14.030325 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:14.921940 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:14.921904 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:15.625329 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:15.625297 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:16.330325 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:16.330270 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:17.026264 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:17.026225 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:17.586140 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:42:17.586108 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:42:17.724021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:17.723979 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:18.420832 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.420796 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:18.509987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.509943 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7"] Nov 02 17:42:18.515110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.515086 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.524307 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.524276 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-66eed1\"/\"kube-root-ca.crt\"" Nov 02 17:42:18.524307 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.524300 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-66eed1\"/\"default-dockercfg-7nb7p\"" Nov 02 17:42:18.524733 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.524705 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-66eed1\"/\"premium-w2-66eed1-kserve-self-signed-certs\"" Nov 02 17:42:18.524868 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.524756 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-66eed1\"/\"openshift-service-ca.crt\"" Nov 02 17:42:18.526234 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.526211 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7"] Nov 02 17:42:18.594639 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.594610 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:18.602996 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.602948 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-model-cache\") pod \"premium-w2-66eed1-kserve-7864b56cf-4jmh7\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.603075 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.603054 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-dshm\") pod \"premium-w2-66eed1-kserve-7864b56cf-4jmh7\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.603118 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.603075 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66w5v\" (UniqueName: \"kubernetes.io/projected/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-kube-api-access-66w5v\") pod \"premium-w2-66eed1-kserve-7864b56cf-4jmh7\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.603118 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.603100 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-tmp-dir\") pod \"premium-w2-66eed1-kserve-7864b56cf-4jmh7\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.603182 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.603125 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-tls-certs\") pod \"premium-w2-66eed1-kserve-7864b56cf-4jmh7\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.603228 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.603209 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-home\") pod \"premium-w2-66eed1-kserve-7864b56cf-4jmh7\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.704120 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.704091 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-model-cache\") pod \"premium-w2-66eed1-kserve-7864b56cf-4jmh7\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.704302 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.704162 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-dshm\") pod \"premium-w2-66eed1-kserve-7864b56cf-4jmh7\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.704302 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.704180 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66w5v\" (UniqueName: \"kubernetes.io/projected/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-kube-api-access-66w5v\") pod \"premium-w2-66eed1-kserve-7864b56cf-4jmh7\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.704302 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.704199 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-tmp-dir\") pod \"premium-w2-66eed1-kserve-7864b56cf-4jmh7\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.704302 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.704227 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-tls-certs\") pod \"premium-w2-66eed1-kserve-7864b56cf-4jmh7\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.704302 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.704253 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-home\") pod \"premium-w2-66eed1-kserve-7864b56cf-4jmh7\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.704626 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.704584 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-model-cache\") pod \"premium-w2-66eed1-kserve-7864b56cf-4jmh7\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.704750 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.704688 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-home\") pod \"premium-w2-66eed1-kserve-7864b56cf-4jmh7\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.704750 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.704707 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-tmp-dir\") pod \"premium-w2-66eed1-kserve-7864b56cf-4jmh7\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.706351 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.706332 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-dshm\") pod \"premium-w2-66eed1-kserve-7864b56cf-4jmh7\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.706605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.706589 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-tls-certs\") pod \"premium-w2-66eed1-kserve-7864b56cf-4jmh7\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.713614 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.713589 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66w5v\" (UniqueName: \"kubernetes.io/projected/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-kube-api-access-66w5v\") pod \"premium-w2-66eed1-kserve-7864b56cf-4jmh7\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:18.828025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:18.827989 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:19.101382 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:19.101288 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7"] Nov 02 17:42:19.104491 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:42:19.104456 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6296c79a_bf7c_40ff_b2d2_37d3d9f46309.slice/crio-e85bb884cc28a310c48b83089b5f6d974071f4821fbb8692d225e60d4d96c2b9 WatchSource:0}: Error finding container e85bb884cc28a310c48b83089b5f6d974071f4821fbb8692d225e60d4d96c2b9: Status 404 returned error can't find the container with id e85bb884cc28a310c48b83089b5f6d974071f4821fbb8692d225e60d4d96c2b9 Nov 02 17:42:19.217969 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:19.217460 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:19.960105 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:19.959801 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:20.016447 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:20.016409 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" event={"ID":"6296c79a-bf7c-40ff-b2d2-37d3d9f46309","Type":"ContainerStarted","Data":"a409a9c8a19c7681b4119ebebf449ec21a4cec36702a80b17e61a5c6d573b6ee"} Nov 02 17:42:20.016447 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:20.016459 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" event={"ID":"6296c79a-bf7c-40ff-b2d2-37d3d9f46309","Type":"ContainerStarted","Data":"e85bb884cc28a310c48b83089b5f6d974071f4821fbb8692d225e60d4d96c2b9"} Nov 02 17:42:20.037025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:20.036941 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podStartSLOduration=2.036924861 podStartE2EDuration="2.036924861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:20.035988151 +0000 UTC m=+511.824493679" watchObservedRunningTime="2026-11-02 17:42:20.036924861 +0000 UTC m=+511.825430390" Nov 02 17:42:20.719050 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:20.719012 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:20.828921 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:20.828886 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:21.422323 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:21.422285 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:22.097663 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:22.096237 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:22.184624 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:22.184593 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:42:22.989589 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:22.989551 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:23.659378 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:23.659340 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:23.888200 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:23.888163 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7"] Nov 02 17:42:23.892854 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:23.892829 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:23.902983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:23.902517 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b97c6\"/\"premium-w0-4b97c6-kserve-self-signed-certs\"" Nov 02 17:42:23.902983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:23.902576 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b97c6\"/\"default-dockercfg-dct95\"" Nov 02 17:42:23.905182 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:23.905154 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7"] Nov 02 17:42:23.970675 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:23.970645 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-model-cache\") pod \"premium-w0-4b97c6-kserve-bd5875887-c5sm7\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:23.970821 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:23.970703 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdsl7\" (UniqueName: \"kubernetes.io/projected/3043cd28-bbab-4205-872c-2428bc46bbdc-kube-api-access-pdsl7\") pod \"premium-w0-4b97c6-kserve-bd5875887-c5sm7\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:23.970874 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:23.970854 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-home\") pod \"premium-w0-4b97c6-kserve-bd5875887-c5sm7\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:23.970911 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:23.970881 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-dshm\") pod \"premium-w0-4b97c6-kserve-bd5875887-c5sm7\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:23.970911 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:23.970904 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-tmp-dir\") pod \"premium-w0-4b97c6-kserve-bd5875887-c5sm7\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:23.971058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:23.971042 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3043cd28-bbab-4205-872c-2428bc46bbdc-tls-certs\") pod \"premium-w0-4b97c6-kserve-bd5875887-c5sm7\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:23.988635 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:23.988606 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4b97c6\"/\"kube-root-ca.crt\"" Nov 02 17:42:23.988789 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:23.988731 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4b97c6\"/\"openshift-service-ca.crt\"" Nov 02 17:42:24.003612 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:24.003581 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:24.071895 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:24.071864 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-home\") pod \"premium-w0-4b97c6-kserve-bd5875887-c5sm7\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:24.071895 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:24.071895 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-dshm\") pod \"premium-w0-4b97c6-kserve-bd5875887-c5sm7\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:24.072139 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:24.071916 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-tmp-dir\") pod \"premium-w0-4b97c6-kserve-bd5875887-c5sm7\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:24.072139 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:24.071941 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3043cd28-bbab-4205-872c-2428bc46bbdc-tls-certs\") pod \"premium-w0-4b97c6-kserve-bd5875887-c5sm7\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:24.072139 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:24.072011 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-model-cache\") pod \"premium-w0-4b97c6-kserve-bd5875887-c5sm7\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:24.072139 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:24.072067 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdsl7\" (UniqueName: \"kubernetes.io/projected/3043cd28-bbab-4205-872c-2428bc46bbdc-kube-api-access-pdsl7\") pod \"premium-w0-4b97c6-kserve-bd5875887-c5sm7\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:24.072381 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:24.072352 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-home\") pod \"premium-w0-4b97c6-kserve-bd5875887-c5sm7\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:24.072461 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:24.072414 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-tmp-dir\") pod \"premium-w0-4b97c6-kserve-bd5875887-c5sm7\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:24.072529 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:24.072513 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-model-cache\") pod \"premium-w0-4b97c6-kserve-bd5875887-c5sm7\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:24.074395 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:24.074368 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-dshm\") pod \"premium-w0-4b97c6-kserve-bd5875887-c5sm7\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:24.074495 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:24.074436 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3043cd28-bbab-4205-872c-2428bc46bbdc-tls-certs\") pod \"premium-w0-4b97c6-kserve-bd5875887-c5sm7\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:24.080225 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:24.080204 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdsl7\" (UniqueName: \"kubernetes.io/projected/3043cd28-bbab-4205-872c-2428bc46bbdc-kube-api-access-pdsl7\") pod \"premium-w0-4b97c6-kserve-bd5875887-c5sm7\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:24.206984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:24.206937 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:24.413466 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:24.412068 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:24.480566 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:24.480541 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7"] Nov 02 17:42:24.482560 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:42:24.482523 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3043cd28_bbab_4205_872c_2428bc46bbdc.slice/crio-5438695cad60f2b9492252d4d750437a253fe1ce17499e4883db4d23d773f497 WatchSource:0}: Error finding container 5438695cad60f2b9492252d4d750437a253fe1ce17499e4883db4d23d773f497: Status 404 returned error can't find the container with id 5438695cad60f2b9492252d4d750437a253fe1ce17499e4883db4d23d773f497 Nov 02 17:42:25.054343 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:25.054308 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" event={"ID":"3043cd28-bbab-4205-872c-2428bc46bbdc","Type":"ContainerStarted","Data":"78bc05cdbb5f1bacf6a2cc15e367c72e19d9b1300a59157a2cfc8d60fd699cc4"} Nov 02 17:42:25.054343 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:25.054349 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" event={"ID":"3043cd28-bbab-4205-872c-2428bc46bbdc","Type":"ContainerStarted","Data":"5438695cad60f2b9492252d4d750437a253fe1ce17499e4883db4d23d773f497"} Nov 02 17:42:25.072339 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:25.072292 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podStartSLOduration=2.072280574 podStartE2EDuration="2.072280574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:25.071172682 +0000 UTC m=+516.859678222" watchObservedRunningTime="2026-11-02 17:42:25.072280574 +0000 UTC m=+516.860786139" Nov 02 17:42:25.110197 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:25.110165 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:25.207069 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:25.207035 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:25.816020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:25.815979 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:26.499813 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:26.499731 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:27.132806 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:27.132766 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:27.560888 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:27.560860 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:27.891652 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:27.891562 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:28.813262 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:28.811239 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:28.828986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:28.828931 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:28.842002 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:28.841977 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:29.085559 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:29.085479 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:42:29.518326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:29.518274 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:30.330497 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:30.330443 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:30.966211 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:30.966179 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:31.192335 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:31.192300 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:31.905041 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:31.904774 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:32.816062 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:32.816032 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:33.608915 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:33.608865 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:34.207361 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:34.207329 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:34.219906 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:34.219882 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:34.420038 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:34.420001 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:34.743649 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:34.743613 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:42:35.118074 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:35.117995 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:42:35.225930 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:35.225895 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:36.124308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:36.124261 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:37.026012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:37.022705 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:37.871522 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:37.871487 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:38.868523 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:38.868487 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:41.187723 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:41.187685 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:41.726424 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:41.726384 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:42.528893 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:42.528857 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:43.222629 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:43.222594 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:44.324200 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:44.324149 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:45.312341 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.312259 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:45.470591 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.470556 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn"] Nov 02 17:42:45.474654 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.474638 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.478930 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.478901 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a37cf0-openshift-default-dockercfg-mbxtp\"" Nov 02 17:42:45.485338 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.485311 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn"] Nov 02 17:42:45.534823 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.534787 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trhj7\" (UniqueName: \"kubernetes.io/projected/dd458c81-9d15-45ba-92b1-473e1378c087-kube-api-access-trhj7\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.535019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.534853 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-istio-data\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.535019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.534936 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd458c81-9d15-45ba-92b1-473e1378c087-istiod-ca-cert\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.535019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.535002 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd458c81-9d15-45ba-92b1-473e1378c087-istio-podinfo\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.535160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.535030 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-credential-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.535201 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.535147 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd458c81-9d15-45ba-92b1-473e1378c087-istio-token\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.535201 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.535181 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-workload-certs\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.535274 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.535214 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-workload-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.535309 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.535276 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-istio-envoy\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.636238 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.636157 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-istio-data\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.636238 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.636201 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd458c81-9d15-45ba-92b1-473e1378c087-istiod-ca-cert\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.636238 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.636238 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd458c81-9d15-45ba-92b1-473e1378c087-istio-podinfo\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.636459 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.636266 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-credential-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.636459 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.636414 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd458c81-9d15-45ba-92b1-473e1378c087-istio-token\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.636566 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.636449 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-workload-certs\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.636566 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.636498 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-workload-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.636566 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.636557 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-istio-envoy\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.636704 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.636608 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trhj7\" (UniqueName: \"kubernetes.io/projected/dd458c81-9d15-45ba-92b1-473e1378c087-kube-api-access-trhj7\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.636704 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.636622 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-istio-data\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.636816 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.636739 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-credential-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.636816 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.636793 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-workload-certs\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.637007 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.636984 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-workload-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.637080 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.637065 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd458c81-9d15-45ba-92b1-473e1378c087-istiod-ca-cert\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.638727 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.638701 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-istio-envoy\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.638857 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.638835 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd458c81-9d15-45ba-92b1-473e1378c087-istio-podinfo\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.644096 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.644073 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd458c81-9d15-45ba-92b1-473e1378c087-istio-token\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.644442 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.644424 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trhj7\" (UniqueName: \"kubernetes.io/projected/dd458c81-9d15-45ba-92b1-473e1378c087-kube-api-access-trhj7\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:45.789274 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:45.789227 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:46.009974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:46.005451 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:46.077071 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:46.072474 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn"] Nov 02 17:42:46.078985 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:42:46.078932 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd458c81_9d15_45ba_92b1_473e1378c087.slice/crio-3d7132681b96eda9235a96a8bb1098d4c02e1ad631b3709d5fda1ce677dba7bd WatchSource:0}: Error finding container 3d7132681b96eda9235a96a8bb1098d4c02e1ad631b3709d5fda1ce677dba7bd: Status 404 returned error can't find the container with id 3d7132681b96eda9235a96a8bb1098d4c02e1ad631b3709d5fda1ce677dba7bd Nov 02 17:42:46.081285 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:46.081251 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:42:46.081368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:46.081318 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:42:46.081368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:46.081351 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:42:46.166886 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:46.166849 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" event={"ID":"dd458c81-9d15-45ba-92b1-473e1378c087","Type":"ContainerStarted","Data":"f24e0b9c2c90090fd6804fe972a7ee823ee9d5dc27d870a160a642dd495488df"} Nov 02 17:42:46.167055 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:46.166915 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" event={"ID":"dd458c81-9d15-45ba-92b1-473e1378c087","Type":"ContainerStarted","Data":"3d7132681b96eda9235a96a8bb1098d4c02e1ad631b3709d5fda1ce677dba7bd"} Nov 02 17:42:46.191976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:46.191908 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" podStartSLOduration=1.191894004 podStartE2EDuration="1.191894004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:46.189235475 +0000 UTC m=+537.977741015" watchObservedRunningTime="2026-11-02 17:42:46.191894004 +0000 UTC m=+537.980399531" Nov 02 17:42:46.706724 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:46.706486 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:46.799976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:46.799451 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:47.410018 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:47.405081 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:47.608617 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:42:47.608577 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:42:47.801978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:47.797749 5549 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 17:42:47.801978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:47.797813 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" podUID="dd458c81-9d15-45ba-92b1-473e1378c087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 17:42:48.027486 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:48.027442 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:48.798989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:48.796093 5549 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 17:42:48.798989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:48.796148 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" podUID="dd458c81-9d15-45ba-92b1-473e1378c087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 17:42:48.809315 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:48.807969 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:49.503980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:49.491514 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:49.804470 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:49.804372 5549 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 17:42:49.804470 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:49.804456 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" podUID="dd458c81-9d15-45ba-92b1-473e1378c087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 17:42:50.106839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:50.106749 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:50.795977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:50.793330 5549 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 17:42:50.795977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:50.793402 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" podUID="dd458c81-9d15-45ba-92b1-473e1378c087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 17:42:50.892977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:50.892618 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:51.527605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:51.527275 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:51.804123 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:51.797593 5549 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 17:42:51.804123 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:51.797666 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" podUID="dd458c81-9d15-45ba-92b1-473e1378c087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 17:42:52.241210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:52.241172 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:52.825756 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:52.825720 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:52.826247 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:52.826148 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:52.827689 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:52.827666 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 17:42:52.910134 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:52.910102 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:53.179926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:53.179840 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:42:53.640628 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:53.640590 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:54.418605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:54.416302 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:55.291147 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:55.291088 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:56.224359 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:56.224317 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:56.930933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:56.930875 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:57.618428 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:57.618390 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:57.762498 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:57.762465 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:58.305732 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:58.305691 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:58.987562 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:58.987392 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:59.624631 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:42:59.624589 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:00.296919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:00.296837 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:01.105970 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:01.105929 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:01.817286 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:01.817246 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:02.540348 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:02.540313 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:03.345187 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:03.345154 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:04.089367 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:04.089300 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:04.824042 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:04.824006 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:05.607543 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:05.607504 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:06.304872 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:06.304837 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:07.133436 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:07.133376 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:07.697517 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:07.697484 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:07.840129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:07.840098 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:08.641199 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:08.640189 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:09.357630 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:09.357590 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:10.252556 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:10.252517 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:11.035408 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:11.035373 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:11.811742 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:11.809291 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:12.608478 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:12.608418 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:13.336805 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:13.336763 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:14.014977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:14.014056 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:14.831005 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:14.830942 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:15.609588 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:15.609549 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:16.487978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:16.487089 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:17.317605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:17.316923 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:17.585661 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:43:17.585579 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:43:18.027512 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:18.027476 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:18.713575 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:18.713539 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:19.508790 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:19.508758 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:20.205576 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:20.205533 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:20.994336 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:20.994296 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:21.206071 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.206034 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:21.362745 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.362657 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:21.624573 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.624490 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j"] Nov 02 17:43:21.628655 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.628637 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" Nov 02 17:43:21.637153 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.636239 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3cb9c46838e9\"" Nov 02 17:43:21.637153 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.636402 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 17:43:21.637153 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.636473 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 17:43:21.637153 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.636552 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 17:43:21.637153 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.636610 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 17:43:21.637153 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.636666 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0c3a47e60ef3\"" Nov 02 17:43:21.637153 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.636714 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-4kxjc\"" Nov 02 17:43:21.638515 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.638494 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j"] Nov 02 17:43:21.642677 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.642653 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:21.702461 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.702430 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtmtk\" (UniqueName: \"kubernetes.io/projected/2183d380-e227-4c43-b8b2-1d6226f56f17-kube-api-access-wtmtk\") pod \"payload-processing-external-model-695c8bf8d5-lkd8j\" (UID: \"2183d380-e227-4c43-b8b2-1d6226f56f17\") " pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" Nov 02 17:43:21.702644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.702469 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2183d380-e227-4c43-b8b2-1d6226f56f17-plugins-config-volume\") pod \"payload-processing-external-model-695c8bf8d5-lkd8j\" (UID: \"2183d380-e227-4c43-b8b2-1d6226f56f17\") " pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" Nov 02 17:43:21.702644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.702504 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2183d380-e227-4c43-b8b2-1d6226f56f17-routing-overlay-volume\") pod \"payload-processing-external-model-695c8bf8d5-lkd8j\" (UID: \"2183d380-e227-4c43-b8b2-1d6226f56f17\") " pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" Nov 02 17:43:21.702644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.702592 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2183d380-e227-4c43-b8b2-1d6226f56f17-provider-credentials\") pod \"payload-processing-external-model-695c8bf8d5-lkd8j\" (UID: \"2183d380-e227-4c43-b8b2-1d6226f56f17\") " pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" Nov 02 17:43:21.822282 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.821730 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2183d380-e227-4c43-b8b2-1d6226f56f17-provider-credentials\") pod \"payload-processing-external-model-695c8bf8d5-lkd8j\" (UID: \"2183d380-e227-4c43-b8b2-1d6226f56f17\") " pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" Nov 02 17:43:21.822475 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.822423 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtmtk\" (UniqueName: \"kubernetes.io/projected/2183d380-e227-4c43-b8b2-1d6226f56f17-kube-api-access-wtmtk\") pod \"payload-processing-external-model-695c8bf8d5-lkd8j\" (UID: \"2183d380-e227-4c43-b8b2-1d6226f56f17\") " pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" Nov 02 17:43:21.822544 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.822476 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2183d380-e227-4c43-b8b2-1d6226f56f17-plugins-config-volume\") pod \"payload-processing-external-model-695c8bf8d5-lkd8j\" (UID: \"2183d380-e227-4c43-b8b2-1d6226f56f17\") " pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" Nov 02 17:43:21.822544 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.822534 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2183d380-e227-4c43-b8b2-1d6226f56f17-routing-overlay-volume\") pod \"payload-processing-external-model-695c8bf8d5-lkd8j\" (UID: \"2183d380-e227-4c43-b8b2-1d6226f56f17\") " pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" Nov 02 17:43:21.823426 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.823398 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2183d380-e227-4c43-b8b2-1d6226f56f17-routing-overlay-volume\") pod \"payload-processing-external-model-695c8bf8d5-lkd8j\" (UID: \"2183d380-e227-4c43-b8b2-1d6226f56f17\") " pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" Nov 02 17:43:21.823734 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.823707 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2183d380-e227-4c43-b8b2-1d6226f56f17-plugins-config-volume\") pod \"payload-processing-external-model-695c8bf8d5-lkd8j\" (UID: \"2183d380-e227-4c43-b8b2-1d6226f56f17\") " pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" Nov 02 17:43:21.827333 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.827305 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2183d380-e227-4c43-b8b2-1d6226f56f17-provider-credentials\") pod \"payload-processing-external-model-695c8bf8d5-lkd8j\" (UID: \"2183d380-e227-4c43-b8b2-1d6226f56f17\") " pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" Nov 02 17:43:21.831219 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.831191 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtmtk\" (UniqueName: \"kubernetes.io/projected/2183d380-e227-4c43-b8b2-1d6226f56f17-kube-api-access-wtmtk\") pod \"payload-processing-external-model-695c8bf8d5-lkd8j\" (UID: \"2183d380-e227-4c43-b8b2-1d6226f56f17\") " pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" Nov 02 17:43:21.943292 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:21.943193 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" Nov 02 17:43:22.019880 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:22.019703 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:22.226656 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:22.226617 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j"] Nov 02 17:43:22.229034 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:43:22.229008 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2183d380_e227_4c43_b8b2_1d6226f56f17.slice/crio-34120ff318c82a220b9481ca46f96293fb62de2e94fbf7be16561e7e622f020a WatchSource:0}: Error finding container 34120ff318c82a220b9481ca46f96293fb62de2e94fbf7be16561e7e622f020a: Status 404 returned error can't find the container with id 34120ff318c82a220b9481ca46f96293fb62de2e94fbf7be16561e7e622f020a Nov 02 17:43:22.391910 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:22.391877 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" event={"ID":"2183d380-e227-4c43-b8b2-1d6226f56f17","Type":"ContainerStarted","Data":"d18f8f4989976a321f2814e065ee026d2b14576b29af9b6f93b6dc16c965c23c"} Nov 02 17:43:22.392115 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:22.391933 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" event={"ID":"2183d380-e227-4c43-b8b2-1d6226f56f17","Type":"ContainerStarted","Data":"34120ff318c82a220b9481ca46f96293fb62de2e94fbf7be16561e7e622f020a"} Nov 02 17:43:22.392115 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:22.392093 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" Nov 02 17:43:22.411069 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:22.411015 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" podStartSLOduration=1.4110014309999999 podStartE2EDuration="1.411001431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:22.410049535 +0000 UTC m=+574.198555063" watchObservedRunningTime="2026-11-02 17:43:22.411001431 +0000 UTC m=+574.199506961" Nov 02 17:43:22.821832 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:22.821783 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:23.517655 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:23.517615 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:24.221773 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:24.221725 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:24.914149 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:24.912724 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:25.756719 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:25.756688 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:26.377997 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:26.377963 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:43:26.409973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:26.408934 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:27.286399 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:27.286365 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:28.126878 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:28.126837 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:29.022662 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:29.022583 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:29.818983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:29.818650 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:30.580985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:30.578760 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:31.305226 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:31.305196 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:31.442466 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:31.442430 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:32.263979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:32.263104 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:33.049294 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:33.049264 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:33.404776 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:33.404704 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" Nov 02 17:43:33.961468 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:33.961392 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:34.736862 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:34.733111 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:35.282526 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.282489 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh"] Nov 02 17:43:35.286413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.286397 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.297289 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.297262 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-66eed1-openshift-default-dockercfg-dd2rz\"" Nov 02 17:43:35.300038 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.300016 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh"] Nov 02 17:43:35.420216 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.420184 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.420394 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.420221 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.420394 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.420260 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.420394 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.420354 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmrd4\" (UniqueName: \"kubernetes.io/projected/57692113-df46-4797-9fd6-49b5aad6f5f0-kube-api-access-vmrd4\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.420504 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.420434 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.420504 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.420466 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57692113-df46-4797-9fd6-49b5aad6f5f0-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.420576 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.420541 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.420576 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.420572 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.420645 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.420590 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.517049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.517003 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:35.531736 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.531696 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.532585 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.532519 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmrd4\" (UniqueName: \"kubernetes.io/projected/57692113-df46-4797-9fd6-49b5aad6f5f0-kube-api-access-vmrd4\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.532833 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.532815 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.532938 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.532925 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57692113-df46-4797-9fd6-49b5aad6f5f0-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.533257 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.533236 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.540037 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.536246 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.540037 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.536287 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.540037 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.536349 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.540037 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.536380 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.540037 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.536802 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.540037 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.534329 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57692113-df46-4797-9fd6-49b5aad6f5f0-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.540037 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.539461 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.540037 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.532457 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.540037 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.539726 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.543111 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.543086 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.546357 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.546293 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.550686 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.550661 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmrd4\" (UniqueName: \"kubernetes.io/projected/57692113-df46-4797-9fd6-49b5aad6f5f0-kube-api-access-vmrd4\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.550785 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.550752 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.596060 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.596027 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h"] Nov 02 17:43:35.600470 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.600445 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:35.600818 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.600802 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.604613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.604589 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-4b97c6-openshift-default-dockercfg-smxvn\"" Nov 02 17:43:35.615013 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.614929 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h"] Nov 02 17:43:35.738795 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.738756 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.739199 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.738815 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.739199 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.738938 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.739199 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.739037 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.739199 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.739059 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.739199 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.739078 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.739199 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.739098 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.739199 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.739195 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqb78\" (UniqueName: \"kubernetes.io/projected/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-kube-api-access-cqb78\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.739470 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.739224 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.840757 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.840660 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.840757 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.840721 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.841010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.840779 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.841010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.840820 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.841010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.840859 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.841010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.840891 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.841010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.840911 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.841010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.840935 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqb78\" (UniqueName: \"kubernetes.io/projected/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-kube-api-access-cqb78\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.841369 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.841144 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.841369 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.841184 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.841369 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.841198 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.841369 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.841354 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.841570 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.841444 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.841718 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.841697 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.843181 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.843160 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.843584 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.843563 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.849893 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.849864 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.850071 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.850042 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqb78\" (UniqueName: \"kubernetes.io/projected/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-kube-api-access-cqb78\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:35.892102 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.892071 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh"] Nov 02 17:43:35.894670 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:43:35.894638 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57692113_df46_4797_9fd6_49b5aad6f5f0.slice/crio-d9aced77b8cdbb9d8b65c7d5508335cdee451195abf1e67cf34de20058683d22 WatchSource:0}: Error finding container d9aced77b8cdbb9d8b65c7d5508335cdee451195abf1e67cf34de20058683d22: Status 404 returned error can't find the container with id d9aced77b8cdbb9d8b65c7d5508335cdee451195abf1e67cf34de20058683d22 Nov 02 17:43:35.897158 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.897115 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:35.897229 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.897189 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:35.897229 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.897219 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:35.918663 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:35.918637 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:36.210079 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:36.210049 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h"] Nov 02 17:43:36.212562 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:43:36.212531 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfcfb5bd_0bbe_4347_9eaa_89bbef082fdb.slice/crio-87e71642c8479174be12305661727a5e36b1c3a239f8a9d528387942c91e46a9 WatchSource:0}: Error finding container 87e71642c8479174be12305661727a5e36b1c3a239f8a9d528387942c91e46a9: Status 404 returned error can't find the container with id 87e71642c8479174be12305661727a5e36b1c3a239f8a9d528387942c91e46a9 Nov 02 17:43:36.215134 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:36.215096 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:36.215241 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:36.215167 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:36.215241 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:36.215202 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:36.487146 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:36.486997 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" event={"ID":"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb","Type":"ContainerStarted","Data":"128bd85d288bc303244a07e0aff11a1b7106c6fd7a8a78fc42c6c5afdf521ada"} Nov 02 17:43:36.487146 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:36.487060 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" event={"ID":"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb","Type":"ContainerStarted","Data":"87e71642c8479174be12305661727a5e36b1c3a239f8a9d528387942c91e46a9"} Nov 02 17:43:36.488754 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:36.488724 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" event={"ID":"57692113-df46-4797-9fd6-49b5aad6f5f0","Type":"ContainerStarted","Data":"e859cf0e94e063033c940f8123c0d743d1479552a1d487f516994329efb799bb"} Nov 02 17:43:36.488922 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:36.488772 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" event={"ID":"57692113-df46-4797-9fd6-49b5aad6f5f0","Type":"ContainerStarted","Data":"d9aced77b8cdbb9d8b65c7d5508335cdee451195abf1e67cf34de20058683d22"} Nov 02 17:43:36.494735 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:36.494653 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:36.523555 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:36.523507 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" podStartSLOduration=1.523492547 podStartE2EDuration="1.523492547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:36.521907574 +0000 UTC m=+588.310413099" watchObservedRunningTime="2026-11-02 17:43:36.523492547 +0000 UTC m=+588.311998074" Nov 02 17:43:36.601181 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:36.601149 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:36.602570 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:36.602539 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:43:36.602701 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:36.602588 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" podUID="57692113-df46-4797-9fd6-49b5aad6f5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 17:43:36.926428 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:36.926034 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:37.260975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:37.257691 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:37.613131 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:37.603370 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:43:37.613131 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:37.603428 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" podUID="57692113-df46-4797-9fd6-49b5aad6f5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 17:43:37.933032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:37.932425 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:43:37.933032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:37.932484 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" podUID="bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 17:43:38.149977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:38.149710 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:38.602976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:38.602546 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:43:38.602976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:38.602604 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" podUID="57692113-df46-4797-9fd6-49b5aad6f5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 17:43:38.921033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:38.920257 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:43:38.921033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:38.920327 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" podUID="bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 17:43:39.220704 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:39.220666 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:39.605033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:39.604419 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:43:39.605033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:39.604498 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" podUID="57692113-df46-4797-9fd6-49b5aad6f5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 17:43:39.921035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:39.920344 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:43:39.921035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:39.920423 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" podUID="bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 17:43:39.939975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:39.934255 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:40.601977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:40.601479 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:43:40.601977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:40.601541 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" podUID="57692113-df46-4797-9fd6-49b5aad6f5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 17:43:40.721009 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:40.720632 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:40.921034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:40.920279 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:43:40.921034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:40.920342 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" podUID="bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 17:43:41.509317 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:41.509218 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:41.601977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:41.601400 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:43:41.601977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:41.601466 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" podUID="57692113-df46-4797-9fd6-49b5aad6f5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 17:43:41.924032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:41.923025 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:43:41.924032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:41.923088 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" podUID="bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 17:43:42.482970 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:42.482904 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:42.601776 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:42.601742 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:43:42.601970 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:42.601806 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" podUID="57692113-df46-4797-9fd6-49b5aad6f5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 17:43:42.919340 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:42.919244 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:43:42.919937 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:42.919897 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" podUID="bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 17:43:43.239976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:43.239877 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:44.037250 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:44.037209 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:44.127138 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:44.127077 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:44.127719 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:44.127681 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:44.128842 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:44.128798 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:43:44.153670 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:44.153622 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" podStartSLOduration=9.153608093 podStartE2EDuration="9.153608093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:36.5439073 +0000 UTC m=+588.332412829" watchObservedRunningTime="2026-11-02 17:43:44.153608093 +0000 UTC m=+595.942113620" Nov 02 17:43:44.222050 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:44.222020 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:44.222529 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:44.222506 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:44.223398 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:44.223378 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 17:43:44.948692 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:44.948655 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:45.825254 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:45.825200 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:46.540277 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:46.540216 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:47.272724 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:47.272686 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:43:47.329901 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:47.329860 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:47.588074 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:43:47.587976 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:43:48.161973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:48.153883 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:48.397792 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:48.397758 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 17:43:48.399278 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:48.399252 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 17:43:48.951631 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:48.951543 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:49.969678 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:49.969619 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:49.975079 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:49.975053 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-vbrfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:43:49.975236 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:49.975102 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" podUID="5639111b-8dcd-4b4a-a845-f21825bda0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:50.721913 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:50.721835 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:50.904386 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:50.904341 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:51.537076 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:51.537025 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:51.793700 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:51.793608 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs"] Nov 02 17:43:51.798669 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:51.798644 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:51.803308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:51.803274 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-24cfb6-openshift-default-dockercfg-r2v2f\"" Nov 02 17:43:51.809826 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:51.809804 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs"] Nov 02 17:43:51.980227 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:51.980189 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:51.980405 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:51.980235 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:51.980405 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:51.980327 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx4qs\" (UniqueName: \"kubernetes.io/projected/026e7c96-2366-483a-ae9c-55a92607394c-kube-api-access-tx4qs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:51.980494 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:51.980417 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:51.980494 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:51.980436 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026e7c96-2366-483a-ae9c-55a92607394c-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:51.980494 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:51.980455 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026e7c96-2366-483a-ae9c-55a92607394c-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:51.980610 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:51.980573 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:51.980610 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:51.980602 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026e7c96-2366-483a-ae9c-55a92607394c-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:51.980681 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:51.980633 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.082288 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.082205 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx4qs\" (UniqueName: \"kubernetes.io/projected/026e7c96-2366-483a-ae9c-55a92607394c-kube-api-access-tx4qs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.082446 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.082300 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.082446 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.082319 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026e7c96-2366-483a-ae9c-55a92607394c-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.082446 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.082336 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026e7c96-2366-483a-ae9c-55a92607394c-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.082446 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.082380 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.082446 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.082396 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026e7c96-2366-483a-ae9c-55a92607394c-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.082446 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.082436 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.082772 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.082493 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.082772 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.082527 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.082891 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.082787 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.082891 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.082850 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.083036 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.082893 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.083036 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.083005 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.083255 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.083232 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026e7c96-2366-483a-ae9c-55a92607394c-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.085052 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.085023 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026e7c96-2366-483a-ae9c-55a92607394c-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.085052 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.085035 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.093337 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.093312 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026e7c96-2366-483a-ae9c-55a92607394c-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.093419 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.093372 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx4qs\" (UniqueName: \"kubernetes.io/projected/026e7c96-2366-483a-ae9c-55a92607394c-kube-api-access-tx4qs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.112908 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.112887 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:52.241082 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.239416 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch"] Nov 02 17:43:52.244556 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.244529 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.256786 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.255600 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch"] Nov 02 17:43:52.386144 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.386062 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p94b\" (UniqueName: \"kubernetes.io/projected/b9b1010f-8c24-44d2-8495-f11e545f2187-kube-api-access-4p94b\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.386144 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.386104 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9b1010f-8c24-44d2-8495-f11e545f2187-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.386144 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.386132 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9b1010f-8c24-44d2-8495-f11e545f2187-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.386386 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.386206 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9b1010f-8c24-44d2-8495-f11e545f2187-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.386386 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.386289 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9b1010f-8c24-44d2-8495-f11e545f2187-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.386386 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.386346 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9b1010f-8c24-44d2-8495-f11e545f2187-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.386515 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.386398 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9b1010f-8c24-44d2-8495-f11e545f2187-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.386515 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.386422 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9b1010f-8c24-44d2-8495-f11e545f2187-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.386515 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.386455 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9b1010f-8c24-44d2-8495-f11e545f2187-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.425390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.425353 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs"] Nov 02 17:43:52.446316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.444837 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:52.446316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.444984 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:52.446316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.445031 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:52.458369 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.458323 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:52.487379 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.487347 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9b1010f-8c24-44d2-8495-f11e545f2187-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.487513 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.487399 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9b1010f-8c24-44d2-8495-f11e545f2187-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.487513 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.487441 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9b1010f-8c24-44d2-8495-f11e545f2187-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.487513 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.487499 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9b1010f-8c24-44d2-8495-f11e545f2187-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.487690 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.487563 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p94b\" (UniqueName: \"kubernetes.io/projected/b9b1010f-8c24-44d2-8495-f11e545f2187-kube-api-access-4p94b\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.487690 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.487603 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9b1010f-8c24-44d2-8495-f11e545f2187-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.487690 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.487676 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9b1010f-8c24-44d2-8495-f11e545f2187-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.487840 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.487676 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9b1010f-8c24-44d2-8495-f11e545f2187-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.487840 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.487722 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9b1010f-8c24-44d2-8495-f11e545f2187-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.487840 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.487780 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9b1010f-8c24-44d2-8495-f11e545f2187-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.488032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.487851 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9b1010f-8c24-44d2-8495-f11e545f2187-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.488032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.487922 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9b1010f-8c24-44d2-8495-f11e545f2187-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.488139 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.488066 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9b1010f-8c24-44d2-8495-f11e545f2187-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.488198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.488151 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9b1010f-8c24-44d2-8495-f11e545f2187-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.490056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.490035 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9b1010f-8c24-44d2-8495-f11e545f2187-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.490375 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.490356 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9b1010f-8c24-44d2-8495-f11e545f2187-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.496996 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.496975 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p94b\" (UniqueName: \"kubernetes.io/projected/b9b1010f-8c24-44d2-8495-f11e545f2187-kube-api-access-4p94b\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.497081 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.497048 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9b1010f-8c24-44d2-8495-f11e545f2187-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-fl8ch\" (UID: \"b9b1010f-8c24-44d2-8495-f11e545f2187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.568886 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.568849 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:52.633838 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.633679 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" event={"ID":"026e7c96-2366-483a-ae9c-55a92607394c","Type":"ContainerStarted","Data":"2529251fe28dadded5ae109c0c749b2e90f4fcd3606f44676f5dd1cbfe16422d"} Nov 02 17:43:52.633838 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.633740 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" event={"ID":"026e7c96-2366-483a-ae9c-55a92607394c","Type":"ContainerStarted","Data":"a392ad03dcc764a5a2bb5951088e5edf20ceedf3414655fb90b20e210c0aaeac"} Nov 02 17:43:52.661847 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.661588 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" podStartSLOduration=1.661571743 podStartE2EDuration="1.661571743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:52.660940995 +0000 UTC m=+604.449446527" watchObservedRunningTime="2026-11-02 17:43:52.661571743 +0000 UTC m=+604.450077262" Nov 02 17:43:52.853861 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.853831 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch"] Nov 02 17:43:52.855778 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:43:52.855746 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b1010f_8c24_44d2_8495_f11e545f2187.slice/crio-35a72b67ba062ca9efaa9d4299e7d70e5d3a76f107d56c22c5193c2aea7571bb WatchSource:0}: Error finding container 35a72b67ba062ca9efaa9d4299e7d70e5d3a76f107d56c22c5193c2aea7571bb: Status 404 returned error can't find the container with id 35a72b67ba062ca9efaa9d4299e7d70e5d3a76f107d56c22c5193c2aea7571bb Nov 02 17:43:52.858256 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.858224 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:52.858329 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.858288 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:52.858329 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.858317 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:52.903489 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:52.903466 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:53.113091 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:53.113060 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:43:53.114644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:53.114615 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:43:53.114764 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:53.114672 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" podUID="026e7c96-2366-483a-ae9c-55a92607394c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 17:43:53.322239 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:53.322206 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:53.641634 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:53.641542 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" event={"ID":"b9b1010f-8c24-44d2-8495-f11e545f2187","Type":"ContainerStarted","Data":"5fbd3218537fab780609d949c9e5c0f88b63e2cda022a56f4f7a50708fbed8db"} Nov 02 17:43:53.641634 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:53.641600 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" event={"ID":"b9b1010f-8c24-44d2-8495-f11e545f2187","Type":"ContainerStarted","Data":"35a72b67ba062ca9efaa9d4299e7d70e5d3a76f107d56c22c5193c2aea7571bb"} Nov 02 17:43:53.666854 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:53.666800 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" podStartSLOduration=1.666786887 podStartE2EDuration="1.666786887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:53.664640362 +0000 UTC m=+605.453145892" watchObservedRunningTime="2026-11-02 17:43:53.666786887 +0000 UTC m=+605.455292408" Nov 02 17:43:54.134771 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:54.134734 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:43:54.135090 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:54.135062 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" podUID="026e7c96-2366-483a-ae9c-55a92607394c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 17:43:54.138130 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:54.137109 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j"] Nov 02 17:43:54.203620 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:54.203579 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:54.569462 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:54.569422 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:43:54.570860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:54.570837 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-fl8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:43:54.570985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:54.570889 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" podUID="b9b1010f-8c24-44d2-8495-f11e545f2187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 17:43:55.096132 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:55.096090 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:55.125169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:55.125127 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:43:55.125446 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:55.125413 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" podUID="026e7c96-2366-483a-ae9c-55a92607394c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 17:43:55.569919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:55.569885 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-fl8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:43:55.570123 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:55.569977 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" podUID="b9b1010f-8c24-44d2-8495-f11e545f2187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 17:43:56.114310 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:56.114272 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:43:56.114938 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:56.114893 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" podUID="026e7c96-2366-483a-ae9c-55a92607394c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 17:43:56.202622 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:56.202554 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:56.569870 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:56.569391 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-fl8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:43:56.569870 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:56.569462 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" podUID="b9b1010f-8c24-44d2-8495-f11e545f2187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 17:43:57.114333 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:57.114288 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:43:57.115372 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:57.115334 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" podUID="026e7c96-2366-483a-ae9c-55a92607394c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 17:43:57.151087 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:57.151034 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:57.569335 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:57.569289 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-fl8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:43:57.569544 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:57.569364 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" podUID="b9b1010f-8c24-44d2-8495-f11e545f2187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 17:43:57.717905 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:57.717865 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:57.943623 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:57.943540 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:58.114327 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:58.114289 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:43:58.114634 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:58.114600 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" podUID="026e7c96-2366-483a-ae9c-55a92607394c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 17:43:58.569914 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:58.569880 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-fl8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:43:58.570121 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:58.569948 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" podUID="b9b1010f-8c24-44d2-8495-f11e545f2187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 17:43:58.733510 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:58.733470 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:58.881571 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:58.881480 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:59.113395 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.113361 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:43:59.113683 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.113642 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" podUID="026e7c96-2366-483a-ae9c-55a92607394c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 17:43:59.137932 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.137819 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" podUID="2183d380-e227-4c43-b8b2-1d6226f56f17" containerName="payload-processing" containerID="cri-o://d18f8f4989976a321f2814e065ee026d2b14576b29af9b6f93b6dc16c965c23c" gracePeriod=25 Nov 02 17:43:59.426692 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.426657 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" Nov 02 17:43:59.465835 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.465799 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2183d380-e227-4c43-b8b2-1d6226f56f17-kube-api-access-wtmtk\" (UniqueName: \"kubernetes.io/projected/2183d380-e227-4c43-b8b2-1d6226f56f17-kube-api-access-wtmtk\") pod \"2183d380-e227-4c43-b8b2-1d6226f56f17\" (UID: \"2183d380-e227-4c43-b8b2-1d6226f56f17\") " Nov 02 17:43:59.465835 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.465852 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2183d380-e227-4c43-b8b2-1d6226f56f17-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2183d380-e227-4c43-b8b2-1d6226f56f17-routing-overlay-volume\") pod \"2183d380-e227-4c43-b8b2-1d6226f56f17\" (UID: \"2183d380-e227-4c43-b8b2-1d6226f56f17\") " Nov 02 17:43:59.466117 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.465917 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2183d380-e227-4c43-b8b2-1d6226f56f17-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2183d380-e227-4c43-b8b2-1d6226f56f17-provider-credentials\") pod \"2183d380-e227-4c43-b8b2-1d6226f56f17\" (UID: \"2183d380-e227-4c43-b8b2-1d6226f56f17\") " Nov 02 17:43:59.466117 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.465980 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2183d380-e227-4c43-b8b2-1d6226f56f17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2183d380-e227-4c43-b8b2-1d6226f56f17-plugins-config-volume\") pod \"2183d380-e227-4c43-b8b2-1d6226f56f17\" (UID: \"2183d380-e227-4c43-b8b2-1d6226f56f17\") " Nov 02 17:43:59.466688 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.466644 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2183d380-e227-4c43-b8b2-1d6226f56f17-routing-overlay-volume" pod "2183d380-e227-4c43-b8b2-1d6226f56f17" (UID: "2183d380-e227-4c43-b8b2-1d6226f56f17"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:43:59.466814 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.466773 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2183d380-e227-4c43-b8b2-1d6226f56f17-plugins-config-volume" pod "2183d380-e227-4c43-b8b2-1d6226f56f17" (UID: "2183d380-e227-4c43-b8b2-1d6226f56f17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:43:59.468600 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.468566 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2183d380-e227-4c43-b8b2-1d6226f56f17-kube-api-access-wtmtk" pod "2183d380-e227-4c43-b8b2-1d6226f56f17" (UID: "2183d380-e227-4c43-b8b2-1d6226f56f17"). InnerVolumeSpecName "kube-api-access-wtmtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:59.468600 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.468579 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2183d380-e227-4c43-b8b2-1d6226f56f17-provider-credentials" pod "2183d380-e227-4c43-b8b2-1d6226f56f17" (UID: "2183d380-e227-4c43-b8b2-1d6226f56f17"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:59.567805 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.567767 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtmtk\" (UniqueName: \"kubernetes.io/projected/2183d380-e227-4c43-b8b2-1d6226f56f17-kube-api-access-wtmtk\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:43:59.567805 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.567801 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2183d380-e227-4c43-b8b2-1d6226f56f17-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:43:59.568056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.567815 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2183d380-e227-4c43-b8b2-1d6226f56f17-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:43:59.568056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.567827 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2183d380-e227-4c43-b8b2-1d6226f56f17-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:43:59.569641 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.569611 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-fl8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:43:59.569739 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.569676 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" podUID="b9b1010f-8c24-44d2-8495-f11e545f2187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 17:43:59.678403 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.678306 5549 generic.go:356] "Generic (PLEG): container finished" podID="2183d380-e227-4c43-b8b2-1d6226f56f17" containerID="d18f8f4989976a321f2814e065ee026d2b14576b29af9b6f93b6dc16c965c23c" exitCode=0 Nov 02 17:43:59.678403 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.678376 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" event={"ID":"2183d380-e227-4c43-b8b2-1d6226f56f17","Type":"ContainerDied","Data":"d18f8f4989976a321f2814e065ee026d2b14576b29af9b6f93b6dc16c965c23c"} Nov 02 17:43:59.678629 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.678441 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" Nov 02 17:43:59.678629 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.678457 5549 scope.go:122] "RemoveContainer" containerID="d18f8f4989976a321f2814e065ee026d2b14576b29af9b6f93b6dc16c965c23c" Nov 02 17:43:59.678629 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.678445 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j" event={"ID":"2183d380-e227-4c43-b8b2-1d6226f56f17","Type":"ContainerDied","Data":"34120ff318c82a220b9481ca46f96293fb62de2e94fbf7be16561e7e622f020a"} Nov 02 17:43:59.692521 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.692493 5549 scope.go:122] "RemoveContainer" containerID="d18f8f4989976a321f2814e065ee026d2b14576b29af9b6f93b6dc16c965c23c" Nov 02 17:43:59.692921 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:43:59.692898 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d18f8f4989976a321f2814e065ee026d2b14576b29af9b6f93b6dc16c965c23c\": container with ID starting with d18f8f4989976a321f2814e065ee026d2b14576b29af9b6f93b6dc16c965c23c not found: ID does not exist" containerID="d18f8f4989976a321f2814e065ee026d2b14576b29af9b6f93b6dc16c965c23c" Nov 02 17:43:59.693020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.692934 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18f8f4989976a321f2814e065ee026d2b14576b29af9b6f93b6dc16c965c23c"} err="failed to get container status \"d18f8f4989976a321f2814e065ee026d2b14576b29af9b6f93b6dc16c965c23c\": rpc error: code = NotFound desc = could not find container \"d18f8f4989976a321f2814e065ee026d2b14576b29af9b6f93b6dc16c965c23c\": container with ID starting with d18f8f4989976a321f2814e065ee026d2b14576b29af9b6f93b6dc16c965c23c not found: ID does not exist" Nov 02 17:43:59.708588 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.708549 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j"] Nov 02 17:43:59.713117 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.713082 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-695c8bf8d5-lkd8j"] Nov 02 17:43:59.753643 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:43:59.753532 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:00.118370 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.118339 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:44:00.118809 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.118790 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:44:00.119674 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.119646 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 17:44:00.330824 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.330790 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2183d380-e227-4c43-b8b2-1d6226f56f17" path="/var/lib/kubelet/pods/2183d380-e227-4c43-b8b2-1d6226f56f17/volumes" Nov 02 17:44:00.570116 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.570075 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-fl8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:44:00.570274 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.570142 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" podUID="b9b1010f-8c24-44d2-8495-f11e545f2187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 17:44:00.620866 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.620828 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns"] Nov 02 17:44:00.621393 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.621374 5549 cpu_manager.go:414] "containerMap: removing container" podUID="2183d380-e227-4c43-b8b2-1d6226f56f17" containerName="payload-processing" Nov 02 17:44:00.621393 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.621392 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2183d380-e227-4c43-b8b2-1d6226f56f17" containerName="payload-processing" Nov 02 17:44:00.621526 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.621495 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="2183d380-e227-4c43-b8b2-1d6226f56f17" containerName="payload-processing" Nov 02 17:44:00.626351 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.626329 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.630073 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.630032 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:00.634144 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.634121 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-66eed1\"/\"distinct-w2-66eed1-kserve-self-signed-certs\"" Nov 02 17:44:00.634550 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.634529 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns"] Nov 02 17:44:00.702114 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.702073 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64a10854-4c5f-446a-ac7e-8d12767fa1bf-tls-certs\") pod \"distinct-w2-66eed1-kserve-bcf9f794-7pvns\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.702272 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.702122 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-tmp-dir\") pod \"distinct-w2-66eed1-kserve-bcf9f794-7pvns\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.702272 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.702207 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-home\") pod \"distinct-w2-66eed1-kserve-bcf9f794-7pvns\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.702372 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.702311 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-dshm\") pod \"distinct-w2-66eed1-kserve-bcf9f794-7pvns\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.702372 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.702353 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-model-cache\") pod \"distinct-w2-66eed1-kserve-bcf9f794-7pvns\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.702446 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.702435 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzzpj\" (UniqueName: \"kubernetes.io/projected/64a10854-4c5f-446a-ac7e-8d12767fa1bf-kube-api-access-jzzpj\") pod \"distinct-w2-66eed1-kserve-bcf9f794-7pvns\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.784981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.784926 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:00.803804 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.803761 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzzpj\" (UniqueName: \"kubernetes.io/projected/64a10854-4c5f-446a-ac7e-8d12767fa1bf-kube-api-access-jzzpj\") pod \"distinct-w2-66eed1-kserve-bcf9f794-7pvns\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.804030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.803848 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64a10854-4c5f-446a-ac7e-8d12767fa1bf-tls-certs\") pod \"distinct-w2-66eed1-kserve-bcf9f794-7pvns\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.804030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.803899 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-tmp-dir\") pod \"distinct-w2-66eed1-kserve-bcf9f794-7pvns\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.804030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.803938 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-home\") pod \"distinct-w2-66eed1-kserve-bcf9f794-7pvns\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.804030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.804019 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-dshm\") pod \"distinct-w2-66eed1-kserve-bcf9f794-7pvns\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.804255 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.804057 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-model-cache\") pod \"distinct-w2-66eed1-kserve-bcf9f794-7pvns\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.804825 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.804800 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-home\") pod \"distinct-w2-66eed1-kserve-bcf9f794-7pvns\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.805274 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.805245 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-model-cache\") pod \"distinct-w2-66eed1-kserve-bcf9f794-7pvns\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.805529 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.805507 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-tmp-dir\") pod \"distinct-w2-66eed1-kserve-bcf9f794-7pvns\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.807271 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.807249 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-dshm\") pod \"distinct-w2-66eed1-kserve-bcf9f794-7pvns\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.807557 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.807536 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64a10854-4c5f-446a-ac7e-8d12767fa1bf-tls-certs\") pod \"distinct-w2-66eed1-kserve-bcf9f794-7pvns\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.815456 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.815420 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzzpj\" (UniqueName: \"kubernetes.io/projected/64a10854-4c5f-446a-ac7e-8d12767fa1bf-kube-api-access-jzzpj\") pod \"distinct-w2-66eed1-kserve-bcf9f794-7pvns\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:00.941326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:00.941230 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:01.231298 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:01.231268 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns"] Nov 02 17:44:01.234931 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:44:01.234895 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64a10854_4c5f_446a_ac7e_8d12767fa1bf.slice/crio-40a13bc9153566dd11bff08086171660162688b6d05b5c5cce4ee300e0dacd5a WatchSource:0}: Error finding container 40a13bc9153566dd11bff08086171660162688b6d05b5c5cce4ee300e0dacd5a: Status 404 returned error can't find the container with id 40a13bc9153566dd11bff08086171660162688b6d05b5c5cce4ee300e0dacd5a Nov 02 17:44:01.419611 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:01.419516 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:01.696775 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:01.696733 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" event={"ID":"64a10854-4c5f-446a-ac7e-8d12767fa1bf","Type":"ContainerStarted","Data":"ba04c73f502d01fac0691e95a04d6d265d98842eda58f7f31d36fa1094dac21e"} Nov 02 17:44:01.697003 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:01.696789 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" event={"ID":"64a10854-4c5f-446a-ac7e-8d12767fa1bf","Type":"ContainerStarted","Data":"40a13bc9153566dd11bff08086171660162688b6d05b5c5cce4ee300e0dacd5a"} Nov 02 17:44:01.717281 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:01.717232 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podStartSLOduration=1.7172171600000001 podStartE2EDuration="1.71721716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:01.716444452 +0000 UTC m=+613.504949982" watchObservedRunningTime="2026-11-02 17:44:01.71721716 +0000 UTC m=+613.505722687" Nov 02 17:44:01.942452 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:01.942409 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:01.966767 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:01.966690 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:44:01.967014 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:01.966994 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:44:01.968140 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:01.968114 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" Nov 02 17:44:02.164758 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:02.164717 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:03.010704 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:03.010186 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:03.808427 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:03.806937 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:04.515512 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:04.515473 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:05.428983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:05.428033 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:06.192977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:06.189546 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:07.051086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:07.051042 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:07.468283 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:07.468248 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:44:07.840027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:07.837970 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:08.761669 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:08.761453 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:09.877999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:09.872744 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:10.493469 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:10.493432 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:10.949982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:10.946305 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:10.971918 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:10.971888 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:11.692829 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:11.692737 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:11.834356 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:11.834323 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:44:12.522647 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:12.522601 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:13.225006 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:13.223242 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:14.061679 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:14.061621 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:14.841450 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:14.841371 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:15.563430 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:15.563390 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:16.433974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:16.425838 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:17.317374 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:17.317308 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:17.586803 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:44:17.586717 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:44:18.047804 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:18.047770 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:18.745655 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:18.745617 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:19.409546 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:19.409498 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:20.166976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.162076 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:20.341910 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.341875 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:20.685932 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.685894 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8"] Nov 02 17:44:20.692019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.691989 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.701045 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.701016 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8"] Nov 02 17:44:20.801249 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.801211 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d4c449-0024-4b94-91c0-41b4cb484bab-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.801422 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.801329 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.801422 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.801356 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.801505 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.801422 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwhm7\" (UniqueName: \"kubernetes.io/projected/34d4c449-0024-4b94-91c0-41b4cb484bab-kube-api-access-jwhm7\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.801505 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.801444 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.801505 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.801461 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.801601 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.801547 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.801601 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.801590 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.801701 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.801687 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.909539 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.909486 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwhm7\" (UniqueName: \"kubernetes.io/projected/34d4c449-0024-4b94-91c0-41b4cb484bab-kube-api-access-jwhm7\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.913284 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.913251 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.915170 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.915143 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.915525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.915493 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.915886 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.915870 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.916922 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.916897 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.917509 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.917305 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d4c449-0024-4b94-91c0-41b4cb484bab-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.919084 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.918897 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.919793 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.919771 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:20.943406 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.941688 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwhm7\" (UniqueName: \"kubernetes.io/projected/34d4c449-0024-4b94-91c0-41b4cb484bab-kube-api-access-jwhm7\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.944264 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.944234 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d4c449-0024-4b94-91c0-41b4cb484bab-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.960965 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.944661 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.960965 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.946596 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.960965 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.946843 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.960965 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.947033 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.960965 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.954184 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.960965 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.954659 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.960965 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.955233 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:20.961673 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:20.961634 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:21.011499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:21.011455 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:21.293359 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:21.293327 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8"] Nov 02 17:44:21.295013 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:44:21.294986 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d4c449_0024_4b94_91c0_41b4cb484bab.slice/crio-20de5eb9733bab19a5e92c6fb3878551cc9c2e46a046eb0acfd7eb33bc135591 WatchSource:0}: Error finding container 20de5eb9733bab19a5e92c6fb3878551cc9c2e46a046eb0acfd7eb33bc135591: Status 404 returned error can't find the container with id 20de5eb9733bab19a5e92c6fb3878551cc9c2e46a046eb0acfd7eb33bc135591 Nov 02 17:44:21.297380 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:21.297348 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:21.297461 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:21.297414 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:21.297461 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:21.297444 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:21.629024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:21.627233 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:21.890081 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:21.889990 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" event={"ID":"34d4c449-0024-4b94-91c0-41b4cb484bab","Type":"ContainerStarted","Data":"296a9df9680c75e30a22ab2e477775be89e3596d995e40a01858e4ceb031246c"} Nov 02 17:44:21.890081 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:21.890042 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" event={"ID":"34d4c449-0024-4b94-91c0-41b4cb484bab","Type":"ContainerStarted","Data":"20de5eb9733bab19a5e92c6fb3878551cc9c2e46a046eb0acfd7eb33bc135591"} Nov 02 17:44:21.913769 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:21.913707 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" podStartSLOduration=1.913689183 podStartE2EDuration="1.913689183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:21.910887013 +0000 UTC m=+633.699392541" watchObservedRunningTime="2026-11-02 17:44:21.913689183 +0000 UTC m=+633.702194712" Nov 02 17:44:22.014098 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:22.014052 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:44:22.014324 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:22.014119 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" podUID="34d4c449-0024-4b94-91c0-41b4cb484bab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:44:22.014536 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:22.014397 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:22.370974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:22.368409 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:23.005909 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:23.005874 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:23.012228 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:23.012202 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:44:23.012374 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:23.012255 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" podUID="34d4c449-0024-4b94-91c0-41b4cb484bab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:44:23.228590 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:23.228551 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:44:23.339060 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:23.336858 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:24.012113 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:24.012075 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:24.032013 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:24.025354 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:44:24.032013 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:24.025421 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" podUID="34d4c449-0024-4b94-91c0-41b4cb484bab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:44:24.604720 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:24.604592 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:24.825437 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:24.825398 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:25.012346 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:25.012313 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:44:25.012819 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:25.012385 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" podUID="34d4c449-0024-4b94-91c0-41b4cb484bab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:44:25.454053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:25.454019 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:25.606978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:25.599638 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:25.786164 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:25.786070 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:26.012776 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.012730 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:44:26.013235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.012831 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" podUID="34d4c449-0024-4b94-91c0-41b4cb484bab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:44:26.122217 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.122136 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr"] Nov 02 17:44:26.127566 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.127541 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" Nov 02 17:44:26.136910 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.136877 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 17:44:26.137081 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.136989 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2fd63f0aef48\"" Nov 02 17:44:26.137081 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.137022 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0c3a47e60ef3\"" Nov 02 17:44:26.137081 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.137045 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-4kxjc\"" Nov 02 17:44:26.137081 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.137075 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 17:44:26.137265 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.137107 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 17:44:26.137265 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.137202 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f873830d518c\"" Nov 02 17:44:26.137265 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.137218 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 17:44:26.139615 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.139592 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr"] Nov 02 17:44:26.262258 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.262215 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754803aa-c60f-4e58-a75e-2ace7db46790-plugins-config-volume\") pod \"payload-processing-external-model-6699fcb7fd-jl6qr\" (UID: \"754803aa-c60f-4e58-a75e-2ace7db46790\") " pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" Nov 02 17:44:26.262455 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.262283 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wfmk\" (UniqueName: \"kubernetes.io/projected/754803aa-c60f-4e58-a75e-2ace7db46790-kube-api-access-4wfmk\") pod \"payload-processing-external-model-6699fcb7fd-jl6qr\" (UID: \"754803aa-c60f-4e58-a75e-2ace7db46790\") " pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" Nov 02 17:44:26.262455 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.262386 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/754803aa-c60f-4e58-a75e-2ace7db46790-provider-credentials\") pod \"payload-processing-external-model-6699fcb7fd-jl6qr\" (UID: \"754803aa-c60f-4e58-a75e-2ace7db46790\") " pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" Nov 02 17:44:26.262575 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.262560 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/754803aa-c60f-4e58-a75e-2ace7db46790-routing-overlay-volume\") pod \"payload-processing-external-model-6699fcb7fd-jl6qr\" (UID: \"754803aa-c60f-4e58-a75e-2ace7db46790\") " pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" Nov 02 17:44:26.363572 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.363531 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/754803aa-c60f-4e58-a75e-2ace7db46790-routing-overlay-volume\") pod \"payload-processing-external-model-6699fcb7fd-jl6qr\" (UID: \"754803aa-c60f-4e58-a75e-2ace7db46790\") " pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" Nov 02 17:44:26.363729 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.363593 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754803aa-c60f-4e58-a75e-2ace7db46790-plugins-config-volume\") pod \"payload-processing-external-model-6699fcb7fd-jl6qr\" (UID: \"754803aa-c60f-4e58-a75e-2ace7db46790\") " pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" Nov 02 17:44:26.363729 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.363628 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wfmk\" (UniqueName: \"kubernetes.io/projected/754803aa-c60f-4e58-a75e-2ace7db46790-kube-api-access-4wfmk\") pod \"payload-processing-external-model-6699fcb7fd-jl6qr\" (UID: \"754803aa-c60f-4e58-a75e-2ace7db46790\") " pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" Nov 02 17:44:26.363729 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.363662 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/754803aa-c60f-4e58-a75e-2ace7db46790-provider-credentials\") pod \"payload-processing-external-model-6699fcb7fd-jl6qr\" (UID: \"754803aa-c60f-4e58-a75e-2ace7db46790\") " pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" Nov 02 17:44:26.364330 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.364306 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/754803aa-c60f-4e58-a75e-2ace7db46790-routing-overlay-volume\") pod \"payload-processing-external-model-6699fcb7fd-jl6qr\" (UID: \"754803aa-c60f-4e58-a75e-2ace7db46790\") " pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" Nov 02 17:44:26.364438 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.364391 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754803aa-c60f-4e58-a75e-2ace7db46790-plugins-config-volume\") pod \"payload-processing-external-model-6699fcb7fd-jl6qr\" (UID: \"754803aa-c60f-4e58-a75e-2ace7db46790\") " pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" Nov 02 17:44:26.366466 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.366438 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/754803aa-c60f-4e58-a75e-2ace7db46790-provider-credentials\") pod \"payload-processing-external-model-6699fcb7fd-jl6qr\" (UID: \"754803aa-c60f-4e58-a75e-2ace7db46790\") " pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" Nov 02 17:44:26.373207 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.373138 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wfmk\" (UniqueName: \"kubernetes.io/projected/754803aa-c60f-4e58-a75e-2ace7db46790-kube-api-access-4wfmk\") pod \"payload-processing-external-model-6699fcb7fd-jl6qr\" (UID: \"754803aa-c60f-4e58-a75e-2ace7db46790\") " pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" Nov 02 17:44:26.443975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.443915 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" Nov 02 17:44:26.710081 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.709936 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:26.776063 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.775663 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr"] Nov 02 17:44:26.787989 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:44:26.779019 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754803aa_c60f_4e58_a75e_2ace7db46790.slice/crio-6c7efbc981623b729030637f24a720e6bb11c7f90a5fc9e91d8e408b814158fe WatchSource:0}: Error finding container 6c7efbc981623b729030637f24a720e6bb11c7f90a5fc9e91d8e408b814158fe: Status 404 returned error can't find the container with id 6c7efbc981623b729030637f24a720e6bb11c7f90a5fc9e91d8e408b814158fe Nov 02 17:44:26.943451 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.943413 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" event={"ID":"754803aa-c60f-4e58-a75e-2ace7db46790","Type":"ContainerStarted","Data":"70a04227ebc1c5255e69bf712ef916f3b7ae74f7a307121c37131a5c9128d9d2"} Nov 02 17:44:26.943640 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.943474 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" event={"ID":"754803aa-c60f-4e58-a75e-2ace7db46790","Type":"ContainerStarted","Data":"6c7efbc981623b729030637f24a720e6bb11c7f90a5fc9e91d8e408b814158fe"} Nov 02 17:44:26.943688 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.943620 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" Nov 02 17:44:26.964366 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:26.964260 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" podStartSLOduration=0.964245345 podStartE2EDuration="964.245345ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:26.961310819 +0000 UTC m=+638.749816347" watchObservedRunningTime="2026-11-02 17:44:26.964245345 +0000 UTC m=+638.752750874" Nov 02 17:44:27.012254 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:27.012217 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:44:27.012432 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:27.012279 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" podUID="34d4c449-0024-4b94-91c0-41b4cb484bab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:44:27.741888 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:27.741832 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:28.012538 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:28.012451 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:44:28.012538 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:28.012508 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" podUID="34d4c449-0024-4b94-91c0-41b4cb484bab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:44:28.428281 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:28.428192 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:29.016984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:29.016918 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:29.017438 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:29.017412 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:29.018664 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:29.018638 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:44:29.124303 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:29.124244 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:29.932148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:29.932107 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:30.639708 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:30.639607 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:31.462096 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:31.462050 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:32.258614 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:32.258536 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:32.614778 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:32.614699 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:33.173987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:33.173933 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:34.257719 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:34.257677 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:35.402060 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.402007 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k"] Nov 02 17:44:35.436977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.422261 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k"] Nov 02 17:44:35.436977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.422387 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.773156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.773117 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.773401 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.773379 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7kj5\" (UniqueName: \"kubernetes.io/projected/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-kube-api-access-q7kj5\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.773541 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.773521 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.773701 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.773684 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.773831 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.773815 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.774030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.774010 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.774307 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.774289 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.774440 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.774423 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.774578 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.774561 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.846601 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.846412 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:35.876903 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.876837 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.877346 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.877238 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.877632 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.877556 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.877824 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.877793 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.877984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.877924 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.878168 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.878137 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.878407 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.878343 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.878407 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.878376 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7kj5\" (UniqueName: \"kubernetes.io/projected/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-kube-api-access-q7kj5\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.878633 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.878576 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.879335 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.879317 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.880444 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.880426 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.880820 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.880802 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.881899 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.881879 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.883008 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.882926 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.941973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.926354 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.941973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.928494 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:35.958151 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:35.958104 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:36.014838 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:36.014696 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7kj5\" (UniqueName: \"kubernetes.io/projected/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-kube-api-access-q7kj5\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:36.079695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:36.079648 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:36.861187 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:36.861105 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k"] Nov 02 17:44:36.871162 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:44:36.871140 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e748d4a_ed12_40fc_bbd0_7a85ef51b0b5.slice/crio-f8972be8c22c4fbfc1ed2c9bb33c93325357b79b385daf18195e0fdc59bc3fcb WatchSource:0}: Error finding container f8972be8c22c4fbfc1ed2c9bb33c93325357b79b385daf18195e0fdc59bc3fcb: Status 404 returned error can't find the container with id f8972be8c22c4fbfc1ed2c9bb33c93325357b79b385daf18195e0fdc59bc3fcb Nov 02 17:44:36.874029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:36.873989 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:36.874121 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:36.874080 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:36.874168 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:36.874119 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:36.978107 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:36.948104 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd"] Nov 02 17:44:36.978107 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:36.954292 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:36.978107 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:36.968670 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd"] Nov 02 17:44:37.134929 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.134840 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af43d18-1d68-45cb-839f-bc220435b550-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.135350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.135323 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2pk5\" (UniqueName: \"kubernetes.io/projected/7af43d18-1d68-45cb-839f-bc220435b550-kube-api-access-z2pk5\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.135520 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.135501 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af43d18-1d68-45cb-839f-bc220435b550-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.135682 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.135664 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.135882 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.135812 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af43d18-1d68-45cb-839f-bc220435b550-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.136078 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.136059 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.136211 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.136196 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.136362 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.136342 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.136545 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.136527 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.141426 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.141382 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" event={"ID":"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5","Type":"ContainerStarted","Data":"f8972be8c22c4fbfc1ed2c9bb33c93325357b79b385daf18195e0fdc59bc3fcb"} Nov 02 17:44:37.231876 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.231833 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:37.240636 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.240592 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2pk5\" (UniqueName: \"kubernetes.io/projected/7af43d18-1d68-45cb-839f-bc220435b550-kube-api-access-z2pk5\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.240808 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.240656 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af43d18-1d68-45cb-839f-bc220435b550-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.240808 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.240702 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.240808 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.240761 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af43d18-1d68-45cb-839f-bc220435b550-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.241010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.240830 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.241010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.240870 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.241010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.240910 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.241180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.241027 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.241180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.241066 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af43d18-1d68-45cb-839f-bc220435b550-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.278067 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.257615 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af43d18-1d68-45cb-839f-bc220435b550-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.278067 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.257945 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.278067 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.258058 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af43d18-1d68-45cb-839f-bc220435b550-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.278067 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.258795 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.278067 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.259094 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.278067 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.259341 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.278067 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.262129 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.362975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.356290 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af43d18-1d68-45cb-839f-bc220435b550-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.362975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.356922 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2pk5\" (UniqueName: \"kubernetes.io/projected/7af43d18-1d68-45cb-839f-bc220435b550-kube-api-access-z2pk5\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.438025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.437664 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:37.952121 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.952093 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" Nov 02 17:44:37.982751 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.982697 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:37.982872 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.982851 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:37.982995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.982969 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:37.986128 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:37.986111 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd"] Nov 02 17:44:38.148832 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:38.148794 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" event={"ID":"7af43d18-1d68-45cb-839f-bc220435b550","Type":"ContainerStarted","Data":"3797c44e3da50c979557ed0e15ca8e421b69176f863fb7610b391154b4a50559"} Nov 02 17:44:38.149023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:38.148855 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" event={"ID":"7af43d18-1d68-45cb-839f-bc220435b550","Type":"ContainerStarted","Data":"e42d5526b1e0c27161222123f5aefa7ca6b45ab1af6fbbddcabe36661fe908fb"} Nov 02 17:44:38.150393 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:38.150358 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" event={"ID":"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5","Type":"ContainerStarted","Data":"7996a3bda2218120283bac2912d2893c97793bd0ef20db36a8f2d87c9408a899"} Nov 02 17:44:38.174732 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:38.174671 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" podStartSLOduration=2.174652762 podStartE2EDuration="2.174652762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:38.173603002 +0000 UTC m=+649.962108534" watchObservedRunningTime="2026-11-02 17:44:38.174652762 +0000 UTC m=+649.963158291" Nov 02 17:44:38.202974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:38.201642 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" podStartSLOduration=3.201622464 podStartE2EDuration="3.201622464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:38.200520487 +0000 UTC m=+649.989026017" watchObservedRunningTime="2026-11-02 17:44:38.201622464 +0000 UTC m=+649.990127993" Nov 02 17:44:38.340089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:38.339999 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:38.449979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:38.442134 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:39.102972 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:39.083276 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:39.113977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:39.105090 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:39.113977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:39.105160 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" podUID="9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 17:44:39.231289 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:39.231223 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:39.440139 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:39.440058 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:39.440139 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:39.440125 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" podUID="7af43d18-1d68-45cb-839f-bc220435b550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 17:44:40.094977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:40.087095 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:40.094977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:40.087188 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" podUID="9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 17:44:40.110130 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:40.108068 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:40.439582 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:40.439497 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:40.439582 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:40.439552 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" podUID="7af43d18-1d68-45cb-839f-bc220435b550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 17:44:41.081368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:41.081328 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:41.081663 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:41.081628 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" podUID="9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 17:44:41.213038 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:41.208485 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:41.438623 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:41.438537 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:41.438623 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:41.438594 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" podUID="7af43d18-1d68-45cb-839f-bc220435b550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 17:44:42.087021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:42.080411 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:42.087021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:42.080498 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" podUID="9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 17:44:42.104989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:42.096882 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:42.438995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:42.438883 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:42.438995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:42.438946 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" podUID="7af43d18-1d68-45cb-839f-bc220435b550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 17:44:42.896175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:42.896136 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:42.921487 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:42.921449 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:44:43.082981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:43.080282 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:43.082981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:43.080335 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" podUID="9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 17:44:43.439358 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:43.439323 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:43.439754 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:43.439380 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" podUID="7af43d18-1d68-45cb-839f-bc220435b550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 17:44:43.717404 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:43.717355 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:44.085411 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:44.085320 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:44.085692 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:44.085659 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" podUID="9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 17:44:44.443046 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:44.442414 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:44.443046 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:44.442490 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" podUID="7af43d18-1d68-45cb-839f-bc220435b550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 17:44:45.007396 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:45.007358 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:45.447505 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:45.447471 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:45.452216 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:45.452148 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" podUID="7af43d18-1d68-45cb-839f-bc220435b550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 17:44:45.758465 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:45.758371 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:45.760198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:45.760178 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:45.773433 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:45.773364 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 17:44:46.795449 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:46.795385 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:46.838449 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:46.838053 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5lphd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:46.838449 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:46.838138 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" podUID="781f545f-9659-4d21-9e53-85d4b82a7214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.564091 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:47.564054 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:47.564598 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:47.564563 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" podUID="7af43d18-1d68-45cb-839f-bc220435b550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.860035 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:44:47.858566 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:44:47.877609 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:47.877575 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:47.879645 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:47.879618 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:47.885201 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:47.883207 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 17:44:48.421713 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:48.421664 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:49.017537 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:49.017494 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:49.849565 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:49.849521 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:50.744669 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.744631 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:50.752668 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.752635 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h"] Nov 02 17:44:50.767892 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.762306 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.790141 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.790091 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h"] Nov 02 17:44:50.798748 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.798715 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.798898 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.798816 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.798898 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.798855 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.798898 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.798885 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.799110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.798928 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.799110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.798975 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm9xz\" (UniqueName: \"kubernetes.io/projected/7095e3de-ee27-40f3-aabc-c7c149a5e66f-kube-api-access-mm9xz\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.799110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.799010 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.799110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.799097 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.799316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.799292 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.900504 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.900471 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.900504 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.900510 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.900753 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.900526 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.900753 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.900546 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.900753 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.900568 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm9xz\" (UniqueName: \"kubernetes.io/projected/7095e3de-ee27-40f3-aabc-c7c149a5e66f-kube-api-access-mm9xz\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.900753 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.900590 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.900976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.900783 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.900976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.900903 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.900976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.900922 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.900976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.900900 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.901181 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.901064 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.901181 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.901068 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.901326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.901299 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.901387 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.901351 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.903214 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.903187 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.903453 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.903431 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.912254 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.912228 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm9xz\" (UniqueName: \"kubernetes.io/projected/7095e3de-ee27-40f3-aabc-c7c149a5e66f-kube-api-access-mm9xz\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:50.918277 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:50.918256 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:51.104652 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:51.104568 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:51.383813 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:51.383783 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h"] Nov 02 17:44:51.385973 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:44:51.385933 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7095e3de_ee27_40f3_aabc_c7c149a5e66f.slice/crio-d3fe4b1c897a27bddd54ef132818cd05e0dcf0c806eff3b67d1c41a71e8ddfc2 WatchSource:0}: Error finding container d3fe4b1c897a27bddd54ef132818cd05e0dcf0c806eff3b67d1c41a71e8ddfc2: Status 404 returned error can't find the container with id d3fe4b1c897a27bddd54ef132818cd05e0dcf0c806eff3b67d1c41a71e8ddfc2 Nov 02 17:44:51.388351 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:51.388321 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:51.388423 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:51.388384 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:51.388423 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:51.388413 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:51.513020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:51.512879 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:51.769379 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:51.769339 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" event={"ID":"7095e3de-ee27-40f3-aabc-c7c149a5e66f","Type":"ContainerStarted","Data":"650575ab2dc870660acec8c54713817b64f073630aa09866d76717b5e52fe3fd"} Nov 02 17:44:51.769908 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:51.769403 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" event={"ID":"7095e3de-ee27-40f3-aabc-c7c149a5e66f","Type":"ContainerStarted","Data":"d3fe4b1c897a27bddd54ef132818cd05e0dcf0c806eff3b67d1c41a71e8ddfc2"} Nov 02 17:44:51.793097 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:51.793052 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" podStartSLOduration=1.7930384959999999 podStartE2EDuration="1.793038496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:51.79108095 +0000 UTC m=+663.579586478" watchObservedRunningTime="2026-11-02 17:44:51.793038496 +0000 UTC m=+663.581544024" Nov 02 17:44:51.953845 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:51.953805 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw"] Nov 02 17:44:51.957871 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:51.957847 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:51.990608 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:51.990578 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw"] Nov 02 17:44:52.018406 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.018371 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.018567 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.018527 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.018642 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.018625 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.018706 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.018684 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.018792 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.018770 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.018871 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.018855 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.018970 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.018943 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.019043 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.018998 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncwx5\" (UniqueName: \"kubernetes.io/projected/75712abb-4a4d-4b51-9b9d-620589c1ffb9-kube-api-access-ncwx5\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.019105 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.019066 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.105090 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.105002 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:44:52.106693 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.106655 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:52.106835 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.106717 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" podUID="7095e3de-ee27-40f3-aabc-c7c149a5e66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 17:44:52.120689 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.120665 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.120823 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.120750 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.120823 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.120788 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.120943 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.120908 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.121016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.120979 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.121016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.121007 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.121122 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.121036 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncwx5\" (UniqueName: \"kubernetes.io/projected/75712abb-4a4d-4b51-9b9d-620589c1ffb9-kube-api-access-ncwx5\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.121122 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.121078 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.121238 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.121121 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.121238 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.121186 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.121328 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.121232 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.121328 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.121279 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.121603 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.121584 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.121668 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.121598 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.123384 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.123361 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.123900 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.123851 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.131091 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.131072 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.131832 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.131811 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncwx5\" (UniqueName: \"kubernetes.io/projected/75712abb-4a4d-4b51-9b9d-620589c1ffb9-kube-api-access-ncwx5\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.271758 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.271722 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:52.574824 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.574798 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw"] Nov 02 17:44:52.576101 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:44:52.576073 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75712abb_4a4d_4b51_9b9d_620589c1ffb9.slice/crio-b1f1621fb811d7b6e151f4000c132125532c81f10625c289dcb47f93ee09ac2b WatchSource:0}: Error finding container b1f1621fb811d7b6e151f4000c132125532c81f10625c289dcb47f93ee09ac2b: Status 404 returned error can't find the container with id b1f1621fb811d7b6e151f4000c132125532c81f10625c289dcb47f93ee09ac2b Nov 02 17:44:52.578452 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.578417 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:52.578550 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.578500 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:52.578621 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.578546 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:52.697999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.697879 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:52.820978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.820094 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" event={"ID":"75712abb-4a4d-4b51-9b9d-620589c1ffb9","Type":"ContainerStarted","Data":"75461ba9df4b2743da54a3a84f65e48980a8ebcdac81ec891379c173b685531a"} Nov 02 17:44:52.820978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.820164 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" event={"ID":"75712abb-4a4d-4b51-9b9d-620589c1ffb9","Type":"ContainerStarted","Data":"b1f1621fb811d7b6e151f4000c132125532c81f10625c289dcb47f93ee09ac2b"} Nov 02 17:44:52.941135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:52.941042 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:53.107385 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:53.107342 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:53.107665 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:53.107630 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" podUID="7095e3de-ee27-40f3-aabc-c7c149a5e66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 17:44:53.277363 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:53.277330 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:44:53.286135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:53.286099 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:53.286413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:53.286382 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" podUID="75712abb-4a4d-4b51-9b9d-620589c1ffb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 17:44:54.108611 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:54.108561 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:54.109709 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:54.109682 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:54.109929 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:54.109903 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" podUID="7095e3de-ee27-40f3-aabc-c7c149a5e66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 17:44:54.284427 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:54.284385 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:54.284744 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:54.284714 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" podUID="75712abb-4a4d-4b51-9b9d-620589c1ffb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 17:44:55.124985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:55.111355 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:55.136812 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:55.136766 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" podUID="7095e3de-ee27-40f3-aabc-c7c149a5e66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 17:44:55.272631 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:55.272598 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:55.272817 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:55.272651 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" podUID="75712abb-4a4d-4b51-9b9d-620589c1ffb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 17:44:55.624981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:55.621490 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:56.105413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:56.105372 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:56.105713 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:56.105683 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" podUID="7095e3de-ee27-40f3-aabc-c7c149a5e66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 17:44:56.272650 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:56.272617 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:56.273072 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:56.272671 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" podUID="75712abb-4a4d-4b51-9b9d-620589c1ffb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 17:44:56.803766 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:56.803725 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:57.106510 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:57.106407 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:57.106791 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:57.106757 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" podUID="7095e3de-ee27-40f3-aabc-c7c149a5e66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 17:44:57.272611 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:57.272573 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:57.272815 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:57.272638 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" podUID="75712abb-4a4d-4b51-9b9d-620589c1ffb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 17:44:57.752033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:57.752000 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:57.917511 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:57.917472 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:58.108180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:58.106288 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:58.108180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:58.106349 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" podUID="7095e3de-ee27-40f3-aabc-c7c149a5e66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 17:44:58.272361 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:58.272319 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:58.272529 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:58.272389 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" podUID="75712abb-4a4d-4b51-9b9d-620589c1ffb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 17:44:58.958748 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:58.958717 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:59.272466 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:59.272377 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:59.272466 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:59.272450 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" podUID="75712abb-4a4d-4b51-9b9d-620589c1ffb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 17:44:59.936051 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:44:59.927875 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:00.147103 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.147057 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:45:00.147632 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.147115 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" podUID="7095e3de-ee27-40f3-aabc-c7c149a5e66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:00.179233 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.179181 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" podStartSLOduration=9.17916617 podStartE2EDuration="9.17916617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:52.857161601 +0000 UTC m=+664.645667129" watchObservedRunningTime="2026-11-02 17:45:00.17916617 +0000 UTC m=+671.967671698" Nov 02 17:45:00.179401 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.179336 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6"] Nov 02 17:45:00.186151 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.184933 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" Nov 02 17:45:00.201685 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.201654 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6"] Nov 02 17:45:00.277869 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.277834 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d92cd\" (UniqueName: \"kubernetes.io/projected/7baa3125-a22e-4ea2-aa82-7c6d361eb324-kube-api-access-d92cd\") pod \"maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6\" (UID: \"7baa3125-a22e-4ea2-aa82-7c6d361eb324\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" Nov 02 17:45:00.278025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.277977 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7baa3125-a22e-4ea2-aa82-7c6d361eb324-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6\" (UID: \"7baa3125-a22e-4ea2-aa82-7c6d361eb324\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" Nov 02 17:45:00.291360 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.291330 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\"" Nov 02 17:45:00.291746 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.291727 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:45:00.292993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.292946 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:45:00.294273 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.294227 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 17:45:00.379670 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.379639 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d92cd\" (UniqueName: \"kubernetes.io/projected/7baa3125-a22e-4ea2-aa82-7c6d361eb324-kube-api-access-d92cd\") pod \"maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6\" (UID: \"7baa3125-a22e-4ea2-aa82-7c6d361eb324\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" Nov 02 17:45:00.379967 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.379933 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7baa3125-a22e-4ea2-aa82-7c6d361eb324-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6\" (UID: \"7baa3125-a22e-4ea2-aa82-7c6d361eb324\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" Nov 02 17:45:00.382595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.382568 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7baa3125-a22e-4ea2-aa82-7c6d361eb324-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6\" (UID: \"7baa3125-a22e-4ea2-aa82-7c6d361eb324\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" Nov 02 17:45:00.403866 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.403824 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d92cd\" (UniqueName: \"kubernetes.io/projected/7baa3125-a22e-4ea2-aa82-7c6d361eb324-kube-api-access-d92cd\") pod \"maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6\" (UID: \"7baa3125-a22e-4ea2-aa82-7c6d361eb324\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" Nov 02 17:45:00.508524 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.508483 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" Nov 02 17:45:00.788197 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.788171 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6"] Nov 02 17:45:00.790401 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:45:00.790364 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7baa3125_a22e_4ea2_aa82_7c6d361eb324.slice/crio-652d426d71742ad5093be843bc6935ab395bc58d5b7b73eb7a90197af86e86ca WatchSource:0}: Error finding container 652d426d71742ad5093be843bc6935ab395bc58d5b7b73eb7a90197af86e86ca: Status 404 returned error can't find the container with id 652d426d71742ad5093be843bc6935ab395bc58d5b7b73eb7a90197af86e86ca Nov 02 17:45:00.813625 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.813582 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:45:00.814849 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.814831 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:45:00.816833 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:00.816810 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 17:45:01.012447 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:01.012418 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:45:01.054119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:01.054030 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" event={"ID":"7baa3125-a22e-4ea2-aa82-7c6d361eb324","Type":"ContainerStarted","Data":"652d426d71742ad5093be843bc6935ab395bc58d5b7b73eb7a90197af86e86ca"} Nov 02 17:45:01.289288 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:01.289183 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:02.064608 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:02.064556 5549 generic.go:356] "Generic (PLEG): container finished" podID="7baa3125-a22e-4ea2-aa82-7c6d361eb324" containerID="da427b4a2e388ffa871ec80519584ba0e2a9dd15e119c984abd84adf796008cb" exitCode=60 Nov 02 17:45:02.064796 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:02.064744 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" event={"ID":"7baa3125-a22e-4ea2-aa82-7c6d361eb324","Type":"ContainerDied","Data":"da427b4a2e388ffa871ec80519584ba0e2a9dd15e119c984abd84adf796008cb"} Nov 02 17:45:02.064971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:02.064935 5549 scope.go:122] "RemoveContainer" containerID="da427b4a2e388ffa871ec80519584ba0e2a9dd15e119c984abd84adf796008cb" Nov 02 17:45:02.431973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:02.431614 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:03.254009 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:03.251911 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" event={"ID":"7baa3125-a22e-4ea2-aa82-7c6d361eb324","Type":"ContainerStarted","Data":"3a0b1fb79a23838d44ad8522eb95168a1da61ed1a5b852cbe97edae4086e4258"} Nov 02 17:45:03.877979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:03.877281 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:04.130034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.128983 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 17:45:04.336008 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.335653 5549 generic.go:356] "Generic (PLEG): container finished" podID="7baa3125-a22e-4ea2-aa82-7c6d361eb324" containerID="3a0b1fb79a23838d44ad8522eb95168a1da61ed1a5b852cbe97edae4086e4258" exitCode=60 Nov 02 17:45:04.337413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.337385 5549 scope.go:122] "RemoveContainer" containerID="3a0b1fb79a23838d44ad8522eb95168a1da61ed1a5b852cbe97edae4086e4258" Nov 02 17:45:04.338051 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:45:04.338024 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6_odh-ai-gateway-infra(7baa3125-a22e-4ea2-aa82-7c6d361eb324)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" podUID="7baa3125-a22e-4ea2-aa82-7c6d361eb324" Nov 02 17:45:04.345024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.344655 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" event={"ID":"7baa3125-a22e-4ea2-aa82-7c6d361eb324","Type":"ContainerDied","Data":"3a0b1fb79a23838d44ad8522eb95168a1da61ed1a5b852cbe97edae4086e4258"} Nov 02 17:45:04.345295 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.345248 5549 scope.go:122] "RemoveContainer" containerID="da427b4a2e388ffa871ec80519584ba0e2a9dd15e119c984abd84adf796008cb" Nov 02 17:45:04.632464 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.632436 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:04.651895 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.651864 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75"] Nov 02 17:45:04.656976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.656943 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75" Nov 02 17:45:04.661440 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.661420 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:45:04.665425 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.665398 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75"] Nov 02 17:45:04.698540 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.698349 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7gjj\" (UniqueName: \"kubernetes.io/projected/c8566552-a7ce-4ed8-aab9-2d6b42309265-kube-api-access-c7gjj\") pod \"maas-api-revoke-sub-6f187069-8tk75\" (UID: \"c8566552-a7ce-4ed8-aab9-2d6b42309265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75" Nov 02 17:45:04.704980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.698416 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8566552-a7ce-4ed8-aab9-2d6b42309265-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f187069-8tk75\" (UID: \"c8566552-a7ce-4ed8-aab9-2d6b42309265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75" Nov 02 17:45:04.704980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.699225 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8566552-a7ce-4ed8-aab9-2d6b42309265-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f187069-8tk75\" (UID: \"c8566552-a7ce-4ed8-aab9-2d6b42309265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75" Nov 02 17:45:04.802396 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.802356 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7gjj\" (UniqueName: \"kubernetes.io/projected/c8566552-a7ce-4ed8-aab9-2d6b42309265-kube-api-access-c7gjj\") pod \"maas-api-revoke-sub-6f187069-8tk75\" (UID: \"c8566552-a7ce-4ed8-aab9-2d6b42309265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75" Nov 02 17:45:04.802569 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.802409 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8566552-a7ce-4ed8-aab9-2d6b42309265-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f187069-8tk75\" (UID: \"c8566552-a7ce-4ed8-aab9-2d6b42309265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75" Nov 02 17:45:04.802569 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.802434 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8566552-a7ce-4ed8-aab9-2d6b42309265-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f187069-8tk75\" (UID: \"c8566552-a7ce-4ed8-aab9-2d6b42309265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75" Nov 02 17:45:04.803143 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.803114 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8566552-a7ce-4ed8-aab9-2d6b42309265-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f187069-8tk75\" (UID: \"c8566552-a7ce-4ed8-aab9-2d6b42309265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75" Nov 02 17:45:04.804828 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.804801 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8566552-a7ce-4ed8-aab9-2d6b42309265-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f187069-8tk75\" (UID: \"c8566552-a7ce-4ed8-aab9-2d6b42309265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75" Nov 02 17:45:04.810484 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:04.810461 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7gjj\" (UniqueName: \"kubernetes.io/projected/c8566552-a7ce-4ed8-aab9-2d6b42309265-kube-api-access-c7gjj\") pod \"maas-api-revoke-sub-6f187069-8tk75\" (UID: \"c8566552-a7ce-4ed8-aab9-2d6b42309265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75" Nov 02 17:45:05.025030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.024996 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75" Nov 02 17:45:05.303412 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.303388 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75"] Nov 02 17:45:05.305161 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:45:05.305130 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8566552_a7ce_4ed8_aab9_2d6b42309265.slice/crio-04f048df14e5ec4bea5ac10c6133e39298b65d5c2e119e81ba0fda33bddf2db0 WatchSource:0}: Error finding container 04f048df14e5ec4bea5ac10c6133e39298b65d5c2e119e81ba0fda33bddf2db0: Status 404 returned error can't find the container with id 04f048df14e5ec4bea5ac10c6133e39298b65d5c2e119e81ba0fda33bddf2db0 Nov 02 17:45:05.342148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.342116 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75" event={"ID":"c8566552-a7ce-4ed8-aab9-2d6b42309265","Type":"ContainerStarted","Data":"04f048df14e5ec4bea5ac10c6133e39298b65d5c2e119e81ba0fda33bddf2db0"} Nov 02 17:45:05.344101 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.344081 5549 scope.go:122] "RemoveContainer" containerID="3a0b1fb79a23838d44ad8522eb95168a1da61ed1a5b852cbe97edae4086e4258" Nov 02 17:45:05.344336 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:45:05.344316 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6_odh-ai-gateway-infra(7baa3125-a22e-4ea2-aa82-7c6d361eb324)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" podUID="7baa3125-a22e-4ea2-aa82-7c6d361eb324" Nov 02 17:45:05.423350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.423313 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg"] Nov 02 17:45:05.428782 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.428759 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.438854 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.438831 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg"] Nov 02 17:45:05.537975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.536355 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:05.537975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.537193 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.537975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.537306 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.537975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.537353 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.537975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.537529 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.537975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.537620 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.537975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.537664 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.537975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.537702 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.537975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.537728 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.537975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.537757 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj4pv\" (UniqueName: \"kubernetes.io/projected/78d911a8-7c46-4ea5-b75e-919fad9e1d22-kube-api-access-xj4pv\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.638741 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.638698 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.638887 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.638762 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.638887 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.638799 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.638887 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.638817 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.638887 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.638839 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.638887 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.638870 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj4pv\" (UniqueName: \"kubernetes.io/projected/78d911a8-7c46-4ea5-b75e-919fad9e1d22-kube-api-access-xj4pv\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.639176 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.638947 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.639176 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.639083 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.639176 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.639137 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.639343 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.639248 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.639343 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.639296 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.639485 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.639460 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.639583 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.639559 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.639793 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.639774 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.641074 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.641053 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.641198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.641181 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.647482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.647458 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj4pv\" (UniqueName: \"kubernetes.io/projected/78d911a8-7c46-4ea5-b75e-919fad9e1d22-kube-api-access-xj4pv\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.647602 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.647586 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.743632 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.743597 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:05.755308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.755275 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5"] Nov 02 17:45:05.759914 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.759890 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.777151 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.777118 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5"] Nov 02 17:45:05.842370 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.842334 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.842541 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.842375 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.842541 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.842400 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.842541 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.842447 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.842541 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.842486 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.842541 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.842509 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.842811 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.842558 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xxtr\" (UniqueName: \"kubernetes.io/projected/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-kube-api-access-8xxtr\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.842811 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.842591 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.842811 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.842612 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.943549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.943462 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.943549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.943533 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xxtr\" (UniqueName: \"kubernetes.io/projected/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-kube-api-access-8xxtr\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.943792 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.943570 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.943792 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.943595 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.943792 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.943697 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.943993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.943809 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.943993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.943875 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.943993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.943973 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.944138 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.944042 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.944138 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.944115 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.944234 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.944171 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.944284 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.944228 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.944473 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.944449 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.944573 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.944479 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.946277 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.946255 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.946395 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.946317 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.953239 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.953212 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:05.953460 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:05.953439 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xxtr\" (UniqueName: \"kubernetes.io/projected/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-kube-api-access-8xxtr\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:06.044989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:06.044938 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg"] Nov 02 17:45:06.045878 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:45:06.045853 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d911a8_7c46_4ea5_b75e_919fad9e1d22.slice/crio-2260df6246d329a7d078ff2f1e3655a1e2c7446376a545aef510f35536001bca WatchSource:0}: Error finding container 2260df6246d329a7d078ff2f1e3655a1e2c7446376a545aef510f35536001bca: Status 404 returned error can't find the container with id 2260df6246d329a7d078ff2f1e3655a1e2c7446376a545aef510f35536001bca Nov 02 17:45:06.048393 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:06.048361 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:06.048482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:06.048423 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:06.048482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:06.048457 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:06.078454 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:06.078411 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:06.327011 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:06.313696 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:06.375946 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:06.375637 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" event={"ID":"78d911a8-7c46-4ea5-b75e-919fad9e1d22","Type":"ContainerStarted","Data":"0518a8bc7b9a48726ea2108a3cce5263574462fd176a311f28ce8abec003c792"} Nov 02 17:45:06.375946 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:06.375698 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" event={"ID":"78d911a8-7c46-4ea5-b75e-919fad9e1d22","Type":"ContainerStarted","Data":"2260df6246d329a7d078ff2f1e3655a1e2c7446376a545aef510f35536001bca"} Nov 02 17:45:06.377653 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:06.377629 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75" event={"ID":"c8566552-a7ce-4ed8-aab9-2d6b42309265","Type":"ContainerStarted","Data":"b08842c6275c30a290fa1fecdda09aef14d18bef2c0df484fa1713cc4bc8dbff"} Nov 02 17:45:06.388362 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:06.388340 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5"] Nov 02 17:45:06.389506 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:45:06.389485 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb3c7bb_ce84_4b8b_9f70_8b1da442fdd3.slice/crio-d811ed00ed5dc4447e508c7a57adaf7cd088f0520391c4d5841df36e019a5a03 WatchSource:0}: Error finding container d811ed00ed5dc4447e508c7a57adaf7cd088f0520391c4d5841df36e019a5a03: Status 404 returned error can't find the container with id d811ed00ed5dc4447e508c7a57adaf7cd088f0520391c4d5841df36e019a5a03 Nov 02 17:45:06.392443 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:06.392411 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:06.392540 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:06.392472 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:06.392540 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:06.392501 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:06.424012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:06.423943 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" podStartSLOduration=1.423927148 podStartE2EDuration="1.423927148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:06.400171237 +0000 UTC m=+678.188676768" watchObservedRunningTime="2026-11-02 17:45:06.423927148 +0000 UTC m=+678.212432683" Nov 02 17:45:06.743976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:06.743696 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:07.227976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:07.227450 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:07.427977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:07.427021 5549 generic.go:356] "Generic (PLEG): container finished" podID="c8566552-a7ce-4ed8-aab9-2d6b42309265" containerID="b08842c6275c30a290fa1fecdda09aef14d18bef2c0df484fa1713cc4bc8dbff" exitCode=0 Nov 02 17:45:07.427977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:07.427191 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75" event={"ID":"c8566552-a7ce-4ed8-aab9-2d6b42309265","Type":"ContainerDied","Data":"b08842c6275c30a290fa1fecdda09aef14d18bef2c0df484fa1713cc4bc8dbff"} Nov 02 17:45:07.434337 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:07.434275 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" event={"ID":"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3","Type":"ContainerStarted","Data":"33a69841875be6fe17055d08dfc74eb8eaa1f1bcb40cefb8cca7b7e4b51cb221"} Nov 02 17:45:07.434466 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:07.434366 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" event={"ID":"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3","Type":"ContainerStarted","Data":"d811ed00ed5dc4447e508c7a57adaf7cd088f0520391c4d5841df36e019a5a03"} Nov 02 17:45:07.460620 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:07.460570 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" podStartSLOduration=2.46055537 podStartE2EDuration="2.46055537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:07.456124211 +0000 UTC m=+679.244629739" watchObservedRunningTime="2026-11-02 17:45:07.46055537 +0000 UTC m=+679.249060897" Nov 02 17:45:07.667702 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:07.667679 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75" Nov 02 17:45:07.745756 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:07.745715 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 17:45:07.745926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:07.745769 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" podUID="78d911a8-7c46-4ea5-b75e-919fad9e1d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 17:45:07.776894 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:07.776799 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8566552-a7ce-4ed8-aab9-2d6b42309265-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8566552-a7ce-4ed8-aab9-2d6b42309265-maas-api-openshift-ca\") pod \"c8566552-a7ce-4ed8-aab9-2d6b42309265\" (UID: \"c8566552-a7ce-4ed8-aab9-2d6b42309265\") " Nov 02 17:45:07.777090 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:07.776968 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8566552-a7ce-4ed8-aab9-2d6b42309265-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8566552-a7ce-4ed8-aab9-2d6b42309265-maas-api-xks-ca\") pod \"c8566552-a7ce-4ed8-aab9-2d6b42309265\" (UID: \"c8566552-a7ce-4ed8-aab9-2d6b42309265\") " Nov 02 17:45:07.777090 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:07.777005 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8566552-a7ce-4ed8-aab9-2d6b42309265-kube-api-access-c7gjj\" (UniqueName: \"kubernetes.io/projected/c8566552-a7ce-4ed8-aab9-2d6b42309265-kube-api-access-c7gjj\") pod \"c8566552-a7ce-4ed8-aab9-2d6b42309265\" (UID: \"c8566552-a7ce-4ed8-aab9-2d6b42309265\") " Nov 02 17:45:07.777298 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:07.777208 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8566552-a7ce-4ed8-aab9-2d6b42309265-maas-api-openshift-ca" pod "c8566552-a7ce-4ed8-aab9-2d6b42309265" (UID: "c8566552-a7ce-4ed8-aab9-2d6b42309265"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:07.777420 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:07.777403 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8566552-a7ce-4ed8-aab9-2d6b42309265-maas-api-openshift-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:07.794078 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:07.791104 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8566552-a7ce-4ed8-aab9-2d6b42309265-maas-api-xks-ca" pod "c8566552-a7ce-4ed8-aab9-2d6b42309265" (UID: "c8566552-a7ce-4ed8-aab9-2d6b42309265"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:45:07.817468 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:07.802114 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8566552-a7ce-4ed8-aab9-2d6b42309265-kube-api-access-c7gjj" pod "c8566552-a7ce-4ed8-aab9-2d6b42309265" (UID: "c8566552-a7ce-4ed8-aab9-2d6b42309265"). InnerVolumeSpecName "kube-api-access-c7gjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:07.878600 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:07.878570 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8566552-a7ce-4ed8-aab9-2d6b42309265-maas-api-xks-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:07.878600 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:07.878596 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7gjj\" (UniqueName: \"kubernetes.io/projected/c8566552-a7ce-4ed8-aab9-2d6b42309265-kube-api-access-c7gjj\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:08.043467 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:08.043378 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:08.080780 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:08.080745 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:08.082545 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:08.082515 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 17:45:08.082772 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:08.082745 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" podUID="1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 17:45:08.441286 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:08.441254 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75" Nov 02 17:45:08.441740 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:08.441299 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75" event={"ID":"c8566552-a7ce-4ed8-aab9-2d6b42309265","Type":"ContainerDied","Data":"04f048df14e5ec4bea5ac10c6133e39298b65d5c2e119e81ba0fda33bddf2db0"} Nov 02 17:45:08.441740 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:08.441361 5549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04f048df14e5ec4bea5ac10c6133e39298b65d5c2e119e81ba0fda33bddf2db0" Nov 02 17:45:08.744012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:08.743974 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 17:45:08.744168 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:08.744031 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" podUID="78d911a8-7c46-4ea5-b75e-919fad9e1d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 17:45:08.833259 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:08.833127 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:09.101032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:09.100418 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 17:45:09.101032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:09.100486 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" podUID="1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 17:45:09.630696 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:09.630570 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:09.767045 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:09.767011 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 17:45:09.767306 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:09.767278 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" podUID="78d911a8-7c46-4ea5-b75e-919fad9e1d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 17:45:10.082975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:10.078986 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 17:45:10.082975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:10.079050 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" podUID="1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 17:45:10.467538 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:10.467473 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:10.744337 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:10.744239 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 17:45:10.744337 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:10.744299 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" podUID="78d911a8-7c46-4ea5-b75e-919fad9e1d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 17:45:11.084156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:11.084060 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 17:45:11.084417 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:11.084385 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" podUID="1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 17:45:11.426673 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:11.426585 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:11.744563 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:11.744535 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 17:45:11.745043 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:11.745016 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" podUID="78d911a8-7c46-4ea5-b75e-919fad9e1d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 17:45:12.082893 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:12.082805 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 17:45:12.083235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:12.083185 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" podUID="1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 17:45:12.432742 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:12.432519 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:12.746977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:12.746921 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 17:45:12.747627 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:12.747580 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" podUID="78d911a8-7c46-4ea5-b75e-919fad9e1d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 17:45:13.080407 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:13.080311 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 17:45:13.080711 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:13.080674 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" podUID="1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 17:45:13.746976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:13.746352 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 17:45:13.746976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:13.746434 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" podUID="78d911a8-7c46-4ea5-b75e-919fad9e1d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 17:45:14.023570 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:14.023485 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:14.079390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:14.079349 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 17:45:14.079726 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:14.079688 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" podUID="1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 17:45:14.165169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:14.165126 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:14.357147 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:14.350346 5549 scope.go:122] "RemoveContainer" containerID="3a0b1fb79a23838d44ad8522eb95168a1da61ed1a5b852cbe97edae4086e4258" Nov 02 17:45:14.946987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:14.934697 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:15.067977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.067060 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:45:15.067977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.067127 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" podUID="34d4c449-0024-4b94-91c0-41b4cb484bab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:15.095978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.095630 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns"] Nov 02 17:45:15.116140 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.096661 5549 cpu_manager.go:414] "containerMap: removing container" podUID="c8566552-a7ce-4ed8-aab9-2d6b42309265" containerName="revoke-keys" Nov 02 17:45:15.116140 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.096685 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8566552-a7ce-4ed8-aab9-2d6b42309265" containerName="revoke-keys" Nov 02 17:45:15.116140 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.096823 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8566552-a7ce-4ed8-aab9-2d6b42309265" containerName="revoke-keys" Nov 02 17:45:15.181914 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.175413 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns"] Nov 02 17:45:15.181914 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.175538 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns" Nov 02 17:45:15.186588 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.183762 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:45:15.352855 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.352760 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz5md\" (UniqueName: \"kubernetes.io/projected/bce8b771-3728-48e2-a7c5-23a7e1decd4a-kube-api-access-wz5md\") pod \"maas-api-revoke-sub-4c14ce4b-6xvns\" (UID: \"bce8b771-3728-48e2-a7c5-23a7e1decd4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns" Nov 02 17:45:15.352855 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.352844 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bce8b771-3728-48e2-a7c5-23a7e1decd4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c14ce4b-6xvns\" (UID: \"bce8b771-3728-48e2-a7c5-23a7e1decd4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns" Nov 02 17:45:15.353121 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.352870 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bce8b771-3728-48e2-a7c5-23a7e1decd4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c14ce4b-6xvns\" (UID: \"bce8b771-3728-48e2-a7c5-23a7e1decd4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns" Nov 02 17:45:15.366260 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.366227 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:15.366669 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.366640 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:15.367508 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.367486 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:45:15.444118 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.444084 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:15.447476 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.447442 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:15.448297 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.448271 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:45:15.456668 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.456614 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bce8b771-3728-48e2-a7c5-23a7e1decd4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c14ce4b-6xvns\" (UID: \"bce8b771-3728-48e2-a7c5-23a7e1decd4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns" Nov 02 17:45:15.456849 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.456687 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bce8b771-3728-48e2-a7c5-23a7e1decd4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c14ce4b-6xvns\" (UID: \"bce8b771-3728-48e2-a7c5-23a7e1decd4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns" Nov 02 17:45:15.457312 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.457285 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz5md\" (UniqueName: \"kubernetes.io/projected/bce8b771-3728-48e2-a7c5-23a7e1decd4a-kube-api-access-wz5md\") pod \"maas-api-revoke-sub-4c14ce4b-6xvns\" (UID: \"bce8b771-3728-48e2-a7c5-23a7e1decd4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns" Nov 02 17:45:15.458588 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.458561 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bce8b771-3728-48e2-a7c5-23a7e1decd4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c14ce4b-6xvns\" (UID: \"bce8b771-3728-48e2-a7c5-23a7e1decd4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns" Nov 02 17:45:15.459922 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.459894 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bce8b771-3728-48e2-a7c5-23a7e1decd4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c14ce4b-6xvns\" (UID: \"bce8b771-3728-48e2-a7c5-23a7e1decd4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns" Nov 02 17:45:15.467180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.467156 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz5md\" (UniqueName: \"kubernetes.io/projected/bce8b771-3728-48e2-a7c5-23a7e1decd4a-kube-api-access-wz5md\") pod \"maas-api-revoke-sub-4c14ce4b-6xvns\" (UID: \"bce8b771-3728-48e2-a7c5-23a7e1decd4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns" Nov 02 17:45:15.504301 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.504258 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns" Nov 02 17:45:15.662291 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.661091 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" event={"ID":"7baa3125-a22e-4ea2-aa82-7c6d361eb324","Type":"ContainerStarted","Data":"717aa94966a857bf97ee2475bde9d909946d44530c1aa2e027b0e0a69988005f"} Nov 02 17:45:15.686689 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.681578 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" podStartSLOduration=15.681561849 podStartE2EDuration="15.681561849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:03.322842238 +0000 UTC m=+675.111347768" watchObservedRunningTime="2026-11-02 17:45:15.681561849 +0000 UTC m=+687.470067378" Nov 02 17:45:15.817068 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.817035 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns"] Nov 02 17:45:15.818437 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:45:15.818412 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce8b771_3728_48e2_a7c5_23a7e1decd4a.slice/crio-c48b0c242ca189dce75f87279a651a074f368ecb1c44db89eed5fd87baa616ff WatchSource:0}: Error finding container c48b0c242ca189dce75f87279a651a074f368ecb1c44db89eed5fd87baa616ff: Status 404 returned error can't find the container with id c48b0c242ca189dce75f87279a651a074f368ecb1c44db89eed5fd87baa616ff Nov 02 17:45:15.859316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:15.859267 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:16.668903 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:16.668862 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns" event={"ID":"bce8b771-3728-48e2-a7c5-23a7e1decd4a","Type":"ContainerStarted","Data":"d87c57dfa30342cde736c2232b13f2fa04540deddb0662332757fe74ca3461c4"} Nov 02 17:45:16.669403 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:16.668967 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns" event={"ID":"bce8b771-3728-48e2-a7c5-23a7e1decd4a","Type":"ContainerStarted","Data":"c48b0c242ca189dce75f87279a651a074f368ecb1c44db89eed5fd87baa616ff"} Nov 02 17:45:16.670890 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:16.670863 5549 generic.go:356] "Generic (PLEG): container finished" podID="7baa3125-a22e-4ea2-aa82-7c6d361eb324" containerID="717aa94966a857bf97ee2475bde9d909946d44530c1aa2e027b0e0a69988005f" exitCode=60 Nov 02 17:45:16.671038 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:16.670940 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" event={"ID":"7baa3125-a22e-4ea2-aa82-7c6d361eb324","Type":"ContainerDied","Data":"717aa94966a857bf97ee2475bde9d909946d44530c1aa2e027b0e0a69988005f"} Nov 02 17:45:16.671038 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:16.671013 5549 scope.go:122] "RemoveContainer" containerID="3a0b1fb79a23838d44ad8522eb95168a1da61ed1a5b852cbe97edae4086e4258" Nov 02 17:45:16.671399 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:16.671382 5549 scope.go:122] "RemoveContainer" containerID="717aa94966a857bf97ee2475bde9d909946d44530c1aa2e027b0e0a69988005f" Nov 02 17:45:16.671731 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:45:16.671705 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6_odh-ai-gateway-infra(7baa3125-a22e-4ea2-aa82-7c6d361eb324)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" podUID="7baa3125-a22e-4ea2-aa82-7c6d361eb324" Nov 02 17:45:16.694987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:16.692132 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6"] Nov 02 17:45:16.704888 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:16.704838 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns" podStartSLOduration=2.704825049 podStartE2EDuration="2.704825049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:16.700255905 +0000 UTC m=+688.488761470" watchObservedRunningTime="2026-11-02 17:45:16.704825049 +0000 UTC m=+688.493330576" Nov 02 17:45:16.760160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:16.756322 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:16.932881 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:16.932845 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:17.587756 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:45:17.587721 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:45:17.612933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:17.612814 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:17.684488 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:17.684457 5549 generic.go:356] "Generic (PLEG): container finished" podID="bce8b771-3728-48e2-a7c5-23a7e1decd4a" containerID="d87c57dfa30342cde736c2232b13f2fa04540deddb0662332757fe74ca3461c4" exitCode=0 Nov 02 17:45:17.684999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:17.684556 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns" event={"ID":"bce8b771-3728-48e2-a7c5-23a7e1decd4a","Type":"ContainerDied","Data":"d87c57dfa30342cde736c2232b13f2fa04540deddb0662332757fe74ca3461c4"} Nov 02 17:45:17.906080 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:17.906053 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" Nov 02 17:45:17.927818 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:17.927785 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7baa3125-a22e-4ea2-aa82-7c6d361eb324-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7baa3125-a22e-4ea2-aa82-7c6d361eb324-maas-api-service-ca\") pod \"7baa3125-a22e-4ea2-aa82-7c6d361eb324\" (UID: \"7baa3125-a22e-4ea2-aa82-7c6d361eb324\") " Nov 02 17:45:17.928183 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:17.928158 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7baa3125-a22e-4ea2-aa82-7c6d361eb324-maas-api-service-ca" pod "7baa3125-a22e-4ea2-aa82-7c6d361eb324" (UID: "7baa3125-a22e-4ea2-aa82-7c6d361eb324"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:17.928268 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:17.928192 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7baa3125-a22e-4ea2-aa82-7c6d361eb324-kube-api-access-d92cd\" (UniqueName: \"kubernetes.io/projected/7baa3125-a22e-4ea2-aa82-7c6d361eb324-kube-api-access-d92cd\") pod \"7baa3125-a22e-4ea2-aa82-7c6d361eb324\" (UID: \"7baa3125-a22e-4ea2-aa82-7c6d361eb324\") " Nov 02 17:45:17.928873 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:17.928709 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7baa3125-a22e-4ea2-aa82-7c6d361eb324-maas-api-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:17.930755 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:17.930730 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7baa3125-a22e-4ea2-aa82-7c6d361eb324-kube-api-access-d92cd" pod "7baa3125-a22e-4ea2-aa82-7c6d361eb324" (UID: "7baa3125-a22e-4ea2-aa82-7c6d361eb324"). InnerVolumeSpecName "kube-api-access-d92cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:18.027909 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.027875 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:18.029559 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.029535 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d92cd\" (UniqueName: \"kubernetes.io/projected/7baa3125-a22e-4ea2-aa82-7c6d361eb324-kube-api-access-d92cd\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:18.198069 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.192279 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:18.202046 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.202003 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn"] Nov 02 17:45:18.220281 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.220248 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7baa3125-a22e-4ea2-aa82-7c6d361eb324" containerName="cleanup" Nov 02 17:45:18.220486 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.220469 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7baa3125-a22e-4ea2-aa82-7c6d361eb324" containerName="cleanup" Nov 02 17:45:18.220568 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.220558 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7baa3125-a22e-4ea2-aa82-7c6d361eb324" containerName="cleanup" Nov 02 17:45:18.220647 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.220636 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7baa3125-a22e-4ea2-aa82-7c6d361eb324" containerName="cleanup" Nov 02 17:45:18.220975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.220934 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7baa3125-a22e-4ea2-aa82-7c6d361eb324" containerName="cleanup" Nov 02 17:45:18.221139 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.221125 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7baa3125-a22e-4ea2-aa82-7c6d361eb324" containerName="cleanup" Nov 02 17:45:18.221271 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.221257 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7baa3125-a22e-4ea2-aa82-7c6d361eb324" containerName="cleanup" Nov 02 17:45:18.233165 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.233140 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn"] Nov 02 17:45:18.233497 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.233468 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" Nov 02 17:45:18.351063 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.350980 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c927ce-1e6f-4d29-88bd-5c83105326a2-plugins-config-volume\") pod \"payload-processing-external-model-bb6976b77-pkvhn\" (UID: \"22c927ce-1e6f-4d29-88bd-5c83105326a2\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" Nov 02 17:45:18.351277 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.351074 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22c927ce-1e6f-4d29-88bd-5c83105326a2-provider-credentials\") pod \"payload-processing-external-model-bb6976b77-pkvhn\" (UID: \"22c927ce-1e6f-4d29-88bd-5c83105326a2\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" Nov 02 17:45:18.351277 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.351196 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcwqm\" (UniqueName: \"kubernetes.io/projected/22c927ce-1e6f-4d29-88bd-5c83105326a2-kube-api-access-fcwqm\") pod \"payload-processing-external-model-bb6976b77-pkvhn\" (UID: \"22c927ce-1e6f-4d29-88bd-5c83105326a2\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" Nov 02 17:45:18.351277 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.351250 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22c927ce-1e6f-4d29-88bd-5c83105326a2-routing-overlay-volume\") pod \"payload-processing-external-model-bb6976b77-pkvhn\" (UID: \"22c927ce-1e6f-4d29-88bd-5c83105326a2\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" Nov 02 17:45:18.474023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.453469 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcwqm\" (UniqueName: \"kubernetes.io/projected/22c927ce-1e6f-4d29-88bd-5c83105326a2-kube-api-access-fcwqm\") pod \"payload-processing-external-model-bb6976b77-pkvhn\" (UID: \"22c927ce-1e6f-4d29-88bd-5c83105326a2\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" Nov 02 17:45:18.474023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.453540 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22c927ce-1e6f-4d29-88bd-5c83105326a2-routing-overlay-volume\") pod \"payload-processing-external-model-bb6976b77-pkvhn\" (UID: \"22c927ce-1e6f-4d29-88bd-5c83105326a2\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" Nov 02 17:45:18.474023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.453753 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c927ce-1e6f-4d29-88bd-5c83105326a2-plugins-config-volume\") pod \"payload-processing-external-model-bb6976b77-pkvhn\" (UID: \"22c927ce-1e6f-4d29-88bd-5c83105326a2\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" Nov 02 17:45:18.474023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.453817 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22c927ce-1e6f-4d29-88bd-5c83105326a2-provider-credentials\") pod \"payload-processing-external-model-bb6976b77-pkvhn\" (UID: \"22c927ce-1e6f-4d29-88bd-5c83105326a2\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" Nov 02 17:45:18.474023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.455408 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22c927ce-1e6f-4d29-88bd-5c83105326a2-routing-overlay-volume\") pod \"payload-processing-external-model-bb6976b77-pkvhn\" (UID: \"22c927ce-1e6f-4d29-88bd-5c83105326a2\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" Nov 02 17:45:18.474023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.456801 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c927ce-1e6f-4d29-88bd-5c83105326a2-plugins-config-volume\") pod \"payload-processing-external-model-bb6976b77-pkvhn\" (UID: \"22c927ce-1e6f-4d29-88bd-5c83105326a2\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" Nov 02 17:45:18.488973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.487686 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22c927ce-1e6f-4d29-88bd-5c83105326a2-provider-credentials\") pod \"payload-processing-external-model-bb6976b77-pkvhn\" (UID: \"22c927ce-1e6f-4d29-88bd-5c83105326a2\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" Nov 02 17:45:18.501974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.496295 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcwqm\" (UniqueName: \"kubernetes.io/projected/22c927ce-1e6f-4d29-88bd-5c83105326a2-kube-api-access-fcwqm\") pod \"payload-processing-external-model-bb6976b77-pkvhn\" (UID: \"22c927ce-1e6f-4d29-88bd-5c83105326a2\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" Nov 02 17:45:18.580391 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.579192 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" Nov 02 17:45:18.697581 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.697554 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" Nov 02 17:45:18.698071 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.698025 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6" event={"ID":"7baa3125-a22e-4ea2-aa82-7c6d361eb324","Type":"ContainerDied","Data":"652d426d71742ad5093be843bc6935ab395bc58d5b7b73eb7a90197af86e86ca"} Nov 02 17:45:18.698152 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.698108 5549 scope.go:122] "RemoveContainer" containerID="717aa94966a857bf97ee2475bde9d909946d44530c1aa2e027b0e0a69988005f" Nov 02 17:45:18.736195 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.736163 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6"] Nov 02 17:45:18.743707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.743681 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894025-tqqs6"] Nov 02 17:45:18.851813 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.836598 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:18.946534 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:45:18.943391 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c927ce_1e6f_4d29_88bd_5c83105326a2.slice/crio-6ed1534f176dfbbaabafcebc68d74b78ce84d656200b169020f14fa5ae5be8a7 WatchSource:0}: Error finding container 6ed1534f176dfbbaabafcebc68d74b78ce84d656200b169020f14fa5ae5be8a7: Status 404 returned error can't find the container with id 6ed1534f176dfbbaabafcebc68d74b78ce84d656200b169020f14fa5ae5be8a7 Nov 02 17:45:18.946534 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.946133 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn"] Nov 02 17:45:18.998438 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:18.998407 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns" Nov 02 17:45:19.066293 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.066262 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf"] Nov 02 17:45:19.067213 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.067193 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7baa3125-a22e-4ea2-aa82-7c6d361eb324" containerName="cleanup" Nov 02 17:45:19.067273 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.067218 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7baa3125-a22e-4ea2-aa82-7c6d361eb324" containerName="cleanup" Nov 02 17:45:19.067273 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.067264 5549 cpu_manager.go:414] "containerMap: removing container" podUID="bce8b771-3728-48e2-a7c5-23a7e1decd4a" containerName="revoke-keys" Nov 02 17:45:19.067352 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.067272 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce8b771-3728-48e2-a7c5-23a7e1decd4a" containerName="revoke-keys" Nov 02 17:45:19.067441 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.067421 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce8b771-3728-48e2-a7c5-23a7e1decd4a" containerName="revoke-keys" Nov 02 17:45:19.071480 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.071458 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf" Nov 02 17:45:19.078507 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.078482 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf"] Nov 02 17:45:19.088027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.087998 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bce8b771-3728-48e2-a7c5-23a7e1decd4a-kube-api-access-wz5md\" (UniqueName: \"kubernetes.io/projected/bce8b771-3728-48e2-a7c5-23a7e1decd4a-kube-api-access-wz5md\") pod \"bce8b771-3728-48e2-a7c5-23a7e1decd4a\" (UID: \"bce8b771-3728-48e2-a7c5-23a7e1decd4a\") " Nov 02 17:45:19.088158 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.088075 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bce8b771-3728-48e2-a7c5-23a7e1decd4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bce8b771-3728-48e2-a7c5-23a7e1decd4a-maas-api-xks-ca\") pod \"bce8b771-3728-48e2-a7c5-23a7e1decd4a\" (UID: \"bce8b771-3728-48e2-a7c5-23a7e1decd4a\") " Nov 02 17:45:19.088234 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.088214 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bce8b771-3728-48e2-a7c5-23a7e1decd4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bce8b771-3728-48e2-a7c5-23a7e1decd4a-maas-api-openshift-ca\") pod \"bce8b771-3728-48e2-a7c5-23a7e1decd4a\" (UID: \"bce8b771-3728-48e2-a7c5-23a7e1decd4a\") " Nov 02 17:45:19.089200 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.089173 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce8b771-3728-48e2-a7c5-23a7e1decd4a-maas-api-openshift-ca" pod "bce8b771-3728-48e2-a7c5-23a7e1decd4a" (UID: "bce8b771-3728-48e2-a7c5-23a7e1decd4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:19.094610 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.091919 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce8b771-3728-48e2-a7c5-23a7e1decd4a-maas-api-xks-ca" pod "bce8b771-3728-48e2-a7c5-23a7e1decd4a" (UID: "bce8b771-3728-48e2-a7c5-23a7e1decd4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:45:19.094610 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.092230 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce8b771-3728-48e2-a7c5-23a7e1decd4a-kube-api-access-wz5md" pod "bce8b771-3728-48e2-a7c5-23a7e1decd4a" (UID: "bce8b771-3728-48e2-a7c5-23a7e1decd4a"). InnerVolumeSpecName "kube-api-access-wz5md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:19.189851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.189817 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0162571e-6tgxf\" (UID: \"4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf" Nov 02 17:45:19.190065 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.189869 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d66j\" (UniqueName: \"kubernetes.io/projected/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-kube-api-access-4d66j\") pod \"maas-api-revoke-sub-0162571e-6tgxf\" (UID: \"4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf" Nov 02 17:45:19.190065 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.190029 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0162571e-6tgxf\" (UID: \"4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf" Nov 02 17:45:19.190198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.190173 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz5md\" (UniqueName: \"kubernetes.io/projected/bce8b771-3728-48e2-a7c5-23a7e1decd4a-kube-api-access-wz5md\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:19.190251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.190195 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bce8b771-3728-48e2-a7c5-23a7e1decd4a-maas-api-xks-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:19.190251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.190211 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bce8b771-3728-48e2-a7c5-23a7e1decd4a-maas-api-openshift-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:19.291022 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.290921 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0162571e-6tgxf\" (UID: \"4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf" Nov 02 17:45:19.291162 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.291101 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0162571e-6tgxf\" (UID: \"4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf" Nov 02 17:45:19.291162 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.291146 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d66j\" (UniqueName: \"kubernetes.io/projected/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-kube-api-access-4d66j\") pod \"maas-api-revoke-sub-0162571e-6tgxf\" (UID: \"4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf" Nov 02 17:45:19.291762 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.291735 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0162571e-6tgxf\" (UID: \"4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf" Nov 02 17:45:19.293230 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.293207 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0162571e-6tgxf\" (UID: \"4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf" Nov 02 17:45:19.299608 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.299589 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d66j\" (UniqueName: \"kubernetes.io/projected/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-kube-api-access-4d66j\") pod \"maas-api-revoke-sub-0162571e-6tgxf\" (UID: \"4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf" Nov 02 17:45:19.390816 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.390782 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf" Nov 02 17:45:19.659932 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.654892 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:19.710858 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.710812 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf"] Nov 02 17:45:19.720913 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.720880 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" event={"ID":"22c927ce-1e6f-4d29-88bd-5c83105326a2","Type":"ContainerStarted","Data":"12f2af7261706c13a664a86714997d97c8d5937ddb88ec83c86357bee65546f0"} Nov 02 17:45:19.721058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.720939 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" event={"ID":"22c927ce-1e6f-4d29-88bd-5c83105326a2","Type":"ContainerStarted","Data":"6ed1534f176dfbbaabafcebc68d74b78ce84d656200b169020f14fa5ae5be8a7"} Nov 02 17:45:19.721152 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.721131 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" Nov 02 17:45:19.722368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.722346 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns" Nov 02 17:45:19.722368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.722354 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns" event={"ID":"bce8b771-3728-48e2-a7c5-23a7e1decd4a","Type":"ContainerDied","Data":"c48b0c242ca189dce75f87279a651a074f368ecb1c44db89eed5fd87baa616ff"} Nov 02 17:45:19.722547 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.722390 5549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c48b0c242ca189dce75f87279a651a074f368ecb1c44db89eed5fd87baa616ff" Nov 02 17:45:19.746147 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:19.746098 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" podStartSLOduration=1.7460842319999998 podStartE2EDuration="1.746084232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:19.743228723 +0000 UTC m=+691.531734252" watchObservedRunningTime="2026-11-02 17:45:19.746084232 +0000 UTC m=+691.534589760" Nov 02 17:45:20.319734 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:20.319703 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7baa3125-a22e-4ea2-aa82-7c6d361eb324" path="/var/lib/kubelet/pods/7baa3125-a22e-4ea2-aa82-7c6d361eb324/volumes" Nov 02 17:45:20.451071 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:20.432940 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:20.738293 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:20.738254 5549 generic.go:356] "Generic (PLEG): container finished" podID="4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2" containerID="5ae610163563cc9b6bde1b33369b85fc7b868a250e86285c55849dcc29339d6b" exitCode=0 Nov 02 17:45:20.738774 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:20.738347 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf" event={"ID":"4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2","Type":"ContainerDied","Data":"5ae610163563cc9b6bde1b33369b85fc7b868a250e86285c55849dcc29339d6b"} Nov 02 17:45:20.738774 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:20.738402 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf" event={"ID":"4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2","Type":"ContainerStarted","Data":"d965205e89ba3a2cc39cda5939013410c64e2f3c037c1d08f23c0f352b2522cb"} Nov 02 17:45:21.350977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:21.349737 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:21.889734 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:21.889711 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf" Nov 02 17:45:21.955252 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:21.955218 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-maas-api-openshift-ca\") pod \"4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2\" (UID: \"4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2\") " Nov 02 17:45:21.955443 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:21.955429 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-maas-api-xks-ca\") pod \"4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2\" (UID: \"4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2\") " Nov 02 17:45:21.955545 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:21.955532 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-kube-api-access-4d66j\" (UniqueName: \"kubernetes.io/projected/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-kube-api-access-4d66j\") pod \"4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2\" (UID: \"4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2\") " Nov 02 17:45:21.955671 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:21.955645 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-maas-api-openshift-ca" pod "4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2" (UID: "4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:21.956087 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:21.956061 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-maas-api-openshift-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:21.957676 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:21.957640 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-kube-api-access-4d66j" pod "4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2" (UID: "4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2"). InnerVolumeSpecName "kube-api-access-4d66j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:21.957915 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:21.957891 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-maas-api-xks-ca" pod "4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2" (UID: "4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:45:22.057337 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:22.057259 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-maas-api-xks-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:22.057337 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:22.057286 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d66j\" (UniqueName: \"kubernetes.io/projected/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2-kube-api-access-4d66j\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:22.275595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:22.275559 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:45:22.333089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:22.332242 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:22.751877 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:22.751841 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf" event={"ID":"4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2","Type":"ContainerDied","Data":"d965205e89ba3a2cc39cda5939013410c64e2f3c037c1d08f23c0f352b2522cb"} Nov 02 17:45:22.751877 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:22.751863 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf" Nov 02 17:45:22.752113 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:22.751913 5549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d965205e89ba3a2cc39cda5939013410c64e2f3c037c1d08f23c0f352b2522cb" Nov 02 17:45:23.451808 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:23.450902 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:25.110978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:25.110063 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:25.118976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:25.117262 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:25.228902 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:25.208099 5549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:25.877578 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:25.877515 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:26.122975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:26.119046 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:26.122975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:26.119861 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:45:26.256979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:26.212847 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:26.277441 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:26.277387 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:45:26.937979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:26.890383 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 17:45:27.225058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:27.151787 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:45:27.298272 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:27.184342 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:27.334462 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:27.334416 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:27.902210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:27.902155 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:27.984680 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:27.984636 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:28.352983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:28.345194 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:28.352983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:28.313333 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 17:45:28.695547 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:28.695513 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:45:28.730349 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:28.730306 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:45:28.914976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:28.914237 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:45:29.356263 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:29.356219 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:30.150718 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:30.150670 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn"] Nov 02 17:45:30.150945 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:30.150786 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr"] Nov 02 17:45:30.154540 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:30.154515 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" Nov 02 17:45:30.212022 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:30.211948 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:30.377593 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:30.377472 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:30.645422 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:30.645389 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:31.653820 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:31.653781 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:32.484043 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:32.484003 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:33.217916 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:33.217876 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:33.935974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:33.932740 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:34.738896 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:34.738837 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:35.152146 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.152060 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" podUID="22c927ce-1e6f-4d29-88bd-5c83105326a2" containerName="payload-processing" containerID="cri-o://12f2af7261706c13a664a86714997d97c8d5937ddb88ec83c86357bee65546f0" gracePeriod=25 Nov 02 17:45:35.152311 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.152068 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" podUID="754803aa-c60f-4e58-a75e-2ace7db46790" containerName="payload-processing" containerID="cri-o://70a04227ebc1c5255e69bf712ef916f3b7ae74f7a307121c37131a5c9128d9d2" gracePeriod=25 Nov 02 17:45:35.464311 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.463097 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:35.474104 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.472622 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" Nov 02 17:45:35.543297 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.543263 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz"] Nov 02 17:45:35.544228 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.544205 5549 cpu_manager.go:414] "containerMap: removing container" podUID="4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2" containerName="revoke-keys" Nov 02 17:45:35.544340 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.544229 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2" containerName="revoke-keys" Nov 02 17:45:35.544340 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.544250 5549 cpu_manager.go:414] "containerMap: removing container" podUID="22c927ce-1e6f-4d29-88bd-5c83105326a2" containerName="payload-processing" Nov 02 17:45:35.544340 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.544259 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c927ce-1e6f-4d29-88bd-5c83105326a2" containerName="payload-processing" Nov 02 17:45:35.544457 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.544432 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2" containerName="revoke-keys" Nov 02 17:45:35.544457 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.544446 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c927ce-1e6f-4d29-88bd-5c83105326a2" containerName="payload-processing" Nov 02 17:45:35.546463 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.546445 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" Nov 02 17:45:35.548175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.548158 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.560826 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.560804 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz"] Nov 02 17:45:35.596643 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.596610 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/754803aa-c60f-4e58-a75e-2ace7db46790-kube-api-access-4wfmk\" (UniqueName: \"kubernetes.io/projected/754803aa-c60f-4e58-a75e-2ace7db46790-kube-api-access-4wfmk\") pod \"754803aa-c60f-4e58-a75e-2ace7db46790\" (UID: \"754803aa-c60f-4e58-a75e-2ace7db46790\") " Nov 02 17:45:35.596855 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.596837 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c927ce-1e6f-4d29-88bd-5c83105326a2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/22c927ce-1e6f-4d29-88bd-5c83105326a2-provider-credentials\") pod \"22c927ce-1e6f-4d29-88bd-5c83105326a2\" (UID: \"22c927ce-1e6f-4d29-88bd-5c83105326a2\") " Nov 02 17:45:35.596912 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.596886 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22c927ce-1e6f-4d29-88bd-5c83105326a2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22c927ce-1e6f-4d29-88bd-5c83105326a2-routing-overlay-volume\") pod \"22c927ce-1e6f-4d29-88bd-5c83105326a2\" (UID: \"22c927ce-1e6f-4d29-88bd-5c83105326a2\") " Nov 02 17:45:35.596996 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.596939 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/754803aa-c60f-4e58-a75e-2ace7db46790-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754803aa-c60f-4e58-a75e-2ace7db46790-plugins-config-volume\") pod \"754803aa-c60f-4e58-a75e-2ace7db46790\" (UID: \"754803aa-c60f-4e58-a75e-2ace7db46790\") " Nov 02 17:45:35.597053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.597007 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22c927ce-1e6f-4d29-88bd-5c83105326a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c927ce-1e6f-4d29-88bd-5c83105326a2-plugins-config-volume\") pod \"22c927ce-1e6f-4d29-88bd-5c83105326a2\" (UID: \"22c927ce-1e6f-4d29-88bd-5c83105326a2\") " Nov 02 17:45:35.597117 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.597065 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/754803aa-c60f-4e58-a75e-2ace7db46790-provider-credentials\" (UniqueName: \"kubernetes.io/projected/754803aa-c60f-4e58-a75e-2ace7db46790-provider-credentials\") pod \"754803aa-c60f-4e58-a75e-2ace7db46790\" (UID: \"754803aa-c60f-4e58-a75e-2ace7db46790\") " Nov 02 17:45:35.597117 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.597105 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c927ce-1e6f-4d29-88bd-5c83105326a2-kube-api-access-fcwqm\" (UniqueName: \"kubernetes.io/projected/22c927ce-1e6f-4d29-88bd-5c83105326a2-kube-api-access-fcwqm\") pod \"22c927ce-1e6f-4d29-88bd-5c83105326a2\" (UID: \"22c927ce-1e6f-4d29-88bd-5c83105326a2\") " Nov 02 17:45:35.597214 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.597140 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/754803aa-c60f-4e58-a75e-2ace7db46790-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/754803aa-c60f-4e58-a75e-2ace7db46790-routing-overlay-volume\") pod \"754803aa-c60f-4e58-a75e-2ace7db46790\" (UID: \"754803aa-c60f-4e58-a75e-2ace7db46790\") " Nov 02 17:45:35.597382 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.597360 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c927ce-1e6f-4d29-88bd-5c83105326a2-routing-overlay-volume" pod "22c927ce-1e6f-4d29-88bd-5c83105326a2" (UID: "22c927ce-1e6f-4d29-88bd-5c83105326a2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:35.597382 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.597357 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.597518 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.597447 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.597569 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.597543 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.597702 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.597613 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.597759 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.597719 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h2t9\" (UniqueName: \"kubernetes.io/projected/c9b89a1f-2e31-4335-9f88-acacb8b00000-kube-api-access-4h2t9\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.597811 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.597767 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b89a1f-2e31-4335-9f88-acacb8b00000-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.597906 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.597891 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.598074 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.598042 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c927ce-1e6f-4d29-88bd-5c83105326a2-plugins-config-volume" pod "22c927ce-1e6f-4d29-88bd-5c83105326a2" (UID: "22c927ce-1e6f-4d29-88bd-5c83105326a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:35.598203 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.598118 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.598203 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.598186 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.598319 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.598245 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c927ce-1e6f-4d29-88bd-5c83105326a2-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:35.598319 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.598257 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22c927ce-1e6f-4d29-88bd-5c83105326a2-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:35.598793 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.598771 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/754803aa-c60f-4e58-a75e-2ace7db46790-plugins-config-volume" pod "754803aa-c60f-4e58-a75e-2ace7db46790" (UID: "754803aa-c60f-4e58-a75e-2ace7db46790"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:35.598940 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.598785 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/754803aa-c60f-4e58-a75e-2ace7db46790-routing-overlay-volume" pod "754803aa-c60f-4e58-a75e-2ace7db46790" (UID: "754803aa-c60f-4e58-a75e-2ace7db46790"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:35.599551 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.599522 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c927ce-1e6f-4d29-88bd-5c83105326a2-provider-credentials" pod "22c927ce-1e6f-4d29-88bd-5c83105326a2" (UID: "22c927ce-1e6f-4d29-88bd-5c83105326a2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:35.600059 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.600027 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c927ce-1e6f-4d29-88bd-5c83105326a2-kube-api-access-fcwqm" pod "22c927ce-1e6f-4d29-88bd-5c83105326a2" (UID: "22c927ce-1e6f-4d29-88bd-5c83105326a2"). InnerVolumeSpecName "kube-api-access-fcwqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:35.600463 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.600438 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754803aa-c60f-4e58-a75e-2ace7db46790-kube-api-access-4wfmk" pod "754803aa-c60f-4e58-a75e-2ace7db46790" (UID: "754803aa-c60f-4e58-a75e-2ace7db46790"). InnerVolumeSpecName "kube-api-access-4wfmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:35.600704 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.600682 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754803aa-c60f-4e58-a75e-2ace7db46790-provider-credentials" pod "754803aa-c60f-4e58-a75e-2ace7db46790" (UID: "754803aa-c60f-4e58-a75e-2ace7db46790"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:35.677568 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.677474 5549 generic.go:356] "Generic (PLEG): container finished" podID="22c927ce-1e6f-4d29-88bd-5c83105326a2" containerID="12f2af7261706c13a664a86714997d97c8d5937ddb88ec83c86357bee65546f0" exitCode=0 Nov 02 17:45:35.677568 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.677522 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" Nov 02 17:45:35.677784 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.677558 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" event={"ID":"22c927ce-1e6f-4d29-88bd-5c83105326a2","Type":"ContainerDied","Data":"12f2af7261706c13a664a86714997d97c8d5937ddb88ec83c86357bee65546f0"} Nov 02 17:45:35.677784 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.677616 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn" event={"ID":"22c927ce-1e6f-4d29-88bd-5c83105326a2","Type":"ContainerDied","Data":"6ed1534f176dfbbaabafcebc68d74b78ce84d656200b169020f14fa5ae5be8a7"} Nov 02 17:45:35.677784 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.677642 5549 scope.go:122] "RemoveContainer" containerID="12f2af7261706c13a664a86714997d97c8d5937ddb88ec83c86357bee65546f0" Nov 02 17:45:35.679061 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.679038 5549 generic.go:356] "Generic (PLEG): container finished" podID="754803aa-c60f-4e58-a75e-2ace7db46790" containerID="70a04227ebc1c5255e69bf712ef916f3b7ae74f7a307121c37131a5c9128d9d2" exitCode=0 Nov 02 17:45:35.679178 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.679085 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" Nov 02 17:45:35.679178 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.679084 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" event={"ID":"754803aa-c60f-4e58-a75e-2ace7db46790","Type":"ContainerDied","Data":"70a04227ebc1c5255e69bf712ef916f3b7ae74f7a307121c37131a5c9128d9d2"} Nov 02 17:45:35.679296 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.679202 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr" event={"ID":"754803aa-c60f-4e58-a75e-2ace7db46790","Type":"ContainerDied","Data":"6c7efbc981623b729030637f24a720e6bb11c7f90a5fc9e91d8e408b814158fe"} Nov 02 17:45:35.689370 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.689352 5549 scope.go:122] "RemoveContainer" containerID="12f2af7261706c13a664a86714997d97c8d5937ddb88ec83c86357bee65546f0" Nov 02 17:45:35.689743 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:45:35.689721 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f2af7261706c13a664a86714997d97c8d5937ddb88ec83c86357bee65546f0\": container with ID starting with 12f2af7261706c13a664a86714997d97c8d5937ddb88ec83c86357bee65546f0 not found: ID does not exist" containerID="12f2af7261706c13a664a86714997d97c8d5937ddb88ec83c86357bee65546f0" Nov 02 17:45:35.689829 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.689753 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f2af7261706c13a664a86714997d97c8d5937ddb88ec83c86357bee65546f0"} err="failed to get container status \"12f2af7261706c13a664a86714997d97c8d5937ddb88ec83c86357bee65546f0\": rpc error: code = NotFound desc = could not find container \"12f2af7261706c13a664a86714997d97c8d5937ddb88ec83c86357bee65546f0\": container with ID starting with 12f2af7261706c13a664a86714997d97c8d5937ddb88ec83c86357bee65546f0 not found: ID does not exist" Nov 02 17:45:35.689829 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.689772 5549 scope.go:122] "RemoveContainer" containerID="70a04227ebc1c5255e69bf712ef916f3b7ae74f7a307121c37131a5c9128d9d2" Nov 02 17:45:35.698327 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.698306 5549 scope.go:122] "RemoveContainer" containerID="70a04227ebc1c5255e69bf712ef916f3b7ae74f7a307121c37131a5c9128d9d2" Nov 02 17:45:35.698553 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:45:35.698534 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a04227ebc1c5255e69bf712ef916f3b7ae74f7a307121c37131a5c9128d9d2\": container with ID starting with 70a04227ebc1c5255e69bf712ef916f3b7ae74f7a307121c37131a5c9128d9d2 not found: ID does not exist" containerID="70a04227ebc1c5255e69bf712ef916f3b7ae74f7a307121c37131a5c9128d9d2" Nov 02 17:45:35.698616 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.698564 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a04227ebc1c5255e69bf712ef916f3b7ae74f7a307121c37131a5c9128d9d2"} err="failed to get container status \"70a04227ebc1c5255e69bf712ef916f3b7ae74f7a307121c37131a5c9128d9d2\": rpc error: code = NotFound desc = could not find container \"70a04227ebc1c5255e69bf712ef916f3b7ae74f7a307121c37131a5c9128d9d2\": container with ID starting with 70a04227ebc1c5255e69bf712ef916f3b7ae74f7a307121c37131a5c9128d9d2 not found: ID does not exist" Nov 02 17:45:35.698841 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.698823 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.698889 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.698858 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.698889 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.698883 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.699050 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.698908 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.699203 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.699188 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.699252 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.699209 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h2t9\" (UniqueName: \"kubernetes.io/projected/c9b89a1f-2e31-4335-9f88-acacb8b00000-kube-api-access-4h2t9\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.699252 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.699213 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.699252 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.699237 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b89a1f-2e31-4335-9f88-acacb8b00000-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.699391 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.699272 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.699447 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.699431 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.699501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.699483 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.699626 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.699606 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/754803aa-c60f-4e58-a75e-2ace7db46790-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:35.699689 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.699628 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcwqm\" (UniqueName: \"kubernetes.io/projected/22c927ce-1e6f-4d29-88bd-5c83105326a2-kube-api-access-fcwqm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:35.699689 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.699647 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/754803aa-c60f-4e58-a75e-2ace7db46790-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:35.699689 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.699662 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wfmk\" (UniqueName: \"kubernetes.io/projected/754803aa-c60f-4e58-a75e-2ace7db46790-kube-api-access-4wfmk\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:35.699689 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.699681 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22c927ce-1e6f-4d29-88bd-5c83105326a2-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:35.699870 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.699694 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754803aa-c60f-4e58-a75e-2ace7db46790-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:35.699930 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.699874 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.700033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.700009 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b89a1f-2e31-4335-9f88-acacb8b00000-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.700089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.700068 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.701562 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.701499 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.701854 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.701812 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.704932 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.704910 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn"] Nov 02 17:45:35.707561 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.707542 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-bb6976b77-pkvhn"] Nov 02 17:45:35.711481 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.711461 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.711597 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.711580 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h2t9\" (UniqueName: \"kubernetes.io/projected/c9b89a1f-2e31-4335-9f88-acacb8b00000-kube-api-access-4h2t9\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:35.720847 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.720826 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr"] Nov 02 17:45:35.732864 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.732841 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6699fcb7fd-jl6qr"] Nov 02 17:45:35.865339 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:35.865305 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:36.161243 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:36.161213 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz"] Nov 02 17:45:36.162258 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:45:36.162232 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b89a1f_2e31_4335_9f88_acacb8b00000.slice/crio-29970c00eec28221373824820509ab134e3fcc7fabb85930794f6aba7fc98975 WatchSource:0}: Error finding container 29970c00eec28221373824820509ab134e3fcc7fabb85930794f6aba7fc98975: Status 404 returned error can't find the container with id 29970c00eec28221373824820509ab134e3fcc7fabb85930794f6aba7fc98975 Nov 02 17:45:36.164574 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:36.164543 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:36.164642 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:36.164608 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:36.164642 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:36.164636 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:36.319718 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:36.319686 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22c927ce-1e6f-4d29-88bd-5c83105326a2" path="/var/lib/kubelet/pods/22c927ce-1e6f-4d29-88bd-5c83105326a2/volumes" Nov 02 17:45:36.320115 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:36.320101 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="754803aa-c60f-4e58-a75e-2ace7db46790" path="/var/lib/kubelet/pods/754803aa-c60f-4e58-a75e-2ace7db46790/volumes" Nov 02 17:45:36.373024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:36.372815 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:36.687434 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:36.687397 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" event={"ID":"c9b89a1f-2e31-4335-9f88-acacb8b00000","Type":"ContainerStarted","Data":"b7b9b7a2f9f1af600e96042257949118f8c2bcfe7ad78863a30780f878323e2f"} Nov 02 17:45:36.687642 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:36.687456 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" event={"ID":"c9b89a1f-2e31-4335-9f88-acacb8b00000","Type":"ContainerStarted","Data":"29970c00eec28221373824820509ab134e3fcc7fabb85930794f6aba7fc98975"} Nov 02 17:45:36.712376 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:36.712323 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" podStartSLOduration=1.7123100820000001 podStartE2EDuration="1.712310082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:36.709155346 +0000 UTC m=+708.497660899" watchObservedRunningTime="2026-11-02 17:45:36.712310082 +0000 UTC m=+708.500815609" Nov 02 17:45:36.865696 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:36.865660 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:37.246007 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.241661 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:37.556330 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.556241 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg"] Nov 02 17:45:37.556903 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.556884 5549 cpu_manager.go:414] "containerMap: removing container" podUID="754803aa-c60f-4e58-a75e-2ace7db46790" containerName="payload-processing" Nov 02 17:45:37.557008 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.556906 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="754803aa-c60f-4e58-a75e-2ace7db46790" containerName="payload-processing" Nov 02 17:45:37.557091 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.557075 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="754803aa-c60f-4e58-a75e-2ace7db46790" containerName="payload-processing" Nov 02 17:45:37.561796 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.561740 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.565478 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.565179 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg"] Nov 02 17:45:37.572889 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.572861 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a37cf0\"/\"default-dockercfg-k8pxt\"" Nov 02 17:45:37.573022 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.572988 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a37cf0\"/\"openshift-service-ca.crt\"" Nov 02 17:45:37.573105 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.573052 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a37cf0\"/\"kube-root-ca.crt\"" Nov 02 17:45:37.573167 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.573108 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a37cf0\"/\"distinct-w3-a37cf0-kserve-self-signed-certs\"" Nov 02 17:45:37.627274 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.627242 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-dshm\") pod \"distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.627445 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.627355 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-home\") pod \"distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.627445 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.627402 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/186b2aa4-3dd7-4329-bbcf-72dd270c371d-tls-certs\") pod \"distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.627445 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.627434 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8xcw\" (UniqueName: \"kubernetes.io/projected/186b2aa4-3dd7-4329-bbcf-72dd270c371d-kube-api-access-d8xcw\") pod \"distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.627561 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.627541 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-model-cache\") pod \"distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.627666 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.627569 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-tmp-dir\") pod \"distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.724979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.724580 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:37.734985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.728800 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-model-cache\") pod \"distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.795132 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.795078 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-tmp-dir\") pod \"distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.795458 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.795404 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-dshm\") pod \"distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.795769 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.795742 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-home\") pod \"distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.795988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.795969 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/186b2aa4-3dd7-4329-bbcf-72dd270c371d-tls-certs\") pod \"distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.796157 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.796140 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8xcw\" (UniqueName: \"kubernetes.io/projected/186b2aa4-3dd7-4329-bbcf-72dd270c371d-kube-api-access-d8xcw\") pod \"distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.796755 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.735859 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-model-cache\") pod \"distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.797228 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.797205 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-tmp-dir\") pod \"distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.812022 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.811472 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-home\") pod \"distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.816622 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.813003 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-dshm\") pod \"distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.818821 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.818794 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/186b2aa4-3dd7-4329-bbcf-72dd270c371d-tls-certs\") pod \"distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.835613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.828831 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8xcw\" (UniqueName: \"kubernetes.io/projected/186b2aa4-3dd7-4329-bbcf-72dd270c371d-kube-api-access-d8xcw\") pod \"distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:37.868981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.868231 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 17:45:37.868981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.868304 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" podUID="c9b89a1f-2e31-4335-9f88-acacb8b00000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 17:45:37.907975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:37.905703 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:38.140353 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:38.139923 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:38.234288 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:38.234242 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg"] Nov 02 17:45:38.238853 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:45:38.238327 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod186b2aa4_3dd7_4329_bbcf_72dd270c371d.slice/crio-b07bf6348e94ee6f485b2db4f09444a2cfbcbfe604b3df46eaa5ceea28bee064 WatchSource:0}: Error finding container b07bf6348e94ee6f485b2db4f09444a2cfbcbfe604b3df46eaa5ceea28bee064: Status 404 returned error can't find the container with id b07bf6348e94ee6f485b2db4f09444a2cfbcbfe604b3df46eaa5ceea28bee064 Nov 02 17:45:38.774290 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:38.774253 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" event={"ID":"186b2aa4-3dd7-4329-bbcf-72dd270c371d","Type":"ContainerStarted","Data":"06117bd3ec453ae0b4dc24b5ca5430f4068896f862e13bedf59f225a014a0bcf"} Nov 02 17:45:38.774471 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:38.774303 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" event={"ID":"186b2aa4-3dd7-4329-bbcf-72dd270c371d","Type":"ContainerStarted","Data":"b07bf6348e94ee6f485b2db4f09444a2cfbcbfe604b3df46eaa5ceea28bee064"} Nov 02 17:45:38.806013 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:38.805931 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podStartSLOduration=1.80591328 podStartE2EDuration="1.80591328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:38.805164742 +0000 UTC m=+710.593670271" watchObservedRunningTime="2026-11-02 17:45:38.80591328 +0000 UTC m=+710.594418809" Nov 02 17:45:38.865656 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:38.865622 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 17:45:38.865809 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:38.865675 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" podUID="c9b89a1f-2e31-4335-9f88-acacb8b00000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 17:45:38.906226 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:38.906188 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:39.318977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:39.311426 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:39.866321 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:39.866286 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 17:45:39.866502 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:39.866348 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" podUID="c9b89a1f-2e31-4335-9f88-acacb8b00000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 17:45:40.148025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:40.147622 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:40.531307 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:40.531265 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:40.699989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:40.696468 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:40.870426 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:40.870341 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 17:45:40.870649 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:40.870622 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" podUID="c9b89a1f-2e31-4335-9f88-acacb8b00000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 17:45:41.250975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:41.237909 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:41.869392 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:41.869351 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 17:45:41.869685 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:41.869653 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" podUID="c9b89a1f-2e31-4335-9f88-acacb8b00000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 17:45:42.915668 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:42.915627 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 17:45:42.934972 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:42.932211 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" podUID="c9b89a1f-2e31-4335-9f88-acacb8b00000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 17:45:43.891517 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:43.891474 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 17:45:43.891859 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:43.891826 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" podUID="c9b89a1f-2e31-4335-9f88-acacb8b00000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 17:45:44.387836 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:44.387791 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:44.871240 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:44.871201 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 17:45:44.871556 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:44.871522 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" podUID="c9b89a1f-2e31-4335-9f88-acacb8b00000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 17:45:44.905073 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:44.905030 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:44.937094 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:44.937039 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:45.032636 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:45.032590 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:45.113067 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:45.081803 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:45:45.113067 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:45.104154 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" podUID="34d4c449-0024-4b94-91c0-41b4cb484bab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:45.187025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:45.186112 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:45:45.187025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:45.186196 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" podUID="57692113-df46-4797-9fd6-49b5aad6f5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:45.223090 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:45.223040 5549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded" Nov 02 17:45:45.386330 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:45.386284 5549 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:45:45.386612 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:45.386576 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" podUID="bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:45.401492 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:45.401440 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:45.420989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:45.409624 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:45:45.874976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:45.866304 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 17:45:45.874976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:45.866372 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" podUID="c9b89a1f-2e31-4335-9f88-acacb8b00000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 17:45:45.936985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:45.935693 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:45.936985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:45.936324 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:45:45.955822 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:45.952515 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:45.955822 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:45.953074 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:45:46.042199 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:46.021226 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:45:46.164033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:46.143142 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:45:46.164033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:46.143482 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:45:46.549763 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:46.549727 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:46.866358 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:46.866276 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 17:45:46.866358 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:46.866343 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" podUID="c9b89a1f-2e31-4335-9f88-acacb8b00000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 17:45:47.229015 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:47.228980 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:47.589024 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:45:47.588902 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:45:47.906922 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:47.906829 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:47.925539 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:47.925512 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:48.184570 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:48.184492 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:45:48.359892 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:48.359850 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:48.648803 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:48.648769 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:48.649327 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:48.649308 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:48.650601 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:48.650578 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 17:45:49.336188 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:49.336148 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:50.138989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:50.138933 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:50.929513 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:50.929480 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:51.353901 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:51.353859 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:45:51.850978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:51.849929 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:52.266352 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.266316 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh"] Nov 02 17:45:52.346058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.346017 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.395212 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.395162 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh"] Nov 02 17:45:52.452190 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.452150 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.452389 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.452202 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/034a5399-ee48-4bfb-885c-0bcfd316985e-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.452389 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.452304 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.452389 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.452342 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.452389 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.452367 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.452624 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.452472 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.452624 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.452509 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.452736 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.452702 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sw7z\" (UniqueName: \"kubernetes.io/projected/034a5399-ee48-4bfb-885c-0bcfd316985e-kube-api-access-4sw7z\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.453014 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.452989 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.753979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.751463 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.753979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.751541 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.753979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.751575 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/034a5399-ee48-4bfb-885c-0bcfd316985e-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.753979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.751667 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.753979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.751708 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.753979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.751735 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.753979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.751848 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.753979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.751883 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.753979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.751969 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sw7z\" (UniqueName: \"kubernetes.io/projected/034a5399-ee48-4bfb-885c-0bcfd316985e-kube-api-access-4sw7z\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.778979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.776846 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.782973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.779833 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/034a5399-ee48-4bfb-885c-0bcfd316985e-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.782973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.780124 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.782973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.780403 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.782973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.781106 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.869326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.862580 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sw7z\" (UniqueName: \"kubernetes.io/projected/034a5399-ee48-4bfb-885c-0bcfd316985e-kube-api-access-4sw7z\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.869326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.864242 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.881555 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.881524 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.921975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.921592 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:52.999188 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:52.999150 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:53.461614 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:53.457565 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:54.253134 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:54.252041 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:54.658986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:54.658937 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh"] Nov 02 17:45:54.710067 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:54.710011 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:54.710359 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:54.710338 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:54.717159 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:54.717108 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:55.088977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:55.088056 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:55.239144 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:55.239044 5549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:55.243286 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:55.242100 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:55.266342 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:55.266202 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:55.279059 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:55.267245 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:45:55.536513 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:55.534059 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:55.769980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:55.755534 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:45:55.776636 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:55.776600 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" event={"ID":"034a5399-ee48-4bfb-885c-0bcfd316985e","Type":"ContainerStarted","Data":"8f60f99aca8d8f34ab7f6179f99a3fde7c25ed5c4e113361477775b1959d1483"} Nov 02 17:45:56.522543 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:56.522506 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:56.808305 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:56.808219 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" event={"ID":"034a5399-ee48-4bfb-885c-0bcfd316985e","Type":"ContainerStarted","Data":"25f459a452a6a9c3c43cc2c233924826015ec6317989ce4df6ebf40fd9a58443"} Nov 02 17:45:56.834283 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:56.834233 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" podStartSLOduration=4.834218863 podStartE2EDuration="4.834218863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:56.831989142 +0000 UTC m=+728.620494671" watchObservedRunningTime="2026-11-02 17:45:56.834218863 +0000 UTC m=+728.622724392" Nov 02 17:45:57.000155 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:57.000116 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:45:57.005505 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:57.005472 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:45:57.005664 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:57.005527 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" podUID="034a5399-ee48-4bfb-885c-0bcfd316985e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 17:45:57.368350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:57.368301 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:57.526710 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:57.526673 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:57.805370 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:57.805325 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:58.004428 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:58.004388 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:45:58.004805 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:58.004459 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" podUID="034a5399-ee48-4bfb-885c-0bcfd316985e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 17:45:58.526658 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:58.526617 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:58.684857 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:58.684764 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:58.833620 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:58.833536 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:59.001370 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:59.001330 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:45:59.001658 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:59.001628 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" podUID="034a5399-ee48-4bfb-885c-0bcfd316985e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 17:45:59.635838 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:45:59.635803 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:00.004019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:00.001305 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:46:00.004019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:00.001366 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" podUID="034a5399-ee48-4bfb-885c-0bcfd316985e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 17:46:00.511633 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:00.511590 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:01.000407 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:01.000369 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:46:01.001869 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:01.001822 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" podUID="034a5399-ee48-4bfb-885c-0bcfd316985e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 17:46:02.000348 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:02.000314 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:46:02.000663 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:02.000634 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" podUID="034a5399-ee48-4bfb-885c-0bcfd316985e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 17:46:02.077119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:02.077069 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:02.174110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:02.174061 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 02 17:46:02.762182 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:02.762113 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:02.797695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:02.797655 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:03.000399 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:03.000357 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:46:03.000716 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:03.000672 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" podUID="034a5399-ee48-4bfb-885c-0bcfd316985e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 17:46:03.084056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:03.080073 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 17:46:03.084056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:03.080743 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:46:03.251978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:03.213099 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 17:46:03.268387 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:03.252160 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 02 17:46:03.293979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:03.269272 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:46:03.776575 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:03.776546 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:46:03.803995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:03.803842 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:03.818086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:03.812427 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:03.818086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:03.813869 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:46:04.122110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:04.118749 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:46:04.122110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:04.118823 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" podUID="034a5399-ee48-4bfb-885c-0bcfd316985e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 17:46:04.242529 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:04.242496 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:46:04.271669 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:04.270568 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:04.455886 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:04.454555 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:46:04.540724 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:04.540675 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:05.105095 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:05.105055 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:46:05.105630 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:05.105610 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:46:05.106526 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:05.106505 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 17:46:05.236344 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:05.236310 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:05.942651 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:05.942620 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq"] Nov 02 17:46:05.943632 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:05.943602 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:05.946990 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:05.946967 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:05.957720 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:05.957692 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a37cf0\"/\"distinct-2-w3-a37cf0-kserve-self-signed-certs\"" Nov 02 17:46:05.961372 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:05.961348 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq"] Nov 02 17:46:06.058492 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.058455 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-dshm\") pod \"distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.058696 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.058529 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-model-cache\") pod \"distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.058776 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.058723 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5jnn\" (UniqueName: \"kubernetes.io/projected/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-kube-api-access-d5jnn\") pod \"distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.058865 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.058845 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-home\") pod \"distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.058935 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.058912 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-tmp-dir\") pod \"distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.059035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.059019 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-tls-certs\") pod \"distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.094020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.092390 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:06.162644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.159744 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5jnn\" (UniqueName: \"kubernetes.io/projected/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-kube-api-access-d5jnn\") pod \"distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.162644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.159832 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-home\") pod \"distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.162644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.159908 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-tmp-dir\") pod \"distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.162644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.159937 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-tls-certs\") pod \"distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.162644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.159985 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-dshm\") pod \"distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.162644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.160108 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-model-cache\") pod \"distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.162644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.161021 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-model-cache\") pod \"distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.162644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.161285 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-tmp-dir\") pod \"distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.162644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.162204 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-home\") pod \"distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.182032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.180248 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-dshm\") pod \"distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.208064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.199136 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5jnn\" (UniqueName: \"kubernetes.io/projected/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-kube-api-access-d5jnn\") pod \"distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.208064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.202850 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-tls-certs\") pod \"distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.268802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.261359 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:06.547148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.547110 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq"] Nov 02 17:46:06.548074 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:46:06.548044 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fa67943_f0d5_401a_a6fa_79be9f6bdb95.slice/crio-a5a8345c45d5a8ce13123e28068f8434126521a0f152b686cd8030314f5bac78 WatchSource:0}: Error finding container a5a8345c45d5a8ce13123e28068f8434126521a0f152b686cd8030314f5bac78: Status 404 returned error can't find the container with id a5a8345c45d5a8ce13123e28068f8434126521a0f152b686cd8030314f5bac78 Nov 02 17:46:06.648566 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:06.636815 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:07.163415 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:07.163378 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" event={"ID":"2fa67943-f0d5-401a-a6fa-79be9f6bdb95","Type":"ContainerStarted","Data":"b7f1122692abed2551abb878a79bd691af783825b87fab61c22ab7fd9d0df329"} Nov 02 17:46:07.163595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:07.163428 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" event={"ID":"2fa67943-f0d5-401a-a6fa-79be9f6bdb95","Type":"ContainerStarted","Data":"a5a8345c45d5a8ce13123e28068f8434126521a0f152b686cd8030314f5bac78"} Nov 02 17:46:07.186308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:07.186256 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" podStartSLOduration=2.186242573 podStartE2EDuration="2.186242573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:07.184850126 +0000 UTC m=+738.973355684" watchObservedRunningTime="2026-11-02 17:46:07.186242573 +0000 UTC m=+738.974748101" Nov 02 17:46:07.265098 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:07.265068 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:07.419998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:07.419892 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:07.648929 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:07.648891 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:08.309005 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:08.305726 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:09.152471 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:09.152425 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:09.922982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:09.903353 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:10.287072 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:10.287040 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:46:10.312400 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:10.312372 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:10.739148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:10.737535 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:11.880463 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:11.880425 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:13.243976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:13.243386 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:13.282117 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:13.282061 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:13.811133 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:13.786608 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:14.165147 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:14.165045 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:14.380213 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:14.380150 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:14.432263 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:14.404285 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:46:14.782975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:14.782258 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:14.852999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:14.838516 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:14.919097 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:14.919060 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:15.058376 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:15.058291 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:46:15.058882 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:15.058861 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:46:15.066600 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:15.066583 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:46:15.067968 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:15.067936 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:46:15.068160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:15.068137 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:46:16.021159 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:16.017544 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:16.265715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:16.265680 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:16.278228 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:16.278155 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:16.841461 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:16.840626 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:16.949457 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:16.949429 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:17.586672 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:46:17.586639 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:46:17.725976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:17.721258 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:18.402902 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:18.402845 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:18.630757 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:18.629439 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:19.606972 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:19.605273 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:20.595459 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:20.595425 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:20.872870 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:20.872777 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:21.015160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:21.015130 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:21.244010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:21.241406 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:22.156129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:22.156089 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:22.938028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:22.937990 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:23.751101 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:23.738985 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:24.718626 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:24.718586 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:25.434082 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:25.434041 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:26.237558 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:26.237505 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:26.579271 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:26.579184 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:27.014260 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:27.014221 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:27.168166 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:27.168132 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:27.337645 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:27.337553 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:28.781105 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:28.781048 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:29.033035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:29.030747 5549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:29.120073 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:29.120026 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:29.120612 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:29.120582 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 17:46:29.170978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:29.146029 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 17:46:29.265578 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:29.265530 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:29.972978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:29.972099 5549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:30.036979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:30.036765 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 17:46:30.058411 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:30.058357 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:46:30.231077 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:30.221254 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 17:46:30.296990 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:30.282342 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:30.350414 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:30.350366 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 17:46:30.413978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:30.353707 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:46:30.796004 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:30.641153 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 02 17:46:30.796004 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:30.443243 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:31.059540 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:31.059410 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:31.313858 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:31.313749 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 17:46:31.318044 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:31.317992 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:46:31.318297 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:31.318268 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" podUID="026e7c96-2366-483a-ae9c-55a92607394c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:31.358017 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:31.351716 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:31.848040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:31.644816 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:46:31.896112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:31.875185 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:46:31.896112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:31.848099 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 02 17:46:31.923573 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:31.848142 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 17:46:32.029980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:31.997626 5549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:32.071366 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:32.071143 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:46:32.071366 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:32.071255 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" podUID="75712abb-4a4d-4b51-9b9d-620589c1ffb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:32.081484 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:32.072124 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:32.113222 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:32.113128 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:46:32.116976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:32.113467 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" podUID="78d911a8-7c46-4ea5-b75e-919fad9e1d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:32.123377 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:32.123350 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:46:32.289149 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:32.289113 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:46:32.290577 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:32.290554 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:46:32.324237 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:32.323547 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:32.353110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:32.353069 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:32.705978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:32.704836 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:32.819717 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:32.819684 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:46:32.985035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:32.978309 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:46:33.070978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:33.061008 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-fl8ch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:46:33.070978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:33.061102 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" podUID="b9b1010f-8c24-44d2-8495-f11e545f2187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:33.332040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:33.331320 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:33.846089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:33.846007 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:34.737798 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:34.737757 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:35.455875 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:35.455833 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:35.609079 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:35.609029 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:35.958472 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:35.958436 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:36.243098 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:36.242995 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:36.624574 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:36.624489 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:36.978602 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:36.978546 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:37.139028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:37.138990 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:37.579340 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:37.579294 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:38.192814 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.192783 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:38.217916 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.217887 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9"] Nov 02 17:46:38.222080 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.222063 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" Nov 02 17:46:38.233843 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.233821 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9"] Nov 02 17:46:38.234166 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.234150 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2893d1\"" Nov 02 17:46:38.240249 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.240220 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e48880d-426b-4dfe-9776-339d86fd5577-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9\" (UID: \"6e48880d-426b-4dfe-9776-339d86fd5577\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" Nov 02 17:46:38.240366 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.240276 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99twd\" (UniqueName: \"kubernetes.io/projected/6e48880d-426b-4dfe-9776-339d86fd5577-kube-api-access-99twd\") pod \"payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9\" (UID: \"6e48880d-426b-4dfe-9776-339d86fd5577\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" Nov 02 17:46:38.254764 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.254739 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2"] Nov 02 17:46:38.258848 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.258832 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2" Nov 02 17:46:38.272634 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.272598 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2"] Nov 02 17:46:38.369045 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.359386 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2893d1-dockercfg-rb8wg\"" Nov 02 17:46:38.576127 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.563634 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcr9h\" (UniqueName: \"kubernetes.io/projected/098aa06e-097a-4d85-9309-baab9ced9815-kube-api-access-wcr9h\") pod \"payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2\" (UID: \"098aa06e-097a-4d85-9309-baab9ced9815\") " pod="openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2" Nov 02 17:46:38.576127 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.563746 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098aa06e-097a-4d85-9309-baab9ced9815-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2\" (UID: \"098aa06e-097a-4d85-9309-baab9ced9815\") " pod="openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2" Nov 02 17:46:38.576127 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.563807 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e48880d-426b-4dfe-9776-339d86fd5577-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9\" (UID: \"6e48880d-426b-4dfe-9776-339d86fd5577\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" Nov 02 17:46:38.576127 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.563850 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99twd\" (UniqueName: \"kubernetes.io/projected/6e48880d-426b-4dfe-9776-339d86fd5577-kube-api-access-99twd\") pod \"payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9\" (UID: \"6e48880d-426b-4dfe-9776-339d86fd5577\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" Nov 02 17:46:38.588406 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.588350 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:38.593152 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.593117 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e48880d-426b-4dfe-9776-339d86fd5577-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9\" (UID: \"6e48880d-426b-4dfe-9776-339d86fd5577\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" Nov 02 17:46:38.666893 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.666854 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098aa06e-097a-4d85-9309-baab9ced9815-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2\" (UID: \"098aa06e-097a-4d85-9309-baab9ced9815\") " pod="openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2" Nov 02 17:46:38.667184 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.667163 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcr9h\" (UniqueName: \"kubernetes.io/projected/098aa06e-097a-4d85-9309-baab9ced9815-kube-api-access-wcr9h\") pod \"payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2\" (UID: \"098aa06e-097a-4d85-9309-baab9ced9815\") " pod="openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2" Nov 02 17:46:38.669131 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.668539 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098aa06e-097a-4d85-9309-baab9ced9815-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2\" (UID: \"098aa06e-097a-4d85-9309-baab9ced9815\") " pod="openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2" Nov 02 17:46:38.676119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.676064 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99twd\" (UniqueName: \"kubernetes.io/projected/6e48880d-426b-4dfe-9776-339d86fd5577-kube-api-access-99twd\") pod \"payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9\" (UID: \"6e48880d-426b-4dfe-9776-339d86fd5577\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" Nov 02 17:46:38.676448 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.676426 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcr9h\" (UniqueName: \"kubernetes.io/projected/098aa06e-097a-4d85-9309-baab9ced9815-kube-api-access-wcr9h\") pod \"payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2\" (UID: \"098aa06e-097a-4d85-9309-baab9ced9815\") " pod="openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2" Nov 02 17:46:38.732154 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.732118 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:38.836914 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.836833 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" Nov 02 17:46:38.881053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:38.881019 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2" Nov 02 17:46:39.117017 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:39.116992 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9"] Nov 02 17:46:39.117631 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:46:39.117603 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e48880d_426b_4dfe_9776_339d86fd5577.slice/crio-ab9a02156282ef8cd441de75102a291169940df375d3b811360d4fdbffdfa4e9 WatchSource:0}: Error finding container ab9a02156282ef8cd441de75102a291169940df375d3b811360d4fdbffdfa4e9: Status 404 returned error can't find the container with id ab9a02156282ef8cd441de75102a291169940df375d3b811360d4fdbffdfa4e9 Nov 02 17:46:39.177478 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:39.177431 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2"] Nov 02 17:46:39.178380 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:46:39.178351 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098aa06e_097a_4d85_9309_baab9ced9815.slice/crio-e5c520d8b0209a7205898c4ee5a2dbdac11dabfe25beac35b96e7bb6b39099db WatchSource:0}: Error finding container e5c520d8b0209a7205898c4ee5a2dbdac11dabfe25beac35b96e7bb6b39099db: Status 404 returned error can't find the container with id e5c520d8b0209a7205898c4ee5a2dbdac11dabfe25beac35b96e7bb6b39099db Nov 02 17:46:39.335985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:39.333632 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:39.351981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:39.343936 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" event={"ID":"6e48880d-426b-4dfe-9776-339d86fd5577","Type":"ContainerStarted","Data":"68375cd32410ab2271d9bb327ff56def6f421328e41525f4d2082f36295f5a0c"} Nov 02 17:46:39.351981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:39.344014 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" event={"ID":"6e48880d-426b-4dfe-9776-339d86fd5577","Type":"ContainerStarted","Data":"ab9a02156282ef8cd441de75102a291169940df375d3b811360d4fdbffdfa4e9"} Nov 02 17:46:39.351981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:39.344060 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" Nov 02 17:46:39.374025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:39.368469 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2" event={"ID":"098aa06e-097a-4d85-9309-baab9ced9815","Type":"ContainerStarted","Data":"37bcb8d1556483e4aeaad1e0251c9714a72193fe0e9c54f09250e700f309ea54"} Nov 02 17:46:39.374025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:39.368537 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2" event={"ID":"098aa06e-097a-4d85-9309-baab9ced9815","Type":"ContainerStarted","Data":"e5c520d8b0209a7205898c4ee5a2dbdac11dabfe25beac35b96e7bb6b39099db"} Nov 02 17:46:39.374025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:39.369392 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2" Nov 02 17:46:39.401452 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:39.401402 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2" podStartSLOduration=1.4013874290000001 podStartE2EDuration="1.401387429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:39.400351349 +0000 UTC m=+771.188856876" watchObservedRunningTime="2026-11-02 17:46:39.401387429 +0000 UTC m=+771.189892996" Nov 02 17:46:39.403659 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:39.403623 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" podStartSLOduration=1.403611227 podStartE2EDuration="1.403611227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:39.381681497 +0000 UTC m=+771.170187029" watchObservedRunningTime="2026-11-02 17:46:39.403611227 +0000 UTC m=+771.192116755" Nov 02 17:46:39.911081 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:39.911048 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:40.084926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:40.084888 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:40.247923 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:40.247891 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:41.062974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.061934 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:41.238272 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.238235 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp"] Nov 02 17:46:41.242569 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.242552 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" Nov 02 17:46:41.249309 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.249281 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:41.252702 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.252677 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j"] Nov 02 17:46:41.254315 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.254293 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-da5f07e6\"" Nov 02 17:46:41.262140 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.262114 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp"] Nov 02 17:46:41.262320 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.262241 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" Nov 02 17:46:41.268736 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.268708 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j"] Nov 02 17:46:41.271514 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.271492 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-da5f07e6-dockercfg-p2lch\"" Nov 02 17:46:41.301819 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.301786 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrbqp\" (UniqueName: \"kubernetes.io/projected/53edf2f7-a11d-456f-820d-2727ca8bdada-kube-api-access-mrbqp\") pod \"payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp\" (UID: \"53edf2f7-a11d-456f-820d-2727ca8bdada\") " pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" Nov 02 17:46:41.302046 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.302023 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c629cfcd-ff99-4340-ae7f-dc417b197b43-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j\" (UID: \"c629cfcd-ff99-4340-ae7f-dc417b197b43\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" Nov 02 17:46:41.302134 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.302056 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49jvq\" (UniqueName: \"kubernetes.io/projected/c629cfcd-ff99-4340-ae7f-dc417b197b43-kube-api-access-49jvq\") pod \"payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j\" (UID: \"c629cfcd-ff99-4340-ae7f-dc417b197b43\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" Nov 02 17:46:41.302210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.302191 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53edf2f7-a11d-456f-820d-2727ca8bdada-plugins-config-volume\") pod \"payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp\" (UID: \"53edf2f7-a11d-456f-820d-2727ca8bdada\") " pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" Nov 02 17:46:41.433033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.411278 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c629cfcd-ff99-4340-ae7f-dc417b197b43-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j\" (UID: \"c629cfcd-ff99-4340-ae7f-dc417b197b43\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" Nov 02 17:46:41.433033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.411342 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49jvq\" (UniqueName: \"kubernetes.io/projected/c629cfcd-ff99-4340-ae7f-dc417b197b43-kube-api-access-49jvq\") pod \"payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j\" (UID: \"c629cfcd-ff99-4340-ae7f-dc417b197b43\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" Nov 02 17:46:41.433033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.411463 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53edf2f7-a11d-456f-820d-2727ca8bdada-plugins-config-volume\") pod \"payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp\" (UID: \"53edf2f7-a11d-456f-820d-2727ca8bdada\") " pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" Nov 02 17:46:41.433033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.411505 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrbqp\" (UniqueName: \"kubernetes.io/projected/53edf2f7-a11d-456f-820d-2727ca8bdada-kube-api-access-mrbqp\") pod \"payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp\" (UID: \"53edf2f7-a11d-456f-820d-2727ca8bdada\") " pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" Nov 02 17:46:41.433033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.413094 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c629cfcd-ff99-4340-ae7f-dc417b197b43-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j\" (UID: \"c629cfcd-ff99-4340-ae7f-dc417b197b43\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" Nov 02 17:46:41.433033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.414334 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53edf2f7-a11d-456f-820d-2727ca8bdada-plugins-config-volume\") pod \"payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp\" (UID: \"53edf2f7-a11d-456f-820d-2727ca8bdada\") " pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" Nov 02 17:46:41.472050 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.465539 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:41.472050 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.471808 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49jvq\" (UniqueName: \"kubernetes.io/projected/c629cfcd-ff99-4340-ae7f-dc417b197b43-kube-api-access-49jvq\") pod \"payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j\" (UID: \"c629cfcd-ff99-4340-ae7f-dc417b197b43\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" Nov 02 17:46:41.485981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.473002 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrbqp\" (UniqueName: \"kubernetes.io/projected/53edf2f7-a11d-456f-820d-2727ca8bdada-kube-api-access-mrbqp\") pod \"payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp\" (UID: \"53edf2f7-a11d-456f-820d-2727ca8bdada\") " pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" Nov 02 17:46:41.568027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.563437 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" Nov 02 17:46:41.589368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.588449 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" Nov 02 17:46:41.870147 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.870116 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp"] Nov 02 17:46:41.871511 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:46:41.871476 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53edf2f7_a11d_456f_820d_2727ca8bdada.slice/crio-a4febf12ab8d9e0e6f4eb04b463c6557ca975b0c900ea6403ad292a9c4872788 WatchSource:0}: Error finding container a4febf12ab8d9e0e6f4eb04b463c6557ca975b0c900ea6403ad292a9c4872788: Status 404 returned error can't find the container with id a4febf12ab8d9e0e6f4eb04b463c6557ca975b0c900ea6403ad292a9c4872788 Nov 02 17:46:41.874531 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:41.874509 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j"] Nov 02 17:46:41.876295 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:46:41.876266 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc629cfcd_ff99_4340_ae7f_dc417b197b43.slice/crio-bcc29d7ddca3b57b6500089972f1e344c8e96da8ce761fd7968a5c9a9c8a380f WatchSource:0}: Error finding container bcc29d7ddca3b57b6500089972f1e344c8e96da8ce761fd7968a5c9a9c8a380f: Status 404 returned error can't find the container with id bcc29d7ddca3b57b6500089972f1e344c8e96da8ce761fd7968a5c9a9c8a380f Nov 02 17:46:42.011101 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:42.011062 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:42.180025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:42.177315 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:42.552985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:42.552929 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" event={"ID":"c629cfcd-ff99-4340-ae7f-dc417b197b43","Type":"ContainerStarted","Data":"998011eaa59211380d0b820133ece3ca38aae5a650fa4578cb9136dd93c684fb"} Nov 02 17:46:42.553168 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:42.553014 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" event={"ID":"c629cfcd-ff99-4340-ae7f-dc417b197b43","Type":"ContainerStarted","Data":"bcc29d7ddca3b57b6500089972f1e344c8e96da8ce761fd7968a5c9a9c8a380f"} Nov 02 17:46:42.576207 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:42.554130 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" Nov 02 17:46:42.582920 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:42.582876 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" event={"ID":"53edf2f7-a11d-456f-820d-2727ca8bdada","Type":"ContainerStarted","Data":"f8714b24f0f5d968ff049acc4487c6dcb7ba60dafd26170ccfef397a5738515a"} Nov 02 17:46:42.583103 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:42.582947 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" event={"ID":"53edf2f7-a11d-456f-820d-2727ca8bdada","Type":"ContainerStarted","Data":"a4febf12ab8d9e0e6f4eb04b463c6557ca975b0c900ea6403ad292a9c4872788"} Nov 02 17:46:42.583728 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:42.583700 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" Nov 02 17:46:42.630805 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:42.630750 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" podStartSLOduration=1.630730842 podStartE2EDuration="1.630730842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:42.604751992 +0000 UTC m=+774.393257527" watchObservedRunningTime="2026-11-02 17:46:42.630730842 +0000 UTC m=+774.419236370" Nov 02 17:46:42.631663 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:42.631631 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" podStartSLOduration=1.6316202130000002 podStartE2EDuration="1.631620213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:42.628191088 +0000 UTC m=+774.416696616" watchObservedRunningTime="2026-11-02 17:46:42.631620213 +0000 UTC m=+774.420125743" Nov 02 17:46:42.837623 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:42.837471 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:43.694015 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:43.691834 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:44.316415 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:44.316363 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:45.012647 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.012613 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq"] Nov 02 17:46:45.017114 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.017092 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" Nov 02 17:46:45.024251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.024194 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-da5f07e6-serving-cert\"" Nov 02 17:46:45.028306 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.028285 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq"] Nov 02 17:46:45.123522 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.123487 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:45.142819 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.142791 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-metrics-certs\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq\" (UID: \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" Nov 02 17:46:45.143004 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.142833 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmvd6\" (UniqueName: \"kubernetes.io/projected/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-kube-api-access-kmvd6\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq\" (UID: \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" Nov 02 17:46:45.143004 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.142869 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-maas-api-tls\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq\" (UID: \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" Nov 02 17:46:45.143129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.142998 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-openshift-service-ca\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq\" (UID: \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" Nov 02 17:46:45.250869 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.250818 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-maas-api-tls\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq\" (UID: \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" Nov 02 17:46:45.251089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.250887 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-openshift-service-ca\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq\" (UID: \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" Nov 02 17:46:45.258015 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.252228 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-openshift-service-ca\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq\" (UID: \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" Nov 02 17:46:45.258015 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.252574 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-metrics-certs\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq\" (UID: \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" Nov 02 17:46:45.258015 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.252631 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmvd6\" (UniqueName: \"kubernetes.io/projected/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-kube-api-access-kmvd6\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq\" (UID: \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" Nov 02 17:46:45.271226 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.269836 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-metrics-certs\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq\" (UID: \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" Nov 02 17:46:45.280398 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.277856 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmvd6\" (UniqueName: \"kubernetes.io/projected/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-kube-api-access-kmvd6\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq\" (UID: \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" Nov 02 17:46:45.280398 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.280101 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-maas-api-tls\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq\" (UID: \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" Nov 02 17:46:45.332726 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.332688 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" Nov 02 17:46:45.430331 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.430295 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:45.498412 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.498380 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:46:45.617293 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:46:45.617248 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5f44cf_7b31_4a72_a1fa_99fa7efb2217.slice/crio-73cc343b8b401a3d80cd821291b2529dfb59a1d4bd2cfcdb4c9025940221bf71 WatchSource:0}: Error finding container 73cc343b8b401a3d80cd821291b2529dfb59a1d4bd2cfcdb4c9025940221bf71: Status 404 returned error can't find the container with id 73cc343b8b401a3d80cd821291b2529dfb59a1d4bd2cfcdb4c9025940221bf71 Nov 02 17:46:45.617672 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.617647 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq"] Nov 02 17:46:45.619097 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:45.619070 5549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:46:46.150056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:46.141188 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:46.704974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:46.698607 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" event={"ID":"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217","Type":"ContainerStarted","Data":"84e2d9add46e1b34c9e01506eca65f1e7b953fd71cebb78f33be1b4edd9a9852"} Nov 02 17:46:46.704974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:46.698724 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" event={"ID":"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217","Type":"ContainerStarted","Data":"73cc343b8b401a3d80cd821291b2529dfb59a1d4bd2cfcdb4c9025940221bf71"} Nov 02 17:46:46.704974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:46.699981 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" Nov 02 17:46:46.786026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:46.785937 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" podStartSLOduration=2.367909042 podStartE2EDuration="2.785913814s" totalImagesPullingTime="418.004772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:46.732573103 +0000 UTC m=+778.521078631" watchObservedRunningTime="2026-11-02 17:46:46.785913814 +0000 UTC m=+778.574419344" Nov 02 17:46:47.104026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:47.103515 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:47.609505 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:46:47.609452 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:46:47.999017 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:47.998967 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:48.722975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:48.704824 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:49.424018 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:49.420460 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:50.216692 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:50.216653 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:50.378286 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:50.378254 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" Nov 02 17:46:50.506195 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:50.506108 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:50.861568 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:50.861415 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:51.019976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:51.017693 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:51.166028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:51.165029 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:51.472253 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:51.472224 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2" Nov 02 17:46:51.826105 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:51.825352 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:52.373386 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.373354 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw"] Nov 02 17:46:52.377885 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.377860 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.391248 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.391222 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw"] Nov 02 17:46:52.534841 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.534793 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f199938d-ac35-4850-bb3b-85cb08feab80-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.535480 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.535447 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.541655 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.535625 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f199938d-ac35-4850-bb3b-85cb08feab80-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.542044 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.542011 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f199938d-ac35-4850-bb3b-85cb08feab80-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.544042 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.544014 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69qzh\" (UniqueName: \"kubernetes.io/projected/f199938d-ac35-4850-bb3b-85cb08feab80-kube-api-access-69qzh\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.553196 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.553136 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.553481 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.553461 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.553695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.553680 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.569979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.553807 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.581988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.570461 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:52.671369 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.671287 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.671369 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.671339 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.671564 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.671404 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.671564 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.671425 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.671564 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.671500 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f199938d-ac35-4850-bb3b-85cb08feab80-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.671564 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.671539 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.671746 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.671564 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f199938d-ac35-4850-bb3b-85cb08feab80-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.671746 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.671655 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f199938d-ac35-4850-bb3b-85cb08feab80-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.671746 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.671695 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69qzh\" (UniqueName: \"kubernetes.io/projected/f199938d-ac35-4850-bb3b-85cb08feab80-kube-api-access-69qzh\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.671886 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.671802 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.672194 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.672168 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.672339 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.672200 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.672339 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.672265 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.672442 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.672420 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f199938d-ac35-4850-bb3b-85cb08feab80-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.674001 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.673979 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.674365 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.674341 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f199938d-ac35-4850-bb3b-85cb08feab80-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.682042 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.682021 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f199938d-ac35-4850-bb3b-85cb08feab80-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.682288 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.682262 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69qzh\" (UniqueName: \"kubernetes.io/projected/f199938d-ac35-4850-bb3b-85cb08feab80-kube-api-access-69qzh\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.700115 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.700091 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:52.995851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.995823 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw"] Nov 02 17:46:52.996510 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:46:52.996484 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf199938d_ac35_4850_bb3b_85cb08feab80.slice/crio-b9a7f593f0ff13a47737cd87d122debcc854de45f2f3c1386149d74f14fe2219 WatchSource:0}: Error finding container b9a7f593f0ff13a47737cd87d122debcc854de45f2f3c1386149d74f14fe2219: Status 404 returned error can't find the container with id b9a7f593f0ff13a47737cd87d122debcc854de45f2f3c1386149d74f14fe2219 Nov 02 17:46:52.999196 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.999165 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:46:52.999263 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.999231 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:46:52.999263 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:52.999258 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:46:53.527977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:53.525189 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:53.762277 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:53.762244 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" Nov 02 17:46:53.826270 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:53.826178 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" event={"ID":"f199938d-ac35-4850-bb3b-85cb08feab80","Type":"ContainerStarted","Data":"ea156511272c1f497a586c2c628e925129034bd532bedc68da72b7cef7ad6e84"} Nov 02 17:46:53.826546 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:53.826517 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" event={"ID":"f199938d-ac35-4850-bb3b-85cb08feab80","Type":"ContainerStarted","Data":"b9a7f593f0ff13a47737cd87d122debcc854de45f2f3c1386149d74f14fe2219"} Nov 02 17:46:53.849260 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:53.849212 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" podStartSLOduration=1.849199593 podStartE2EDuration="1.849199593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:53.845128065 +0000 UTC m=+785.633633595" watchObservedRunningTime="2026-11-02 17:46:53.849199593 +0000 UTC m=+785.637705120" Nov 02 17:46:54.368567 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:54.368535 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:54.524982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:54.524574 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:54.620095 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:54.619148 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" Nov 02 17:46:54.624977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:54.620601 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" Nov 02 17:46:54.714023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:54.712015 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:46:54.714023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:54.712576 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:46:54.714023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:54.712652 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" podUID="f199938d-ac35-4850-bb3b-85cb08feab80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:46:55.346976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:55.346519 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:55.708504 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:55.708461 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:46:55.708712 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:55.708538 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" podUID="f199938d-ac35-4850-bb3b-85cb08feab80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:46:55.726770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:55.726743 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd"] Nov 02 17:46:55.731200 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:55.731181 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd" Nov 02 17:46:55.746997 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:55.746970 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd"] Nov 02 17:46:55.747570 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:55.747546 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/204f34df-a8ac-4f90-89c9-93af7a9cf479-plugins-config-volume\") pod \"payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd\" (UID: \"204f34df-a8ac-4f90-89c9-93af7a9cf479\") " pod="openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd" Nov 02 17:46:55.747726 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:55.747710 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j44ml\" (UniqueName: \"kubernetes.io/projected/204f34df-a8ac-4f90-89c9-93af7a9cf479-kube-api-access-j44ml\") pod \"payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd\" (UID: \"204f34df-a8ac-4f90-89c9-93af7a9cf479\") " pod="openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd" Nov 02 17:46:55.807978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:55.792115 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-58970f\"" Nov 02 17:46:55.807978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:55.792241 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-58970f-dockercfg-4ts5x\"" Nov 02 17:46:55.877975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:55.858529 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/204f34df-a8ac-4f90-89c9-93af7a9cf479-plugins-config-volume\") pod \"payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd\" (UID: \"204f34df-a8ac-4f90-89c9-93af7a9cf479\") " pod="openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd" Nov 02 17:46:55.877975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:55.858716 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j44ml\" (UniqueName: \"kubernetes.io/projected/204f34df-a8ac-4f90-89c9-93af7a9cf479-kube-api-access-j44ml\") pod \"payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd\" (UID: \"204f34df-a8ac-4f90-89c9-93af7a9cf479\") " pod="openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd" Nov 02 17:46:55.951974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:55.949128 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/204f34df-a8ac-4f90-89c9-93af7a9cf479-plugins-config-volume\") pod \"payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd\" (UID: \"204f34df-a8ac-4f90-89c9-93af7a9cf479\") " pod="openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd" Nov 02 17:46:55.960023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:55.954890 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:56.118975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:56.109744 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:56.170979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:56.167608 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j44ml\" (UniqueName: \"kubernetes.io/projected/204f34df-a8ac-4f90-89c9-93af7a9cf479-kube-api-access-j44ml\") pod \"payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd\" (UID: \"204f34df-a8ac-4f90-89c9-93af7a9cf479\") " pod="openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd" Nov 02 17:46:56.372295 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:56.372203 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd" Nov 02 17:46:56.474973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:56.469929 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:56.705349 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:56.701534 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:46:56.705349 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:56.701593 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" podUID="f199938d-ac35-4850-bb3b-85cb08feab80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:46:56.771099 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:46:56.771063 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204f34df_a8ac_4f90_89c9_93af7a9cf479.slice/crio-e46b8151de02a1301c5647538ba6305141f3714cd95431b53053fceabb56f371 WatchSource:0}: Error finding container e46b8151de02a1301c5647538ba6305141f3714cd95431b53053fceabb56f371: Status 404 returned error can't find the container with id e46b8151de02a1301c5647538ba6305141f3714cd95431b53053fceabb56f371 Nov 02 17:46:56.771361 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:56.771338 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd"] Nov 02 17:46:56.923110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:56.923065 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd" event={"ID":"204f34df-a8ac-4f90-89c9-93af7a9cf479","Type":"ContainerStarted","Data":"27c6f85516947e839af872ac4d3ac1d6f0a49836c72761fa2f2a90259b068887"} Nov 02 17:46:56.924294 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:56.924268 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd" Nov 02 17:46:56.924432 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:56.924317 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd" event={"ID":"204f34df-a8ac-4f90-89c9-93af7a9cf479","Type":"ContainerStarted","Data":"e46b8151de02a1301c5647538ba6305141f3714cd95431b53053fceabb56f371"} Nov 02 17:46:56.945796 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:56.945733 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd" podStartSLOduration=1.945720782 podStartE2EDuration="1.945720782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:56.942722111 +0000 UTC m=+788.731227630" watchObservedRunningTime="2026-11-02 17:46:56.945720782 +0000 UTC m=+788.734226309" Nov 02 17:46:57.039163 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:57.039073 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:57.215441 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:57.215395 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:57.704549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:57.704509 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:46:57.725551 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:57.704569 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" podUID="f199938d-ac35-4850-bb3b-85cb08feab80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:46:57.725551 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:57.704843 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:57.971440 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:57.971344 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:58.194989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:58.193446 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:58.702977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:58.701384 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:46:58.702977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:58.701449 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" podUID="f199938d-ac35-4850-bb3b-85cb08feab80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:46:58.744978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:58.743980 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:58.965428 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:58.965323 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:59.701658 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:59.701316 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:46:59.701658 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:59.701379 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" podUID="f199938d-ac35-4850-bb3b-85cb08feab80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:46:59.871034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:46:59.870988 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:00.603586 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.603553 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764"] Nov 02 17:47:00.612911 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.612875 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" Nov 02 17:47:00.620641 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.620610 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-58970f-serving-cert\"" Nov 02 17:47:00.624705 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.624673 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764"] Nov 02 17:47:00.653014 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.652983 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-metrics-certs\") pod \"maas-api-e2e-api-a-58970f-79496696b4-rf764\" (UID: \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" Nov 02 17:47:00.653184 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.653117 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-maas-api-tls\") pod \"maas-api-e2e-api-a-58970f-79496696b4-rf764\" (UID: \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" Nov 02 17:47:00.653257 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.653224 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hskd\" (UniqueName: \"kubernetes.io/projected/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-kube-api-access-7hskd\") pod \"maas-api-e2e-api-a-58970f-79496696b4-rf764\" (UID: \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" Nov 02 17:47:00.653338 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.653324 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-openshift-service-ca\") pod \"maas-api-e2e-api-a-58970f-79496696b4-rf764\" (UID: \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" Nov 02 17:47:00.704447 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.704409 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:00.716758 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.716731 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:47:00.717228 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.717207 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:47:00.718755 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.718732 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 17:47:00.754731 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.754674 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-maas-api-tls\") pod \"maas-api-e2e-api-a-58970f-79496696b4-rf764\" (UID: \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" Nov 02 17:47:00.754910 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.754806 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hskd\" (UniqueName: \"kubernetes.io/projected/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-kube-api-access-7hskd\") pod \"maas-api-e2e-api-a-58970f-79496696b4-rf764\" (UID: \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" Nov 02 17:47:00.755039 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.754925 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-openshift-service-ca\") pod \"maas-api-e2e-api-a-58970f-79496696b4-rf764\" (UID: \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" Nov 02 17:47:00.755113 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.755037 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-metrics-certs\") pod \"maas-api-e2e-api-a-58970f-79496696b4-rf764\" (UID: \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" Nov 02 17:47:00.756636 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.756607 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-openshift-service-ca\") pod \"maas-api-e2e-api-a-58970f-79496696b4-rf764\" (UID: \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" Nov 02 17:47:00.759248 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.759224 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-maas-api-tls\") pod \"maas-api-e2e-api-a-58970f-79496696b4-rf764\" (UID: \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" Nov 02 17:47:00.759349 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.759224 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-metrics-certs\") pod \"maas-api-e2e-api-a-58970f-79496696b4-rf764\" (UID: \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" Nov 02 17:47:00.790686 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.790652 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hskd\" (UniqueName: \"kubernetes.io/projected/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-kube-api-access-7hskd\") pod \"maas-api-e2e-api-a-58970f-79496696b4-rf764\" (UID: \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" Nov 02 17:47:00.874346 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.874256 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:00.956895 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:00.953222 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" Nov 02 17:47:01.013781 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.013518 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg"] Nov 02 17:47:01.020758 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.020725 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.058108 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.042043 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg"] Nov 02 17:47:01.069057 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.069024 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-token\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.069209 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.069083 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27067902-98d8-4f23-ba47-f5bb2fb86876-istiod-ca-cert\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.069209 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.069124 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-data\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.069345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.069149 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2hzv\" (UniqueName: \"kubernetes.io/projected/27067902-98d8-4f23-ba47-f5bb2fb86876-kube-api-access-t2hzv\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.069345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.069328 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-podinfo\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.069515 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.069495 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-workload-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.070598 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.070406 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-credential-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.070598 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.070451 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-workload-certs\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.070598 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.070475 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-envoy\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.172147 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.172059 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-workload-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.172315 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.172162 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-credential-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.172315 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.172193 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-workload-certs\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.172315 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.172218 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-envoy\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.172477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.172319 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-token\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.172477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.172356 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27067902-98d8-4f23-ba47-f5bb2fb86876-istiod-ca-cert\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.172477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.172395 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-data\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.172477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.172419 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2hzv\" (UniqueName: \"kubernetes.io/projected/27067902-98d8-4f23-ba47-f5bb2fb86876-kube-api-access-t2hzv\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.172477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.172467 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-podinfo\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.172685 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.172470 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-workload-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.173248 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.172994 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-data\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.173248 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.173011 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-credential-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.173248 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.173199 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-workload-certs\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.173503 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.173467 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27067902-98d8-4f23-ba47-f5bb2fb86876-istiod-ca-cert\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.174699 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.174679 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-envoy\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.175272 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.175236 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-podinfo\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.184643 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.184612 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-token\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.185254 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.185231 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2hzv\" (UniqueName: \"kubernetes.io/projected/27067902-98d8-4f23-ba47-f5bb2fb86876-kube-api-access-t2hzv\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.367548 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.367516 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764"] Nov 02 17:47:01.368902 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:47:01.368874 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc3422f_a7c8_4eae_9dba_a63aae2ae613.slice/crio-68441b36b272ca2544eeacde8f58dd870b9b52c9a5c9464d29e742493267930f WatchSource:0}: Error finding container 68441b36b272ca2544eeacde8f58dd870b9b52c9a5c9464d29e742493267930f: Status 404 returned error can't find the container with id 68441b36b272ca2544eeacde8f58dd870b9b52c9a5c9464d29e742493267930f Nov 02 17:47:01.386217 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.386189 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:01.671246 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.671217 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg"] Nov 02 17:47:01.672108 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:47:01.672081 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27067902_98d8_4f23_ba47_f5bb2fb86876.slice/crio-3d9262e70cb1e4b6fe05137bf385823754eaf8b36c664c59c2c0d467ae36fa1f WatchSource:0}: Error finding container 3d9262e70cb1e4b6fe05137bf385823754eaf8b36c664c59c2c0d467ae36fa1f: Status 404 returned error can't find the container with id 3d9262e70cb1e4b6fe05137bf385823754eaf8b36c664c59c2c0d467ae36fa1f Nov 02 17:47:01.674789 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.674757 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:47:01.674893 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.674820 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:47:01.674893 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.674849 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:47:01.828321 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:01.828291 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:02.015297 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:02.012572 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:02.046475 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:02.046444 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" event={"ID":"bcc3422f-a7c8-4eae-9dba-a63aae2ae613","Type":"ContainerStarted","Data":"094667f672742c5735d1de461a637287a71f608e4b01880da1617276cbde6a6a"} Nov 02 17:47:02.046681 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:02.046667 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" event={"ID":"bcc3422f-a7c8-4eae-9dba-a63aae2ae613","Type":"ContainerStarted","Data":"68441b36b272ca2544eeacde8f58dd870b9b52c9a5c9464d29e742493267930f"} Nov 02 17:47:02.047338 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:02.047092 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" Nov 02 17:47:02.050221 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:02.050193 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" event={"ID":"27067902-98d8-4f23-ba47-f5bb2fb86876","Type":"ContainerStarted","Data":"c42d4c65252f6713bb0cc470fff362f55e00bc17fc60f064603cbbf395c25b8e"} Nov 02 17:47:02.050339 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:02.050249 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" event={"ID":"27067902-98d8-4f23-ba47-f5bb2fb86876","Type":"ContainerStarted","Data":"3d9262e70cb1e4b6fe05137bf385823754eaf8b36c664c59c2c0d467ae36fa1f"} Nov 02 17:47:02.076384 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:02.076331 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" podStartSLOduration=1.655790036 podStartE2EDuration="2.076316502s" totalImagesPullingTime="420.526466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:02.072766385 +0000 UTC m=+793.861271928" watchObservedRunningTime="2026-11-02 17:47:02.076316502 +0000 UTC m=+793.864822026" Nov 02 17:47:02.097788 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:02.097738 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" podStartSLOduration=2.097725471 podStartE2EDuration="2.097725471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:02.094034899 +0000 UTC m=+793.882540450" watchObservedRunningTime="2026-11-02 17:47:02.097725471 +0000 UTC m=+793.886230998" Nov 02 17:47:02.163638 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:02.163604 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:02.401037 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:02.391317 5549 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 17:47:02.401037 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:02.391437 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" podUID="27067902-98d8-4f23-ba47-f5bb2fb86876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 17:47:02.401037 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:02.391865 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:02.954263 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:02.946120 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:03.305068 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:03.303434 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:03.371746 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:03.371708 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5"] Nov 02 17:47:03.376165 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:03.376146 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5" Nov 02 17:47:03.432977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:03.423623 5549 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 17:47:03.432977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:03.423677 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" podUID="27067902-98d8-4f23-ba47-f5bb2fb86876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 17:47:03.454244 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:03.454187 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-935086-dockercfg-9ckkf\"" Nov 02 17:47:03.475976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:03.473514 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-935086\"" Nov 02 17:47:03.507082 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:03.499568 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:03.568066 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:03.555898 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc76g\" (UniqueName: \"kubernetes.io/projected/253360ac-a72b-426e-8b02-77a9190e3962-kube-api-access-fc76g\") pod \"payload-processing-e2e-api-b-935086-79d69586ff-9lls5\" (UID: \"253360ac-a72b-426e-8b02-77a9190e3962\") " pod="openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5" Nov 02 17:47:03.568066 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:03.556005 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/253360ac-a72b-426e-8b02-77a9190e3962-plugins-config-volume\") pod \"payload-processing-e2e-api-b-935086-79d69586ff-9lls5\" (UID: \"253360ac-a72b-426e-8b02-77a9190e3962\") " pod="openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5" Nov 02 17:47:03.633071 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:03.633034 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5"] Nov 02 17:47:03.682392 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:03.682347 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc76g\" (UniqueName: \"kubernetes.io/projected/253360ac-a72b-426e-8b02-77a9190e3962-kube-api-access-fc76g\") pod \"payload-processing-e2e-api-b-935086-79d69586ff-9lls5\" (UID: \"253360ac-a72b-426e-8b02-77a9190e3962\") " pod="openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5" Nov 02 17:47:03.682665 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:03.682647 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/253360ac-a72b-426e-8b02-77a9190e3962-plugins-config-volume\") pod \"payload-processing-e2e-api-b-935086-79d69586ff-9lls5\" (UID: \"253360ac-a72b-426e-8b02-77a9190e3962\") " pod="openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5" Nov 02 17:47:03.705922 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:03.683847 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/253360ac-a72b-426e-8b02-77a9190e3962-plugins-config-volume\") pod \"payload-processing-e2e-api-b-935086-79d69586ff-9lls5\" (UID: \"253360ac-a72b-426e-8b02-77a9190e3962\") " pod="openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5" Nov 02 17:47:03.715025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:03.711541 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc76g\" (UniqueName: \"kubernetes.io/projected/253360ac-a72b-426e-8b02-77a9190e3962-kube-api-access-fc76g\") pod \"payload-processing-e2e-api-b-935086-79d69586ff-9lls5\" (UID: \"253360ac-a72b-426e-8b02-77a9190e3962\") " pod="openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5" Nov 02 17:47:03.917277 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:03.917188 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5" Nov 02 17:47:03.999601 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:03.999562 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:04.410436 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:04.410395 5549 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 17:47:04.411071 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:04.411030 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" podUID="27067902-98d8-4f23-ba47-f5bb2fb86876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 17:47:04.523225 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:04.523196 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5"] Nov 02 17:47:04.525968 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:47:04.525925 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253360ac_a72b_426e_8b02_77a9190e3962.slice/crio-155482c881bf24236bb1c1d1c8fe44b4743c06167c0993cfe8eacc1564856310 WatchSource:0}: Error finding container 155482c881bf24236bb1c1d1c8fe44b4743c06167c0993cfe8eacc1564856310: Status 404 returned error can't find the container with id 155482c881bf24236bb1c1d1c8fe44b4743c06167c0993cfe8eacc1564856310 Nov 02 17:47:05.067018 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:05.063074 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:05.174884 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:05.174846 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5" event={"ID":"253360ac-a72b-426e-8b02-77a9190e3962","Type":"ContainerStarted","Data":"9168cddae3018cf39b674d263625049662ee01cf1e9cc416fd73397cd4799b80"} Nov 02 17:47:05.175179 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:05.175162 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5" event={"ID":"253360ac-a72b-426e-8b02-77a9190e3962","Type":"ContainerStarted","Data":"155482c881bf24236bb1c1d1c8fe44b4743c06167c0993cfe8eacc1564856310"} Nov 02 17:47:05.176312 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:05.176290 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5" Nov 02 17:47:05.231027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:05.228428 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:05.390034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:05.389328 5549 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 17:47:05.390034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:05.389386 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" podUID="27067902-98d8-4f23-ba47-f5bb2fb86876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 17:47:06.387456 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:06.387419 5549 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 17:47:06.388100 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:06.388049 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" podUID="27067902-98d8-4f23-ba47-f5bb2fb86876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 17:47:06.559020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:06.558857 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:06.719400 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:06.719342 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:07.391822 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:07.391371 5549 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 17:47:07.391822 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:07.391434 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" podUID="27067902-98d8-4f23-ba47-f5bb2fb86876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 17:47:08.027757 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:08.027717 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:08.106188 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:08.106153 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" Nov 02 17:47:08.146244 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:08.146176 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5" podStartSLOduration=5.146154332 podStartE2EDuration="5.146154332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:05.235278447 +0000 UTC m=+797.023783955" watchObservedRunningTime="2026-11-02 17:47:08.146154332 +0000 UTC m=+799.934659864" Nov 02 17:47:08.387100 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:08.387007 5549 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 17:47:08.387276 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:08.387192 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" podUID="27067902-98d8-4f23-ba47-f5bb2fb86876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 17:47:08.962543 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:08.961586 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:09.000340 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:09.000308 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd" Nov 02 17:47:09.787305 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:09.787264 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:09.825990 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:09.825940 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:09.826395 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:09.826371 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:09.827319 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:09.827293 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:47:10.757096 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:10.757023 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:11.663980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:11.658154 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz"] Nov 02 17:47:11.668028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:11.666943 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:47:11.719854 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:11.719821 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-935086-serving-cert\"" Nov 02 17:47:11.745994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:11.745933 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72847c27-b3ec-49d0-9af8-b015051608c1-maas-api-tls\") pod \"maas-api-e2e-api-b-935086-8477ddd99f-c7rtz\" (UID: \"72847c27-b3ec-49d0-9af8-b015051608c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:47:11.746260 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:11.746237 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72847c27-b3ec-49d0-9af8-b015051608c1-openshift-service-ca\") pod \"maas-api-e2e-api-b-935086-8477ddd99f-c7rtz\" (UID: \"72847c27-b3ec-49d0-9af8-b015051608c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:47:11.746512 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:11.746489 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw8c8\" (UniqueName: \"kubernetes.io/projected/72847c27-b3ec-49d0-9af8-b015051608c1-kube-api-access-zw8c8\") pod \"maas-api-e2e-api-b-935086-8477ddd99f-c7rtz\" (UID: \"72847c27-b3ec-49d0-9af8-b015051608c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:47:11.746649 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:11.746632 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72847c27-b3ec-49d0-9af8-b015051608c1-metrics-certs\") pod \"maas-api-e2e-api-b-935086-8477ddd99f-c7rtz\" (UID: \"72847c27-b3ec-49d0-9af8-b015051608c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:47:11.773361 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:11.773308 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:11.774113 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:11.774019 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz"] Nov 02 17:47:11.854116 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:11.854082 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72847c27-b3ec-49d0-9af8-b015051608c1-maas-api-tls\") pod \"maas-api-e2e-api-b-935086-8477ddd99f-c7rtz\" (UID: \"72847c27-b3ec-49d0-9af8-b015051608c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:47:11.854316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:11.854133 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72847c27-b3ec-49d0-9af8-b015051608c1-openshift-service-ca\") pod \"maas-api-e2e-api-b-935086-8477ddd99f-c7rtz\" (UID: \"72847c27-b3ec-49d0-9af8-b015051608c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:47:11.854316 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:47:11.854232 5549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-935086-serving-cert: secret "maas-api-e2e-api-b-935086-serving-cert" not found Nov 02 17:47:11.854316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:11.854260 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw8c8\" (UniqueName: \"kubernetes.io/projected/72847c27-b3ec-49d0-9af8-b015051608c1-kube-api-access-zw8c8\") pod \"maas-api-e2e-api-b-935086-8477ddd99f-c7rtz\" (UID: \"72847c27-b3ec-49d0-9af8-b015051608c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:47:11.854316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:11.854287 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72847c27-b3ec-49d0-9af8-b015051608c1-metrics-certs\") pod \"maas-api-e2e-api-b-935086-8477ddd99f-c7rtz\" (UID: \"72847c27-b3ec-49d0-9af8-b015051608c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:47:11.854316 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:47:11.854301 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72847c27-b3ec-49d0-9af8-b015051608c1-maas-api-tls podName:72847c27-b3ec-49d0-9af8-b015051608c1 nodeName:}" failed. No retries permitted until 2026-11-02 17:47:12.354285666 +0000 UTC m=+804.142791190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/72847c27-b3ec-49d0-9af8-b015051608c1-maas-api-tls") pod "maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" (UID: "72847c27-b3ec-49d0-9af8-b015051608c1") : secret "maas-api-e2e-api-b-935086-serving-cert" not found Nov 02 17:47:11.854857 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:11.854836 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72847c27-b3ec-49d0-9af8-b015051608c1-openshift-service-ca\") pod \"maas-api-e2e-api-b-935086-8477ddd99f-c7rtz\" (UID: \"72847c27-b3ec-49d0-9af8-b015051608c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:47:11.887045 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:11.884724 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72847c27-b3ec-49d0-9af8-b015051608c1-metrics-certs\") pod \"maas-api-e2e-api-b-935086-8477ddd99f-c7rtz\" (UID: \"72847c27-b3ec-49d0-9af8-b015051608c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:47:11.904793 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:11.904755 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw8c8\" (UniqueName: \"kubernetes.io/projected/72847c27-b3ec-49d0-9af8-b015051608c1-kube-api-access-zw8c8\") pod \"maas-api-e2e-api-b-935086-8477ddd99f-c7rtz\" (UID: \"72847c27-b3ec-49d0-9af8-b015051608c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:47:11.966682 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:11.966646 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:12.201156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:12.201127 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:12.361053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:12.360975 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72847c27-b3ec-49d0-9af8-b015051608c1-maas-api-tls\") pod \"maas-api-e2e-api-b-935086-8477ddd99f-c7rtz\" (UID: \"72847c27-b3ec-49d0-9af8-b015051608c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:47:12.363394 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:12.363361 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72847c27-b3ec-49d0-9af8-b015051608c1-maas-api-tls\") pod \"maas-api-e2e-api-b-935086-8477ddd99f-c7rtz\" (UID: \"72847c27-b3ec-49d0-9af8-b015051608c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:47:12.619808 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:12.619705 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:47:12.899175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:12.898868 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:12.977161 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:12.977132 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz"] Nov 02 17:47:13.359883 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:13.359849 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" event={"ID":"72847c27-b3ec-49d0-9af8-b015051608c1","Type":"ContainerStarted","Data":"f7cf8d1aeada8fbbb4de06a4cf138d133a9136bdb5e05386c6f024f268ee58cc"} Nov 02 17:47:13.748987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:13.747371 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:14.336343 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:14.321671 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:14.378639 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:14.378606 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" event={"ID":"72847c27-b3ec-49d0-9af8-b015051608c1","Type":"ContainerStarted","Data":"3d7f1aea001b4ea22197a476f1ac916fae140103f3118709dbed4cf405584d68"} Nov 02 17:47:14.378836 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:14.378736 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:47:14.401006 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:14.400938 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" podStartSLOduration=2.888521206 podStartE2EDuration="3.400924031s" totalImagesPullingTime="512.402825ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:14.397441513 +0000 UTC m=+806.185947035" watchObservedRunningTime="2026-11-02 17:47:14.400924031 +0000 UTC m=+806.189429558" Nov 02 17:47:14.588102 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:14.585033 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:14.995166 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:14.995131 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:15.431005 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:15.430910 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:15.601977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:15.599782 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:16.438047 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:16.437891 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:17.212221 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:17.212185 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5" Nov 02 17:47:17.487667 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:17.477583 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:17.587683 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:47:17.587647 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:47:18.625974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:18.625856 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:19.281397 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:19.281363 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:19.443176 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:19.443125 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:19.709063 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:19.706535 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:20.392762 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.392728 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:47:20.467206 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.467172 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb"] Nov 02 17:47:20.482442 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.482414 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb"] Nov 02 17:47:20.482624 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.482527 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb" Nov 02 17:47:20.484860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.484831 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4"] Nov 02 17:47:20.490182 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.490155 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4" Nov 02 17:47:20.495736 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.495702 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2129b9-dockercfg-92tv9\"" Nov 02 17:47:20.497062 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.497034 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2129b9\"" Nov 02 17:47:20.501843 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.499421 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:20.501843 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.501351 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4"] Nov 02 17:47:20.546521 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.546484 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lckw\" (UniqueName: \"kubernetes.io/projected/f2215f0b-29c5-4daf-b19e-898d6d41647c-kube-api-access-9lckw\") pod \"payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb\" (UID: \"f2215f0b-29c5-4daf-b19e-898d6d41647c\") " pod="openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb" Nov 02 17:47:20.546521 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.546524 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2215f0b-29c5-4daf-b19e-898d6d41647c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb\" (UID: \"f2215f0b-29c5-4daf-b19e-898d6d41647c\") " pod="openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb" Nov 02 17:47:20.650026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.649287 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rzfj\" (UniqueName: \"kubernetes.io/projected/33b74e58-f777-4511-a68f-8c39d1eac863-kube-api-access-8rzfj\") pod \"payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4\" (UID: \"33b74e58-f777-4511-a68f-8c39d1eac863\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4" Nov 02 17:47:20.650026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.649526 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lckw\" (UniqueName: \"kubernetes.io/projected/f2215f0b-29c5-4daf-b19e-898d6d41647c-kube-api-access-9lckw\") pod \"payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb\" (UID: \"f2215f0b-29c5-4daf-b19e-898d6d41647c\") " pod="openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb" Nov 02 17:47:20.670166 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.666041 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2215f0b-29c5-4daf-b19e-898d6d41647c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb\" (UID: \"f2215f0b-29c5-4daf-b19e-898d6d41647c\") " pod="openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb" Nov 02 17:47:20.670166 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.666125 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b74e58-f777-4511-a68f-8c39d1eac863-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4\" (UID: \"33b74e58-f777-4511-a68f-8c39d1eac863\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4" Nov 02 17:47:20.670166 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.664079 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:20.670166 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.667238 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2215f0b-29c5-4daf-b19e-898d6d41647c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb\" (UID: \"f2215f0b-29c5-4daf-b19e-898d6d41647c\") " pod="openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb" Nov 02 17:47:20.680799 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.680723 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lckw\" (UniqueName: \"kubernetes.io/projected/f2215f0b-29c5-4daf-b19e-898d6d41647c-kube-api-access-9lckw\") pod \"payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb\" (UID: \"f2215f0b-29c5-4daf-b19e-898d6d41647c\") " pod="openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb" Nov 02 17:47:20.769064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.769017 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b74e58-f777-4511-a68f-8c39d1eac863-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4\" (UID: \"33b74e58-f777-4511-a68f-8c39d1eac863\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4" Nov 02 17:47:20.770679 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.770651 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rzfj\" (UniqueName: \"kubernetes.io/projected/33b74e58-f777-4511-a68f-8c39d1eac863-kube-api-access-8rzfj\") pod \"payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4\" (UID: \"33b74e58-f777-4511-a68f-8c39d1eac863\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4" Nov 02 17:47:20.771419 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.770453 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b74e58-f777-4511-a68f-8c39d1eac863-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4\" (UID: \"33b74e58-f777-4511-a68f-8c39d1eac863\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4" Nov 02 17:47:20.805975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.805312 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rzfj\" (UniqueName: \"kubernetes.io/projected/33b74e58-f777-4511-a68f-8c39d1eac863-kube-api-access-8rzfj\") pod \"payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4\" (UID: \"33b74e58-f777-4511-a68f-8c39d1eac863\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4" Nov 02 17:47:20.812630 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.812599 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb" Nov 02 17:47:20.817229 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:20.817201 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4" Nov 02 17:47:21.172982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:21.172914 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb"] Nov 02 17:47:21.173192 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:47:21.173171 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2215f0b_29c5_4daf_b19e_898d6d41647c.slice/crio-97c11ee6e5be22da3b2f9e10b8442e95cf7e070c2a902aa50af87c35d188715e WatchSource:0}: Error finding container 97c11ee6e5be22da3b2f9e10b8442e95cf7e070c2a902aa50af87c35d188715e: Status 404 returned error can't find the container with id 97c11ee6e5be22da3b2f9e10b8442e95cf7e070c2a902aa50af87c35d188715e Nov 02 17:47:21.178324 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:47:21.178296 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b74e58_f777_4511_a68f_8c39d1eac863.slice/crio-6c7853d5da2c454c6107023d7f9724879a8e839058b2f8695acb58ba4ce16a1a WatchSource:0}: Error finding container 6c7853d5da2c454c6107023d7f9724879a8e839058b2f8695acb58ba4ce16a1a: Status 404 returned error can't find the container with id 6c7853d5da2c454c6107023d7f9724879a8e839058b2f8695acb58ba4ce16a1a Nov 02 17:47:21.178438 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:21.178421 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4"] Nov 02 17:47:21.385985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:21.383496 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:21.440988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:21.440675 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4" event={"ID":"33b74e58-f777-4511-a68f-8c39d1eac863","Type":"ContainerStarted","Data":"bd91421e39b69230b77908169b19ec3370fec383edc4c935ba94c2b77685e127"} Nov 02 17:47:21.440988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:21.440742 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4" event={"ID":"33b74e58-f777-4511-a68f-8c39d1eac863","Type":"ContainerStarted","Data":"6c7853d5da2c454c6107023d7f9724879a8e839058b2f8695acb58ba4ce16a1a"} Nov 02 17:47:21.440988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:21.440768 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4" Nov 02 17:47:21.448358 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:21.443062 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb" event={"ID":"f2215f0b-29c5-4daf-b19e-898d6d41647c","Type":"ContainerStarted","Data":"642eec8f8951c950a4fc442bb0b9168dbecc252455e034bf90205019013e7551"} Nov 02 17:47:21.448358 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:21.443117 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb" event={"ID":"f2215f0b-29c5-4daf-b19e-898d6d41647c","Type":"ContainerStarted","Data":"97c11ee6e5be22da3b2f9e10b8442e95cf7e070c2a902aa50af87c35d188715e"} Nov 02 17:47:21.448358 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:21.443850 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb" Nov 02 17:47:21.466782 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:21.466665 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4" podStartSLOduration=1.466649422 podStartE2EDuration="1.466649422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:21.465994716 +0000 UTC m=+813.254500249" watchObservedRunningTime="2026-11-02 17:47:21.466649422 +0000 UTC m=+813.255154950" Nov 02 17:47:21.488067 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:21.488008 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb" podStartSLOduration=1.48799232 podStartE2EDuration="1.48799232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:21.484404657 +0000 UTC m=+813.272910202" watchObservedRunningTime="2026-11-02 17:47:21.48799232 +0000 UTC m=+813.276497876" Nov 02 17:47:22.587977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:22.584981 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:23.562977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:23.551632 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:24.269977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:24.259539 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:25.052978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:25.049888 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:25.844980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:25.842477 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:26.514976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:26.512690 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:27.017881 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.017846 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr"] Nov 02 17:47:27.030306 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.030275 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" Nov 02 17:47:27.035086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.035056 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2129b9-serving-cert\"" Nov 02 17:47:27.035231 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.035193 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr"] Nov 02 17:47:27.149975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.140264 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1efe214f-8d3b-4902-a82f-a5d0be9229b4-metrics-certs\") pod \"maas-api-e2e-api-b-2129b9-66d879b765-d5khr\" (UID: \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" Nov 02 17:47:27.149975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.140336 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1efe214f-8d3b-4902-a82f-a5d0be9229b4-maas-api-tls\") pod \"maas-api-e2e-api-b-2129b9-66d879b765-d5khr\" (UID: \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" Nov 02 17:47:27.149975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.140511 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9sjl\" (UniqueName: \"kubernetes.io/projected/1efe214f-8d3b-4902-a82f-a5d0be9229b4-kube-api-access-k9sjl\") pod \"maas-api-e2e-api-b-2129b9-66d879b765-d5khr\" (UID: \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" Nov 02 17:47:27.149975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.140619 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efe214f-8d3b-4902-a82f-a5d0be9229b4-openshift-service-ca\") pod \"maas-api-e2e-api-b-2129b9-66d879b765-d5khr\" (UID: \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" Nov 02 17:47:27.172833 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.172779 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:27.243471 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.242979 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efe214f-8d3b-4902-a82f-a5d0be9229b4-openshift-service-ca\") pod \"maas-api-e2e-api-b-2129b9-66d879b765-d5khr\" (UID: \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" Nov 02 17:47:27.243471 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.243132 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1efe214f-8d3b-4902-a82f-a5d0be9229b4-metrics-certs\") pod \"maas-api-e2e-api-b-2129b9-66d879b765-d5khr\" (UID: \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" Nov 02 17:47:27.243471 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.243172 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1efe214f-8d3b-4902-a82f-a5d0be9229b4-maas-api-tls\") pod \"maas-api-e2e-api-b-2129b9-66d879b765-d5khr\" (UID: \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" Nov 02 17:47:27.243471 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.243332 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9sjl\" (UniqueName: \"kubernetes.io/projected/1efe214f-8d3b-4902-a82f-a5d0be9229b4-kube-api-access-k9sjl\") pod \"maas-api-e2e-api-b-2129b9-66d879b765-d5khr\" (UID: \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" Nov 02 17:47:27.250975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.244431 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efe214f-8d3b-4902-a82f-a5d0be9229b4-openshift-service-ca\") pod \"maas-api-e2e-api-b-2129b9-66d879b765-d5khr\" (UID: \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" Nov 02 17:47:27.258587 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.258508 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1efe214f-8d3b-4902-a82f-a5d0be9229b4-maas-api-tls\") pod \"maas-api-e2e-api-b-2129b9-66d879b765-d5khr\" (UID: \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" Nov 02 17:47:27.272008 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.271837 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1efe214f-8d3b-4902-a82f-a5d0be9229b4-metrics-certs\") pod \"maas-api-e2e-api-b-2129b9-66d879b765-d5khr\" (UID: \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" Nov 02 17:47:27.284994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.279559 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9sjl\" (UniqueName: \"kubernetes.io/projected/1efe214f-8d3b-4902-a82f-a5d0be9229b4-kube-api-access-k9sjl\") pod \"maas-api-e2e-api-b-2129b9-66d879b765-d5khr\" (UID: \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" Nov 02 17:47:27.365891 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.365852 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" Nov 02 17:47:27.420058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.419975 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:27.585221 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.585126 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:27.650965 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.650918 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr"] Nov 02 17:47:27.651523 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:47:27.651491 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1efe214f_8d3b_4902_a82f_a5d0be9229b4.slice/crio-75a7e831edeef506ee8bdee7e8015c9788e63bb85d89e71bcbe2bce2a367e3c8 WatchSource:0}: Error finding container 75a7e831edeef506ee8bdee7e8015c9788e63bb85d89e71bcbe2bce2a367e3c8: Status 404 returned error can't find the container with id 75a7e831edeef506ee8bdee7e8015c9788e63bb85d89e71bcbe2bce2a367e3c8 Nov 02 17:47:27.821070 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.821031 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:47:27.832403 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.832371 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:27.992111 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:27.990361 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:28.434504 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:28.434464 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:28.547225 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:28.547148 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" event={"ID":"1efe214f-8d3b-4902-a82f-a5d0be9229b4","Type":"ContainerStarted","Data":"97d9851802ace83351b0c082275ed35796d0b930084690f88ac91e219699a8eb"} Nov 02 17:47:28.547225 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:28.547203 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" event={"ID":"1efe214f-8d3b-4902-a82f-a5d0be9229b4","Type":"ContainerStarted","Data":"75a7e831edeef506ee8bdee7e8015c9788e63bb85d89e71bcbe2bce2a367e3c8"} Nov 02 17:47:28.547225 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:28.547225 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" Nov 02 17:47:28.572602 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:28.572542 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" podStartSLOduration=2.108487019 podStartE2EDuration="2.572528797s" totalImagesPullingTime="464.041778ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:28.567735172 +0000 UTC m=+820.356240700" watchObservedRunningTime="2026-11-02 17:47:28.572528797 +0000 UTC m=+820.361034325" Nov 02 17:47:28.970438 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:28.970407 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:29.879126 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:29.879091 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:30.065976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:30.065445 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:31.246038 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:31.241090 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:31.603463 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:31.603371 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:31.763234 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:31.763199 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:32.390341 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:32.390211 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:32.454055 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:32.454025 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4" Nov 02 17:47:32.858469 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:32.858435 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z"] Nov 02 17:47:32.864698 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:32.864680 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z" Nov 02 17:47:32.876621 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:32.876596 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a976bc\"" Nov 02 17:47:32.877941 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:32.877918 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a976bc-dockercfg-ltvf9\"" Nov 02 17:47:32.878946 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:32.878911 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z"] Nov 02 17:47:32.883163 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:32.883139 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h"] Nov 02 17:47:32.891735 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:32.891707 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h" Nov 02 17:47:32.895937 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:32.895908 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:32.905833 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:32.905809 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h"] Nov 02 17:47:32.962927 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:32.962882 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4812a18c-24c8-487f-a327-7670d532a18c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h\" (UID: \"4812a18c-24c8-487f-a327-7670d532a18c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h" Nov 02 17:47:32.964280 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:32.963670 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f7e4cd-9c02-4b92-899b-a5201c88d254-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z\" (UID: \"17f7e4cd-9c02-4b92-899b-a5201c88d254\") " pod="openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z" Nov 02 17:47:32.964453 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:32.964332 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhhd2\" (UniqueName: \"kubernetes.io/projected/17f7e4cd-9c02-4b92-899b-a5201c88d254-kube-api-access-mhhd2\") pod \"payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z\" (UID: \"17f7e4cd-9c02-4b92-899b-a5201c88d254\") " pod="openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z" Nov 02 17:47:32.964686 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:32.964668 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbff2\" (UniqueName: \"kubernetes.io/projected/4812a18c-24c8-487f-a327-7670d532a18c-kube-api-access-wbff2\") pod \"payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h\" (UID: \"4812a18c-24c8-487f-a327-7670d532a18c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h" Nov 02 17:47:33.079255 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:33.079207 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4812a18c-24c8-487f-a327-7670d532a18c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h\" (UID: \"4812a18c-24c8-487f-a327-7670d532a18c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h" Nov 02 17:47:33.079446 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:33.079291 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f7e4cd-9c02-4b92-899b-a5201c88d254-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z\" (UID: \"17f7e4cd-9c02-4b92-899b-a5201c88d254\") " pod="openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z" Nov 02 17:47:33.079446 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:33.079319 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhhd2\" (UniqueName: \"kubernetes.io/projected/17f7e4cd-9c02-4b92-899b-a5201c88d254-kube-api-access-mhhd2\") pod \"payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z\" (UID: \"17f7e4cd-9c02-4b92-899b-a5201c88d254\") " pod="openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z" Nov 02 17:47:33.079611 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:33.079586 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbff2\" (UniqueName: \"kubernetes.io/projected/4812a18c-24c8-487f-a327-7670d532a18c-kube-api-access-wbff2\") pod \"payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h\" (UID: \"4812a18c-24c8-487f-a327-7670d532a18c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h" Nov 02 17:47:33.101629 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:33.081577 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f7e4cd-9c02-4b92-899b-a5201c88d254-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z\" (UID: \"17f7e4cd-9c02-4b92-899b-a5201c88d254\") " pod="openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z" Nov 02 17:47:33.103315 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:33.082782 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:33.105436 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:33.083728 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4812a18c-24c8-487f-a327-7670d532a18c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h\" (UID: \"4812a18c-24c8-487f-a327-7670d532a18c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h" Nov 02 17:47:33.116212 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:33.116123 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbff2\" (UniqueName: \"kubernetes.io/projected/4812a18c-24c8-487f-a327-7670d532a18c-kube-api-access-wbff2\") pod \"payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h\" (UID: \"4812a18c-24c8-487f-a327-7670d532a18c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h" Nov 02 17:47:33.120971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:33.116863 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhhd2\" (UniqueName: \"kubernetes.io/projected/17f7e4cd-9c02-4b92-899b-a5201c88d254-kube-api-access-mhhd2\") pod \"payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z\" (UID: \"17f7e4cd-9c02-4b92-899b-a5201c88d254\") " pod="openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z" Nov 02 17:47:33.198060 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:33.184268 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z" Nov 02 17:47:33.214379 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:33.214348 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h" Nov 02 17:47:33.456988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:33.453685 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:33.492198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:33.488403 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb" Nov 02 17:47:33.530118 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:47:33.530068 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f7e4cd_9c02_4b92_899b_a5201c88d254.slice/crio-d40be8c66bf30b34e71f59038c5a45477af37406483868fed6e3d13d34da44ea WatchSource:0}: Error finding container d40be8c66bf30b34e71f59038c5a45477af37406483868fed6e3d13d34da44ea: Status 404 returned error can't find the container with id d40be8c66bf30b34e71f59038c5a45477af37406483868fed6e3d13d34da44ea Nov 02 17:47:33.538969 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:33.538920 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z"] Nov 02 17:47:33.567198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:33.566500 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h"] Nov 02 17:47:33.567198 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:47:33.567101 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4812a18c_24c8_487f_a327_7670d532a18c.slice/crio-9b30557bbe55e3023efffb860efca5d2e00dc3a4034547cc839b79d056c68fa2 WatchSource:0}: Error finding container 9b30557bbe55e3023efffb860efca5d2e00dc3a4034547cc839b79d056c68fa2: Status 404 returned error can't find the container with id 9b30557bbe55e3023efffb860efca5d2e00dc3a4034547cc839b79d056c68fa2 Nov 02 17:47:33.624208 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:33.624174 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h" event={"ID":"4812a18c-24c8-487f-a327-7670d532a18c","Type":"ContainerStarted","Data":"9b30557bbe55e3023efffb860efca5d2e00dc3a4034547cc839b79d056c68fa2"} Nov 02 17:47:33.626269 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:33.626238 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z" event={"ID":"17f7e4cd-9c02-4b92-899b-a5201c88d254","Type":"ContainerStarted","Data":"d40be8c66bf30b34e71f59038c5a45477af37406483868fed6e3d13d34da44ea"} Nov 02 17:47:34.307975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:34.307459 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:34.590996 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:34.590909 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" Nov 02 17:47:34.637889 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:34.637841 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z" event={"ID":"17f7e4cd-9c02-4b92-899b-a5201c88d254","Type":"ContainerStarted","Data":"31088770e2772069e123584675d38714055bae50973e5a2b60703ed1958aad62"} Nov 02 17:47:34.638367 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:34.638334 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z" Nov 02 17:47:34.640029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:34.640001 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h" event={"ID":"4812a18c-24c8-487f-a327-7670d532a18c","Type":"ContainerStarted","Data":"7f28c669798c563f1609878b8df6e7d87328446e49862648a0107eea86359bdd"} Nov 02 17:47:34.640194 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:34.640178 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h" Nov 02 17:47:34.678626 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:34.678578 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z" podStartSLOduration=2.678563419 podStartE2EDuration="2.678563419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:34.66051331 +0000 UTC m=+826.449018837" watchObservedRunningTime="2026-11-02 17:47:34.678563419 +0000 UTC m=+826.467068946" Nov 02 17:47:34.678791 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:34.678660 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h" podStartSLOduration=2.67865575 podStartE2EDuration="2.67865575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:34.677435097 +0000 UTC m=+826.465940635" watchObservedRunningTime="2026-11-02 17:47:34.67865575 +0000 UTC m=+826.467161277" Nov 02 17:47:34.987135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:34.987099 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2"] Nov 02 17:47:34.999031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:34.999002 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9"] Nov 02 17:47:35.115641 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:35.113811 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:35.269042 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:35.259974 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:36.101221 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:36.101189 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:36.684272 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:36.684233 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:37.028983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:37.028685 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:37.701003 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:37.700862 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:38.313660 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:38.313620 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:39.280359 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:39.280319 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:39.436195 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:39.436143 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:39.988999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:39.988901 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2" podUID="098aa06e-097a-4d85-9309-baab9ced9815" containerName="payload-processing" containerID="cri-o://37bcb8d1556483e4aeaad1e0251c9714a72193fe0e9c54f09250e700f309ea54" gracePeriod=25 Nov 02 17:47:39.999540 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:39.999504 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" podUID="6e48880d-426b-4dfe-9776-339d86fd5577" containerName="payload-pre-processing" containerID="cri-o://68375cd32410ab2271d9bb327ff56def6f421328e41525f4d2082f36295f5a0c" gracePeriod=25 Nov 02 17:47:40.262095 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.261881 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:40.380873 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.380844 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2" Nov 02 17:47:40.403316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.403283 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:40.421593 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.421566 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" Nov 02 17:47:40.490421 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.490383 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e48880d-426b-4dfe-9776-339d86fd5577-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e48880d-426b-4dfe-9776-339d86fd5577-plugins-config-volume\") pod \"6e48880d-426b-4dfe-9776-339d86fd5577\" (UID: \"6e48880d-426b-4dfe-9776-339d86fd5577\") " Nov 02 17:47:40.490616 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.490585 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098aa06e-097a-4d85-9309-baab9ced9815-kube-api-access-wcr9h\" (UniqueName: \"kubernetes.io/projected/098aa06e-097a-4d85-9309-baab9ced9815-kube-api-access-wcr9h\") pod \"098aa06e-097a-4d85-9309-baab9ced9815\" (UID: \"098aa06e-097a-4d85-9309-baab9ced9815\") " Nov 02 17:47:40.491007 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.490980 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e48880d-426b-4dfe-9776-339d86fd5577-kube-api-access-99twd\" (UniqueName: \"kubernetes.io/projected/6e48880d-426b-4dfe-9776-339d86fd5577-kube-api-access-99twd\") pod \"6e48880d-426b-4dfe-9776-339d86fd5577\" (UID: \"6e48880d-426b-4dfe-9776-339d86fd5577\") " Nov 02 17:47:40.491105 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.491009 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e48880d-426b-4dfe-9776-339d86fd5577-plugins-config-volume" pod "6e48880d-426b-4dfe-9776-339d86fd5577" (UID: "6e48880d-426b-4dfe-9776-339d86fd5577"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:47:40.491398 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.491340 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/098aa06e-097a-4d85-9309-baab9ced9815-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098aa06e-097a-4d85-9309-baab9ced9815-plugins-config-volume\") pod \"098aa06e-097a-4d85-9309-baab9ced9815\" (UID: \"098aa06e-097a-4d85-9309-baab9ced9815\") " Nov 02 17:47:40.491845 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.491823 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098aa06e-097a-4d85-9309-baab9ced9815-plugins-config-volume" pod "098aa06e-097a-4d85-9309-baab9ced9815" (UID: "098aa06e-097a-4d85-9309-baab9ced9815"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:47:40.492519 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.492494 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098aa06e-097a-4d85-9309-baab9ced9815-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:47:40.492613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.492528 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e48880d-426b-4dfe-9776-339d86fd5577-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:47:40.494263 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.493772 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098aa06e-097a-4d85-9309-baab9ced9815-kube-api-access-wcr9h" pod "098aa06e-097a-4d85-9309-baab9ced9815" (UID: "098aa06e-097a-4d85-9309-baab9ced9815"). InnerVolumeSpecName "kube-api-access-wcr9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:40.495651 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.495554 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e48880d-426b-4dfe-9776-339d86fd5577-kube-api-access-99twd" pod "6e48880d-426b-4dfe-9776-339d86fd5577" (UID: "6e48880d-426b-4dfe-9776-339d86fd5577"). InnerVolumeSpecName "kube-api-access-99twd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:40.594402 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.594316 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcr9h\" (UniqueName: \"kubernetes.io/projected/098aa06e-097a-4d85-9309-baab9ced9815-kube-api-access-wcr9h\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:47:40.594402 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.594349 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99twd\" (UniqueName: \"kubernetes.io/projected/6e48880d-426b-4dfe-9776-339d86fd5577-kube-api-access-99twd\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:47:40.772206 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.772163 5549 generic.go:356] "Generic (PLEG): container finished" podID="6e48880d-426b-4dfe-9776-339d86fd5577" containerID="68375cd32410ab2271d9bb327ff56def6f421328e41525f4d2082f36295f5a0c" exitCode=0 Nov 02 17:47:40.772400 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.772209 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" Nov 02 17:47:40.772400 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.772242 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" event={"ID":"6e48880d-426b-4dfe-9776-339d86fd5577","Type":"ContainerDied","Data":"68375cd32410ab2271d9bb327ff56def6f421328e41525f4d2082f36295f5a0c"} Nov 02 17:47:40.772400 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.772297 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" event={"ID":"6e48880d-426b-4dfe-9776-339d86fd5577","Type":"ContainerDied","Data":"ab9a02156282ef8cd441de75102a291169940df375d3b811360d4fdbffdfa4e9"} Nov 02 17:47:40.772400 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.772327 5549 scope.go:122] "RemoveContainer" containerID="68375cd32410ab2271d9bb327ff56def6f421328e41525f4d2082f36295f5a0c" Nov 02 17:47:40.774064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.774035 5549 generic.go:356] "Generic (PLEG): container finished" podID="098aa06e-097a-4d85-9309-baab9ced9815" containerID="37bcb8d1556483e4aeaad1e0251c9714a72193fe0e9c54f09250e700f309ea54" exitCode=0 Nov 02 17:47:40.774200 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.774079 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2" event={"ID":"098aa06e-097a-4d85-9309-baab9ced9815","Type":"ContainerDied","Data":"37bcb8d1556483e4aeaad1e0251c9714a72193fe0e9c54f09250e700f309ea54"} Nov 02 17:47:40.774200 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.774127 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2" event={"ID":"098aa06e-097a-4d85-9309-baab9ced9815","Type":"ContainerDied","Data":"e5c520d8b0209a7205898c4ee5a2dbdac11dabfe25beac35b96e7bb6b39099db"} Nov 02 17:47:40.774385 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.774361 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2" Nov 02 17:47:40.785841 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.785820 5549 scope.go:122] "RemoveContainer" containerID="68375cd32410ab2271d9bb327ff56def6f421328e41525f4d2082f36295f5a0c" Nov 02 17:47:40.786162 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:47:40.786142 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68375cd32410ab2271d9bb327ff56def6f421328e41525f4d2082f36295f5a0c\": container with ID starting with 68375cd32410ab2271d9bb327ff56def6f421328e41525f4d2082f36295f5a0c not found: ID does not exist" containerID="68375cd32410ab2271d9bb327ff56def6f421328e41525f4d2082f36295f5a0c" Nov 02 17:47:40.786245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.786171 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68375cd32410ab2271d9bb327ff56def6f421328e41525f4d2082f36295f5a0c"} err="failed to get container status \"68375cd32410ab2271d9bb327ff56def6f421328e41525f4d2082f36295f5a0c\": rpc error: code = NotFound desc = could not find container \"68375cd32410ab2271d9bb327ff56def6f421328e41525f4d2082f36295f5a0c\": container with ID starting with 68375cd32410ab2271d9bb327ff56def6f421328e41525f4d2082f36295f5a0c not found: ID does not exist" Nov 02 17:47:40.786245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.786194 5549 scope.go:122] "RemoveContainer" containerID="37bcb8d1556483e4aeaad1e0251c9714a72193fe0e9c54f09250e700f309ea54" Nov 02 17:47:40.797350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.797237 5549 scope.go:122] "RemoveContainer" containerID="37bcb8d1556483e4aeaad1e0251c9714a72193fe0e9c54f09250e700f309ea54" Nov 02 17:47:40.797813 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:47:40.797777 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37bcb8d1556483e4aeaad1e0251c9714a72193fe0e9c54f09250e700f309ea54\": container with ID starting with 37bcb8d1556483e4aeaad1e0251c9714a72193fe0e9c54f09250e700f309ea54 not found: ID does not exist" containerID="37bcb8d1556483e4aeaad1e0251c9714a72193fe0e9c54f09250e700f309ea54" Nov 02 17:47:40.797877 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.797827 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37bcb8d1556483e4aeaad1e0251c9714a72193fe0e9c54f09250e700f309ea54"} err="failed to get container status \"37bcb8d1556483e4aeaad1e0251c9714a72193fe0e9c54f09250e700f309ea54\": rpc error: code = NotFound desc = could not find container \"37bcb8d1556483e4aeaad1e0251c9714a72193fe0e9c54f09250e700f309ea54\": container with ID starting with 37bcb8d1556483e4aeaad1e0251c9714a72193fe0e9c54f09250e700f309ea54 not found: ID does not exist" Nov 02 17:47:40.799171 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.799151 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9"] Nov 02 17:47:40.802700 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.802678 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9"] Nov 02 17:47:40.813350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.813327 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2"] Nov 02 17:47:40.816303 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.816281 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2893d1-65f66fb94f-mfpb2"] Nov 02 17:47:40.934036 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.933944 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb"] Nov 02 17:47:40.935065 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.935043 5549 cpu_manager.go:414] "containerMap: removing container" podUID="098aa06e-097a-4d85-9309-baab9ced9815" containerName="payload-processing" Nov 02 17:47:40.935171 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.935065 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098aa06e-097a-4d85-9309-baab9ced9815" containerName="payload-processing" Nov 02 17:47:40.935171 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.935113 5549 cpu_manager.go:414] "containerMap: removing container" podUID="6e48880d-426b-4dfe-9776-339d86fd5577" containerName="payload-pre-processing" Nov 02 17:47:40.935171 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.935123 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e48880d-426b-4dfe-9776-339d86fd5577" containerName="payload-pre-processing" Nov 02 17:47:40.935282 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.935273 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e48880d-426b-4dfe-9776-339d86fd5577" containerName="payload-pre-processing" Nov 02 17:47:40.935316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.935297 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="098aa06e-097a-4d85-9309-baab9ced9815" containerName="payload-processing" Nov 02 17:47:40.941307 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.941283 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:47:40.948994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.948232 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a976bc-serving-cert\"" Nov 02 17:47:40.953372 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:40.953304 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb"] Nov 02 17:47:41.001202 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:41.001171 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8277c127-0ecb-4839-b899-5145144dafa9-metrics-certs\") pod \"maas-api-e2e-api-a-a976bc-84b755767-9bzcb\" (UID: \"8277c127-0ecb-4839-b899-5145144dafa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:47:41.001369 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:41.001326 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8277c127-0ecb-4839-b899-5145144dafa9-openshift-service-ca\") pod \"maas-api-e2e-api-a-a976bc-84b755767-9bzcb\" (UID: \"8277c127-0ecb-4839-b899-5145144dafa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:47:41.001427 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:41.001378 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhfxk\" (UniqueName: \"kubernetes.io/projected/8277c127-0ecb-4839-b899-5145144dafa9-kube-api-access-zhfxk\") pod \"maas-api-e2e-api-a-a976bc-84b755767-9bzcb\" (UID: \"8277c127-0ecb-4839-b899-5145144dafa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:47:41.001501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:41.001490 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8277c127-0ecb-4839-b899-5145144dafa9-maas-api-tls\") pod \"maas-api-e2e-api-a-a976bc-84b755767-9bzcb\" (UID: \"8277c127-0ecb-4839-b899-5145144dafa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:47:41.039201 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:41.039167 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:41.107537 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:41.105931 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8277c127-0ecb-4839-b899-5145144dafa9-maas-api-tls\") pod \"maas-api-e2e-api-a-a976bc-84b755767-9bzcb\" (UID: \"8277c127-0ecb-4839-b899-5145144dafa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:47:41.107537 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:41.106069 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8277c127-0ecb-4839-b899-5145144dafa9-metrics-certs\") pod \"maas-api-e2e-api-a-a976bc-84b755767-9bzcb\" (UID: \"8277c127-0ecb-4839-b899-5145144dafa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:47:41.107537 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:47:41.106191 5549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-serving-cert: secret "maas-api-e2e-api-a-a976bc-serving-cert" not found Nov 02 17:47:41.107537 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:41.106253 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8277c127-0ecb-4839-b899-5145144dafa9-openshift-service-ca\") pod \"maas-api-e2e-api-a-a976bc-84b755767-9bzcb\" (UID: \"8277c127-0ecb-4839-b899-5145144dafa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:47:41.107537 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:47:41.106273 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8277c127-0ecb-4839-b899-5145144dafa9-maas-api-tls podName:8277c127-0ecb-4839-b899-5145144dafa9 nodeName:}" failed. No retries permitted until 2026-11-02 17:47:41.606250189 +0000 UTC m=+833.394755713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8277c127-0ecb-4839-b899-5145144dafa9-maas-api-tls") pod "maas-api-e2e-api-a-a976bc-84b755767-9bzcb" (UID: "8277c127-0ecb-4839-b899-5145144dafa9") : secret "maas-api-e2e-api-a-a976bc-serving-cert" not found Nov 02 17:47:41.107537 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:41.106367 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhfxk\" (UniqueName: \"kubernetes.io/projected/8277c127-0ecb-4839-b899-5145144dafa9-kube-api-access-zhfxk\") pod \"maas-api-e2e-api-a-a976bc-84b755767-9bzcb\" (UID: \"8277c127-0ecb-4839-b899-5145144dafa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:47:41.107537 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:41.107024 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8277c127-0ecb-4839-b899-5145144dafa9-openshift-service-ca\") pod \"maas-api-e2e-api-a-a976bc-84b755767-9bzcb\" (UID: \"8277c127-0ecb-4839-b899-5145144dafa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:47:41.121973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:41.117691 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8277c127-0ecb-4839-b899-5145144dafa9-metrics-certs\") pod \"maas-api-e2e-api-a-a976bc-84b755767-9bzcb\" (UID: \"8277c127-0ecb-4839-b899-5145144dafa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:47:41.137974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:41.137826 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhfxk\" (UniqueName: \"kubernetes.io/projected/8277c127-0ecb-4839-b899-5145144dafa9-kube-api-access-zhfxk\") pod \"maas-api-e2e-api-a-a976bc-84b755767-9bzcb\" (UID: \"8277c127-0ecb-4839-b899-5145144dafa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:47:41.180297 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:41.180210 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:41.376133 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:41.376093 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2893d1-6f96d89487-h4hg9" podUID="6e48880d-426b-4dfe-9776-339d86fd5577" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.83:50052\" within 1s: context deadline exceeded" Nov 02 17:47:41.376326 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:47:41.376150 5549 logging.go:55] [core] [Channel #1292 SubChannel #1293]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.83:50052", ServerName: "10.133.0.83:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.83:50052: operation was canceled" Nov 02 17:47:41.614370 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:41.614334 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8277c127-0ecb-4839-b899-5145144dafa9-maas-api-tls\") pod \"maas-api-e2e-api-a-a976bc-84b755767-9bzcb\" (UID: \"8277c127-0ecb-4839-b899-5145144dafa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:47:41.617645 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:41.617606 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8277c127-0ecb-4839-b899-5145144dafa9-maas-api-tls\") pod \"maas-api-e2e-api-a-a976bc-84b755767-9bzcb\" (UID: \"8277c127-0ecb-4839-b899-5145144dafa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:47:41.862381 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:41.862344 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:47:42.144687 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:47:42.144657 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8277c127_0ecb_4839_b899_5145144dafa9.slice/crio-f3affda9a4655f389968d622279bad930bc5838daafccd946910fc7c16975a67 WatchSource:0}: Error finding container f3affda9a4655f389968d622279bad930bc5838daafccd946910fc7c16975a67: Status 404 returned error can't find the container with id f3affda9a4655f389968d622279bad930bc5838daafccd946910fc7c16975a67 Nov 02 17:47:42.144841 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:42.144814 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb"] Nov 02 17:47:42.320708 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:42.320675 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="098aa06e-097a-4d85-9309-baab9ced9815" path="/var/lib/kubelet/pods/098aa06e-097a-4d85-9309-baab9ced9815/volumes" Nov 02 17:47:42.321161 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:42.321145 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e48880d-426b-4dfe-9776-339d86fd5577" path="/var/lib/kubelet/pods/6e48880d-426b-4dfe-9776-339d86fd5577/volumes" Nov 02 17:47:42.857053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:42.853176 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" event={"ID":"8277c127-0ecb-4839-b899-5145144dafa9","Type":"ContainerStarted","Data":"114912530c2efeb31cdd9d68dc641b12c9ecb49ac6d2f5e4a539db3ba5aa93f9"} Nov 02 17:47:42.857053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:42.853243 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" event={"ID":"8277c127-0ecb-4839-b899-5145144dafa9","Type":"ContainerStarted","Data":"f3affda9a4655f389968d622279bad930bc5838daafccd946910fc7c16975a67"} Nov 02 17:47:42.857053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:42.853553 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:47:42.882307 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:42.882254 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" podStartSLOduration=2.453986148 podStartE2EDuration="2.882240844s" totalImagesPullingTime="428.254696ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:42.879717004 +0000 UTC m=+834.668222564" watchObservedRunningTime="2026-11-02 17:47:42.882240844 +0000 UTC m=+834.670746366" Nov 02 17:47:45.704985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:45.704214 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h" Nov 02 17:47:46.660984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:46.658861 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z" Nov 02 17:47:46.712987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:46.710339 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:47.137593 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:47.137558 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:47.349144 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:47.349103 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd"] Nov 02 17:47:47.533971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:47.533910 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:47.761868 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:47.761817 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:47.820708 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:47:47.820594 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:47:48.021979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:48.021079 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-fl8ch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:47:48.021979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:48.021163 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" podUID="b9b1010f-8c24-44d2-8495-f11e545f2187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:48.523483 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:48.523446 5549 scope.go:122] "RemoveContainer" containerID="ae699152a1d7ee8c7e723c14745e15b33ed8cc4befce4608e21d98334fd8e131" Nov 02 17:47:48.846047 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:48.843477 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:48.896275 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:48.896232 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:47:49.622710 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:49.622672 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:50.305781 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:50.305741 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:50.621057 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:50.620976 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:50.820728 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:50.820689 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:50.825353 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:50.825308 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764"] Nov 02 17:47:50.825811 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:50.825738 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" podUID="bcc3422f-a7c8-4eae-9dba-a63aae2ae613" containerName="maas-api" containerID="cri-o://094667f672742c5735d1de461a637287a71f608e4b01880da1617276cbde6a6a" gracePeriod=30 Nov 02 17:47:50.998997 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:50.971377 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:51.131080 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:51.131044 5549 generic.go:356] "Generic (PLEG): container finished" podID="bcc3422f-a7c8-4eae-9dba-a63aae2ae613" containerID="094667f672742c5735d1de461a637287a71f608e4b01880da1617276cbde6a6a" exitCode=0 Nov 02 17:47:51.131491 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:51.131467 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" event={"ID":"bcc3422f-a7c8-4eae-9dba-a63aae2ae613","Type":"ContainerDied","Data":"094667f672742c5735d1de461a637287a71f608e4b01880da1617276cbde6a6a"} Nov 02 17:47:51.301278 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:51.301248 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:51.419721 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:51.418978 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" Nov 02 17:47:51.443048 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:51.443011 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-openshift-service-ca\") pod \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\" (UID: \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\") " Nov 02 17:47:51.443275 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:51.443140 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-kube-api-access-7hskd\" (UniqueName: \"kubernetes.io/projected/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-kube-api-access-7hskd\") pod \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\" (UID: \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\") " Nov 02 17:47:51.443275 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:51.443185 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:51.443365 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:51.443339 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-maas-api-tls\") pod \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\" (UID: \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\") " Nov 02 17:47:51.443402 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:51.443380 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-openshift-service-ca" pod "bcc3422f-a7c8-4eae-9dba-a63aae2ae613" (UID: "bcc3422f-a7c8-4eae-9dba-a63aae2ae613"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:47:51.443437 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:51.443406 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-metrics-certs\") pod \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\" (UID: \"bcc3422f-a7c8-4eae-9dba-a63aae2ae613\") " Nov 02 17:47:51.444082 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:51.444056 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:47:51.446043 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:51.445938 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-kube-api-access-7hskd" pod "bcc3422f-a7c8-4eae-9dba-a63aae2ae613" (UID: "bcc3422f-a7c8-4eae-9dba-a63aae2ae613"). InnerVolumeSpecName "kube-api-access-7hskd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:51.446043 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:51.446041 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-maas-api-tls" pod "bcc3422f-a7c8-4eae-9dba-a63aae2ae613" (UID: "bcc3422f-a7c8-4eae-9dba-a63aae2ae613"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:47:51.446913 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:51.446884 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-metrics-certs" pod "bcc3422f-a7c8-4eae-9dba-a63aae2ae613" (UID: "bcc3422f-a7c8-4eae-9dba-a63aae2ae613"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:47:51.545160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:51.545084 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hskd\" (UniqueName: \"kubernetes.io/projected/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-kube-api-access-7hskd\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:47:51.545160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:51.545117 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:47:51.545160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:51.545130 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc3422f-a7c8-4eae-9dba-a63aae2ae613-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:47:51.730581 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:51.730546 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:52.222976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:52.212011 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" event={"ID":"bcc3422f-a7c8-4eae-9dba-a63aae2ae613","Type":"ContainerDied","Data":"68441b36b272ca2544eeacde8f58dd870b9b52c9a5c9464d29e742493267930f"} Nov 02 17:47:52.222976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:52.212086 5549 scope.go:122] "RemoveContainer" containerID="094667f672742c5735d1de461a637287a71f608e4b01880da1617276cbde6a6a" Nov 02 17:47:52.222976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:52.212251 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764" Nov 02 17:47:52.354213 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:52.354172 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd" podUID="204f34df-a8ac-4f90-89c9-93af7a9cf479" containerName="payload-processing" containerID="cri-o://27c6f85516947e839af872ac4d3ac1d6f0a49836c72761fa2f2a90259b068887" gracePeriod=25 Nov 02 17:47:52.372913 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:52.372881 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764"] Nov 02 17:47:52.381576 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:52.378254 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-58970f-79496696b4-rf764"] Nov 02 17:47:52.387379 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:52.387345 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:52.589355 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:52.589183 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:52.650752 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:52.650719 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd" Nov 02 17:47:52.730606 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:52.730573 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204f34df-a8ac-4f90-89c9-93af7a9cf479-kube-api-access-j44ml\" (UniqueName: \"kubernetes.io/projected/204f34df-a8ac-4f90-89c9-93af7a9cf479-kube-api-access-j44ml\") pod \"204f34df-a8ac-4f90-89c9-93af7a9cf479\" (UID: \"204f34df-a8ac-4f90-89c9-93af7a9cf479\") " Nov 02 17:47:52.730781 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:52.730724 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204f34df-a8ac-4f90-89c9-93af7a9cf479-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/204f34df-a8ac-4f90-89c9-93af7a9cf479-plugins-config-volume\") pod \"204f34df-a8ac-4f90-89c9-93af7a9cf479\" (UID: \"204f34df-a8ac-4f90-89c9-93af7a9cf479\") " Nov 02 17:47:52.731085 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:52.731064 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204f34df-a8ac-4f90-89c9-93af7a9cf479-plugins-config-volume" pod "204f34df-a8ac-4f90-89c9-93af7a9cf479" (UID: "204f34df-a8ac-4f90-89c9-93af7a9cf479"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:47:52.731268 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:52.731254 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/204f34df-a8ac-4f90-89c9-93af7a9cf479-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:47:52.732682 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:52.732663 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204f34df-a8ac-4f90-89c9-93af7a9cf479-kube-api-access-j44ml" pod "204f34df-a8ac-4f90-89c9-93af7a9cf479" (UID: "204f34df-a8ac-4f90-89c9-93af7a9cf479"). InnerVolumeSpecName "kube-api-access-j44ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:52.832460 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:52.832427 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j44ml\" (UniqueName: \"kubernetes.io/projected/204f34df-a8ac-4f90-89c9-93af7a9cf479-kube-api-access-j44ml\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:47:53.146021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:53.143117 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:53.221236 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:53.221198 5549 generic.go:356] "Generic (PLEG): container finished" podID="204f34df-a8ac-4f90-89c9-93af7a9cf479" containerID="27c6f85516947e839af872ac4d3ac1d6f0a49836c72761fa2f2a90259b068887" exitCode=0 Nov 02 17:47:53.221382 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:53.221255 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd" event={"ID":"204f34df-a8ac-4f90-89c9-93af7a9cf479","Type":"ContainerDied","Data":"27c6f85516947e839af872ac4d3ac1d6f0a49836c72761fa2f2a90259b068887"} Nov 02 17:47:53.221382 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:53.221340 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd" event={"ID":"204f34df-a8ac-4f90-89c9-93af7a9cf479","Type":"ContainerDied","Data":"e46b8151de02a1301c5647538ba6305141f3714cd95431b53053fceabb56f371"} Nov 02 17:47:53.221382 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:53.221264 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd" Nov 02 17:47:53.221382 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:53.221364 5549 scope.go:122] "RemoveContainer" containerID="27c6f85516947e839af872ac4d3ac1d6f0a49836c72761fa2f2a90259b068887" Nov 02 17:47:53.232009 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:53.231987 5549 scope.go:122] "RemoveContainer" containerID="27c6f85516947e839af872ac4d3ac1d6f0a49836c72761fa2f2a90259b068887" Nov 02 17:47:53.232292 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:47:53.232271 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c6f85516947e839af872ac4d3ac1d6f0a49836c72761fa2f2a90259b068887\": container with ID starting with 27c6f85516947e839af872ac4d3ac1d6f0a49836c72761fa2f2a90259b068887 not found: ID does not exist" containerID="27c6f85516947e839af872ac4d3ac1d6f0a49836c72761fa2f2a90259b068887" Nov 02 17:47:53.232340 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:53.232302 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c6f85516947e839af872ac4d3ac1d6f0a49836c72761fa2f2a90259b068887"} err="failed to get container status \"27c6f85516947e839af872ac4d3ac1d6f0a49836c72761fa2f2a90259b068887\": rpc error: code = NotFound desc = could not find container \"27c6f85516947e839af872ac4d3ac1d6f0a49836c72761fa2f2a90259b068887\": container with ID starting with 27c6f85516947e839af872ac4d3ac1d6f0a49836c72761fa2f2a90259b068887 not found: ID does not exist" Nov 02 17:47:53.250106 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:53.250085 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd"] Nov 02 17:47:53.253135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:53.253117 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-58970f-5d9f55f84f-vxdmd"] Nov 02 17:47:53.829786 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:53.829503 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:53.973465 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:53.973435 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:54.321081 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:54.321051 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204f34df-a8ac-4f90-89c9-93af7a9cf479" path="/var/lib/kubelet/pods/204f34df-a8ac-4f90-89c9-93af7a9cf479/volumes" Nov 02 17:47:54.321531 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:54.321517 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcc3422f-a7c8-4eae-9dba-a63aae2ae613" path="/var/lib/kubelet/pods/bcc3422f-a7c8-4eae-9dba-a63aae2ae613/volumes" Nov 02 17:47:54.553444 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:54.553389 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:54.699032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:54.698996 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:54.910563 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:54.910529 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:55.545564 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:55.545527 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:55.706839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:55.706769 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:56.264061 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:56.264029 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:56.785106 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:56.785068 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:56.937981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:56.936942 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:57.384565 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:57.384529 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:57.665028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:57.664476 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:57.850975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:57.850920 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:57.993694 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:57.993662 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:58.137800 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:58.137763 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:58.521570 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:58.521530 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:58.734706 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:58.734674 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5"] Nov 02 17:47:58.856497 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:58.856379 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:59.009414 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:59.009380 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:59.539783 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:59.539542 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:59.676909 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:47:59.676868 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:00.059268 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.059228 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz"] Nov 02 17:48:00.059600 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.059566 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" podUID="72847c27-b3ec-49d0-9af8-b015051608c1" containerName="maas-api" containerID="cri-o://3d7f1aea001b4ea22197a476f1ac916fae140103f3118709dbed4cf405584d68" gracePeriod=30 Nov 02 17:48:00.145262 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.145230 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:00.522011 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.516134 5549 generic.go:356] "Generic (PLEG): container finished" podID="72847c27-b3ec-49d0-9af8-b015051608c1" containerID="3d7f1aea001b4ea22197a476f1ac916fae140103f3118709dbed4cf405584d68" exitCode=0 Nov 02 17:48:00.522011 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.516202 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" event={"ID":"72847c27-b3ec-49d0-9af8-b015051608c1","Type":"ContainerDied","Data":"3d7f1aea001b4ea22197a476f1ac916fae140103f3118709dbed4cf405584d68"} Nov 02 17:48:00.535084 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.535048 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:00.596392 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.596364 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:48:00.738395 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.738366 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72847c27-b3ec-49d0-9af8-b015051608c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72847c27-b3ec-49d0-9af8-b015051608c1-maas-api-tls\") pod \"72847c27-b3ec-49d0-9af8-b015051608c1\" (UID: \"72847c27-b3ec-49d0-9af8-b015051608c1\") " Nov 02 17:48:00.738546 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.738440 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72847c27-b3ec-49d0-9af8-b015051608c1-kube-api-access-zw8c8\" (UniqueName: \"kubernetes.io/projected/72847c27-b3ec-49d0-9af8-b015051608c1-kube-api-access-zw8c8\") pod \"72847c27-b3ec-49d0-9af8-b015051608c1\" (UID: \"72847c27-b3ec-49d0-9af8-b015051608c1\") " Nov 02 17:48:00.738546 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.738537 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72847c27-b3ec-49d0-9af8-b015051608c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72847c27-b3ec-49d0-9af8-b015051608c1-openshift-service-ca\") pod \"72847c27-b3ec-49d0-9af8-b015051608c1\" (UID: \"72847c27-b3ec-49d0-9af8-b015051608c1\") " Nov 02 17:48:00.738851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.738829 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72847c27-b3ec-49d0-9af8-b015051608c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72847c27-b3ec-49d0-9af8-b015051608c1-metrics-certs\") pod \"72847c27-b3ec-49d0-9af8-b015051608c1\" (UID: \"72847c27-b3ec-49d0-9af8-b015051608c1\") " Nov 02 17:48:00.738997 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.738855 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72847c27-b3ec-49d0-9af8-b015051608c1-openshift-service-ca" pod "72847c27-b3ec-49d0-9af8-b015051608c1" (UID: "72847c27-b3ec-49d0-9af8-b015051608c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:00.739393 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.739374 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72847c27-b3ec-49d0-9af8-b015051608c1-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:00.740594 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.740573 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72847c27-b3ec-49d0-9af8-b015051608c1-kube-api-access-zw8c8" pod "72847c27-b3ec-49d0-9af8-b015051608c1" (UID: "72847c27-b3ec-49d0-9af8-b015051608c1"). InnerVolumeSpecName "kube-api-access-zw8c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:00.740698 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.740679 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72847c27-b3ec-49d0-9af8-b015051608c1-metrics-certs" pod "72847c27-b3ec-49d0-9af8-b015051608c1" (UID: "72847c27-b3ec-49d0-9af8-b015051608c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:48:00.740788 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.740773 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72847c27-b3ec-49d0-9af8-b015051608c1-maas-api-tls" pod "72847c27-b3ec-49d0-9af8-b015051608c1" (UID: "72847c27-b3ec-49d0-9af8-b015051608c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:48:00.842240 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.842141 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72847c27-b3ec-49d0-9af8-b015051608c1-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:00.842240 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.842188 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72847c27-b3ec-49d0-9af8-b015051608c1-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:00.842240 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:00.842205 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw8c8\" (UniqueName: \"kubernetes.io/projected/72847c27-b3ec-49d0-9af8-b015051608c1-kube-api-access-zw8c8\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:01.341970 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:01.341923 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb"] Nov 02 17:48:01.357012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:01.356974 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4"] Nov 02 17:48:01.455886 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:01.455842 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:01.524675 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:01.524644 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" Nov 02 17:48:01.524675 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:01.524650 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" event={"ID":"72847c27-b3ec-49d0-9af8-b015051608c1","Type":"ContainerDied","Data":"f7cf8d1aeada8fbbb4de06a4cf138d133a9136bdb5e05386c6f024f268ee58cc"} Nov 02 17:48:01.524879 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:01.524706 5549 scope.go:122] "RemoveContainer" containerID="3d7f1aea001b4ea22197a476f1ac916fae140103f3118709dbed4cf405584d68" Nov 02 17:48:01.548691 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:01.548665 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz"] Nov 02 17:48:01.553110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:01.553084 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz"] Nov 02 17:48:01.608031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:01.603311 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:01.742804 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:01.740557 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:02.321072 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:02.321041 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72847c27-b3ec-49d0-9af8-b015051608c1" path="/var/lib/kubelet/pods/72847c27-b3ec-49d0-9af8-b015051608c1/volumes" Nov 02 17:48:02.376150 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:02.376121 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:02.745031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:02.725445 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:03.009290 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.009187 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:03.045507 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.045474 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk"] Nov 02 17:48:03.046188 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.046174 5549 cpu_manager.go:414] "containerMap: removing container" podUID="bcc3422f-a7c8-4eae-9dba-a63aae2ae613" containerName="maas-api" Nov 02 17:48:03.046251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.046189 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc3422f-a7c8-4eae-9dba-a63aae2ae613" containerName="maas-api" Nov 02 17:48:03.046251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.046214 5549 cpu_manager.go:414] "containerMap: removing container" podUID="72847c27-b3ec-49d0-9af8-b015051608c1" containerName="maas-api" Nov 02 17:48:03.046251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.046220 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72847c27-b3ec-49d0-9af8-b015051608c1" containerName="maas-api" Nov 02 17:48:03.046251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.046248 5549 cpu_manager.go:414] "containerMap: removing container" podUID="204f34df-a8ac-4f90-89c9-93af7a9cf479" containerName="payload-processing" Nov 02 17:48:03.046398 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.046253 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204f34df-a8ac-4f90-89c9-93af7a9cf479" containerName="payload-processing" Nov 02 17:48:03.046398 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.046336 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="204f34df-a8ac-4f90-89c9-93af7a9cf479" containerName="payload-processing" Nov 02 17:48:03.046398 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.046348 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc3422f-a7c8-4eae-9dba-a63aae2ae613" containerName="maas-api" Nov 02 17:48:03.046398 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.046356 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="72847c27-b3ec-49d0-9af8-b015051608c1" containerName="maas-api" Nov 02 17:48:03.058598 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.058567 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk" Nov 02 17:48:03.061459 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.061432 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk"] Nov 02 17:48:03.063880 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.063857 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-11f0a3-dockercfg-zzzvj\"" Nov 02 17:48:03.063880 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.063858 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-11f0a3\"" Nov 02 17:48:03.085513 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.085482 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj"] Nov 02 17:48:03.090905 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.090872 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj" Nov 02 17:48:03.092083 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.092055 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj"] Nov 02 17:48:03.171982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.171113 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:03.224016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.209937 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b826c10-7d3c-45e0-ae91-b3c47bb14b33-plugins-config-volume\") pod \"payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj\" (UID: \"6b826c10-7d3c-45e0-ae91-b3c47bb14b33\") " pod="openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj" Nov 02 17:48:03.229098 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.229059 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxpq7\" (UniqueName: \"kubernetes.io/projected/6b826c10-7d3c-45e0-ae91-b3c47bb14b33-kube-api-access-vxpq7\") pod \"payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj\" (UID: \"6b826c10-7d3c-45e0-ae91-b3c47bb14b33\") " pod="openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj" Nov 02 17:48:03.229541 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.229514 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvfdf\" (UniqueName: \"kubernetes.io/projected/bb09e15a-379c-42d4-85cf-a70b727dfcb4-kube-api-access-tvfdf\") pod \"payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk\" (UID: \"bb09e15a-379c-42d4-85cf-a70b727dfcb4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk" Nov 02 17:48:03.229874 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.229855 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb09e15a-379c-42d4-85cf-a70b727dfcb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk\" (UID: \"bb09e15a-379c-42d4-85cf-a70b727dfcb4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk" Nov 02 17:48:03.337025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.335851 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvfdf\" (UniqueName: \"kubernetes.io/projected/bb09e15a-379c-42d4-85cf-a70b727dfcb4-kube-api-access-tvfdf\") pod \"payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk\" (UID: \"bb09e15a-379c-42d4-85cf-a70b727dfcb4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk" Nov 02 17:48:03.337025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.336098 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb09e15a-379c-42d4-85cf-a70b727dfcb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk\" (UID: \"bb09e15a-379c-42d4-85cf-a70b727dfcb4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk" Nov 02 17:48:03.337025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.336197 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b826c10-7d3c-45e0-ae91-b3c47bb14b33-plugins-config-volume\") pod \"payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj\" (UID: \"6b826c10-7d3c-45e0-ae91-b3c47bb14b33\") " pod="openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj" Nov 02 17:48:03.337025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.336232 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxpq7\" (UniqueName: \"kubernetes.io/projected/6b826c10-7d3c-45e0-ae91-b3c47bb14b33-kube-api-access-vxpq7\") pod \"payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj\" (UID: \"6b826c10-7d3c-45e0-ae91-b3c47bb14b33\") " pod="openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj" Nov 02 17:48:03.337784 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.337741 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb09e15a-379c-42d4-85cf-a70b727dfcb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk\" (UID: \"bb09e15a-379c-42d4-85cf-a70b727dfcb4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk" Nov 02 17:48:03.348969 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.338767 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b826c10-7d3c-45e0-ae91-b3c47bb14b33-plugins-config-volume\") pod \"payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj\" (UID: \"6b826c10-7d3c-45e0-ae91-b3c47bb14b33\") " pod="openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj" Nov 02 17:48:03.373168 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.363667 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxpq7\" (UniqueName: \"kubernetes.io/projected/6b826c10-7d3c-45e0-ae91-b3c47bb14b33-kube-api-access-vxpq7\") pod \"payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj\" (UID: \"6b826c10-7d3c-45e0-ae91-b3c47bb14b33\") " pod="openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj" Nov 02 17:48:03.386005 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.374472 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvfdf\" (UniqueName: \"kubernetes.io/projected/bb09e15a-379c-42d4-85cf-a70b727dfcb4-kube-api-access-tvfdf\") pod \"payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk\" (UID: \"bb09e15a-379c-42d4-85cf-a70b727dfcb4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk" Nov 02 17:48:03.408913 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.408871 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj" Nov 02 17:48:03.411866 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.411821 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-935086-8477ddd99f-c7rtz" podUID="72847c27-b3ec-49d0-9af8-b015051608c1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.93:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:48:03.685063 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.681676 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk" Nov 02 17:48:03.688637 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.688458 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:03.711537 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.711504 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj"] Nov 02 17:48:03.712255 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:48:03.712229 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b826c10_7d3c_45e0_ae91_b3c47bb14b33.slice/crio-7e5a4755dde82fc844fd554c8f12b15be2371cbe22ae9c4c1ccd01e1c4bfb800 WatchSource:0}: Error finding container 7e5a4755dde82fc844fd554c8f12b15be2371cbe22ae9c4c1ccd01e1c4bfb800: Status 404 returned error can't find the container with id 7e5a4755dde82fc844fd554c8f12b15be2371cbe22ae9c4c1ccd01e1c4bfb800 Nov 02 17:48:03.735999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.735924 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5" podUID="253360ac-a72b-426e-8b02-77a9190e3962" containerName="payload-processing" containerID="cri-o://9168cddae3018cf39b674d263625049662ee01cf1e9cc416fd73397cd4799b80" gracePeriod=25 Nov 02 17:48:03.974808 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:48:03.974776 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb09e15a_379c_42d4_85cf_a70b727dfcb4.slice/crio-ede8404e5dcb53741fd76957645ba87b32f57fe7575865521a5993c77ddffb35 WatchSource:0}: Error finding container ede8404e5dcb53741fd76957645ba87b32f57fe7575865521a5993c77ddffb35: Status 404 returned error can't find the container with id ede8404e5dcb53741fd76957645ba87b32f57fe7575865521a5993c77ddffb35 Nov 02 17:48:03.975968 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:03.975920 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk"] Nov 02 17:48:04.001436 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.001414 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5" Nov 02 17:48:04.154065 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.154032 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/253360ac-a72b-426e-8b02-77a9190e3962-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/253360ac-a72b-426e-8b02-77a9190e3962-plugins-config-volume\") pod \"253360ac-a72b-426e-8b02-77a9190e3962\" (UID: \"253360ac-a72b-426e-8b02-77a9190e3962\") " Nov 02 17:48:04.154248 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.154167 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253360ac-a72b-426e-8b02-77a9190e3962-kube-api-access-fc76g\" (UniqueName: \"kubernetes.io/projected/253360ac-a72b-426e-8b02-77a9190e3962-kube-api-access-fc76g\") pod \"253360ac-a72b-426e-8b02-77a9190e3962\" (UID: \"253360ac-a72b-426e-8b02-77a9190e3962\") " Nov 02 17:48:04.154414 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.154391 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253360ac-a72b-426e-8b02-77a9190e3962-plugins-config-volume" pod "253360ac-a72b-426e-8b02-77a9190e3962" (UID: "253360ac-a72b-426e-8b02-77a9190e3962"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:04.154713 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.154694 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/253360ac-a72b-426e-8b02-77a9190e3962-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:04.156306 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.156282 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253360ac-a72b-426e-8b02-77a9190e3962-kube-api-access-fc76g" pod "253360ac-a72b-426e-8b02-77a9190e3962" (UID: "253360ac-a72b-426e-8b02-77a9190e3962"). InnerVolumeSpecName "kube-api-access-fc76g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:04.255714 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.255631 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc76g\" (UniqueName: \"kubernetes.io/projected/253360ac-a72b-426e-8b02-77a9190e3962-kube-api-access-fc76g\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:04.435993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.434530 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:04.553089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.552995 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk" event={"ID":"bb09e15a-379c-42d4-85cf-a70b727dfcb4","Type":"ContainerStarted","Data":"f60be58566207744d8eb705a58e696943477eb26c95bb7993fa4c12a06dfb8cc"} Nov 02 17:48:04.553089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.553058 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk" event={"ID":"bb09e15a-379c-42d4-85cf-a70b727dfcb4","Type":"ContainerStarted","Data":"ede8404e5dcb53741fd76957645ba87b32f57fe7575865521a5993c77ddffb35"} Nov 02 17:48:04.553313 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.553252 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk" Nov 02 17:48:04.554413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.554389 5549 generic.go:356] "Generic (PLEG): container finished" podID="253360ac-a72b-426e-8b02-77a9190e3962" containerID="9168cddae3018cf39b674d263625049662ee01cf1e9cc416fd73397cd4799b80" exitCode=0 Nov 02 17:48:04.554547 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.554426 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5" Nov 02 17:48:04.554547 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.554469 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5" event={"ID":"253360ac-a72b-426e-8b02-77a9190e3962","Type":"ContainerDied","Data":"9168cddae3018cf39b674d263625049662ee01cf1e9cc416fd73397cd4799b80"} Nov 02 17:48:04.554547 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.554516 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5" event={"ID":"253360ac-a72b-426e-8b02-77a9190e3962","Type":"ContainerDied","Data":"155482c881bf24236bb1c1d1c8fe44b4743c06167c0993cfe8eacc1564856310"} Nov 02 17:48:04.554547 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.554544 5549 scope.go:122] "RemoveContainer" containerID="9168cddae3018cf39b674d263625049662ee01cf1e9cc416fd73397cd4799b80" Nov 02 17:48:04.556145 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.556123 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj" event={"ID":"6b826c10-7d3c-45e0-ae91-b3c47bb14b33","Type":"ContainerStarted","Data":"0c85cadfa580454e577c2dfc85182b4f21afacdd4b0affdbe2aedcbb025724b7"} Nov 02 17:48:04.556228 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.556163 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj" event={"ID":"6b826c10-7d3c-45e0-ae91-b3c47bb14b33","Type":"ContainerStarted","Data":"7e5a4755dde82fc844fd554c8f12b15be2371cbe22ae9c4c1ccd01e1c4bfb800"} Nov 02 17:48:04.556228 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.556199 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj" Nov 02 17:48:04.565201 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.565181 5549 scope.go:122] "RemoveContainer" containerID="9168cddae3018cf39b674d263625049662ee01cf1e9cc416fd73397cd4799b80" Nov 02 17:48:04.565489 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:48:04.565469 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9168cddae3018cf39b674d263625049662ee01cf1e9cc416fd73397cd4799b80\": container with ID starting with 9168cddae3018cf39b674d263625049662ee01cf1e9cc416fd73397cd4799b80 not found: ID does not exist" containerID="9168cddae3018cf39b674d263625049662ee01cf1e9cc416fd73397cd4799b80" Nov 02 17:48:04.565554 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.565498 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9168cddae3018cf39b674d263625049662ee01cf1e9cc416fd73397cd4799b80"} err="failed to get container status \"9168cddae3018cf39b674d263625049662ee01cf1e9cc416fd73397cd4799b80\": rpc error: code = NotFound desc = could not find container \"9168cddae3018cf39b674d263625049662ee01cf1e9cc416fd73397cd4799b80\": container with ID starting with 9168cddae3018cf39b674d263625049662ee01cf1e9cc416fd73397cd4799b80 not found: ID does not exist" Nov 02 17:48:04.570807 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.570755 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk" podStartSLOduration=1.570738934 podStartE2EDuration="1.570738934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:04.569873661 +0000 UTC m=+856.358379191" watchObservedRunningTime="2026-11-02 17:48:04.570738934 +0000 UTC m=+856.359244463" Nov 02 17:48:04.589294 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.589245 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj" podStartSLOduration=1.5892336139999999 podStartE2EDuration="1.589233614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:04.588237737 +0000 UTC m=+856.376743267" watchObservedRunningTime="2026-11-02 17:48:04.589233614 +0000 UTC m=+856.377739141" Nov 02 17:48:04.604046 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.604020 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5"] Nov 02 17:48:04.608286 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.608264 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-935086-79d69586ff-9lls5"] Nov 02 17:48:04.888488 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.888398 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr"] Nov 02 17:48:04.888702 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.888647 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" podUID="1efe214f-8d3b-4902-a82f-a5d0be9229b4" containerName="maas-api" containerID="cri-o://97d9851802ace83351b0c082275ed35796d0b930084690f88ac91e219699a8eb" gracePeriod=30 Nov 02 17:48:04.977612 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:04.977582 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:05.258427 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.241969 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:05.469436 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.469404 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" Nov 02 17:48:05.564160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.564075 5549 generic.go:356] "Generic (PLEG): container finished" podID="1efe214f-8d3b-4902-a82f-a5d0be9229b4" containerID="97d9851802ace83351b0c082275ed35796d0b930084690f88ac91e219699a8eb" exitCode=0 Nov 02 17:48:05.564160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.564125 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" Nov 02 17:48:05.564387 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.564160 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" event={"ID":"1efe214f-8d3b-4902-a82f-a5d0be9229b4","Type":"ContainerDied","Data":"97d9851802ace83351b0c082275ed35796d0b930084690f88ac91e219699a8eb"} Nov 02 17:48:05.564387 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.564224 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr" event={"ID":"1efe214f-8d3b-4902-a82f-a5d0be9229b4","Type":"ContainerDied","Data":"75a7e831edeef506ee8bdee7e8015c9788e63bb85d89e71bcbe2bce2a367e3c8"} Nov 02 17:48:05.564387 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.564256 5549 scope.go:122] "RemoveContainer" containerID="97d9851802ace83351b0c082275ed35796d0b930084690f88ac91e219699a8eb" Nov 02 17:48:05.567452 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.567420 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1efe214f-8d3b-4902-a82f-a5d0be9229b4-kube-api-access-k9sjl\" (UniqueName: \"kubernetes.io/projected/1efe214f-8d3b-4902-a82f-a5d0be9229b4-kube-api-access-k9sjl\") pod \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\" (UID: \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\") " Nov 02 17:48:05.567599 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.567531 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1efe214f-8d3b-4902-a82f-a5d0be9229b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1efe214f-8d3b-4902-a82f-a5d0be9229b4-maas-api-tls\") pod \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\" (UID: \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\") " Nov 02 17:48:05.567725 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.567705 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1efe214f-8d3b-4902-a82f-a5d0be9229b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1efe214f-8d3b-4902-a82f-a5d0be9229b4-metrics-certs\") pod \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\" (UID: \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\") " Nov 02 17:48:05.568122 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.567798 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1efe214f-8d3b-4902-a82f-a5d0be9229b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efe214f-8d3b-4902-a82f-a5d0be9229b4-openshift-service-ca\") pod \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\" (UID: \"1efe214f-8d3b-4902-a82f-a5d0be9229b4\") " Nov 02 17:48:05.569127 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.569095 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1efe214f-8d3b-4902-a82f-a5d0be9229b4-openshift-service-ca" pod "1efe214f-8d3b-4902-a82f-a5d0be9229b4" (UID: "1efe214f-8d3b-4902-a82f-a5d0be9229b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:05.569965 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.569922 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1efe214f-8d3b-4902-a82f-a5d0be9229b4-kube-api-access-k9sjl" pod "1efe214f-8d3b-4902-a82f-a5d0be9229b4" (UID: "1efe214f-8d3b-4902-a82f-a5d0be9229b4"). InnerVolumeSpecName "kube-api-access-k9sjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:05.575593 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.575368 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1efe214f-8d3b-4902-a82f-a5d0be9229b4-maas-api-tls" pod "1efe214f-8d3b-4902-a82f-a5d0be9229b4" (UID: "1efe214f-8d3b-4902-a82f-a5d0be9229b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:48:05.580608 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.580581 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1efe214f-8d3b-4902-a82f-a5d0be9229b4-metrics-certs" pod "1efe214f-8d3b-4902-a82f-a5d0be9229b4" (UID: "1efe214f-8d3b-4902-a82f-a5d0be9229b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:48:05.582499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.582476 5549 scope.go:122] "RemoveContainer" containerID="97d9851802ace83351b0c082275ed35796d0b930084690f88ac91e219699a8eb" Nov 02 17:48:05.582818 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:48:05.582794 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d9851802ace83351b0c082275ed35796d0b930084690f88ac91e219699a8eb\": container with ID starting with 97d9851802ace83351b0c082275ed35796d0b930084690f88ac91e219699a8eb not found: ID does not exist" containerID="97d9851802ace83351b0c082275ed35796d0b930084690f88ac91e219699a8eb" Nov 02 17:48:05.582881 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.582828 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d9851802ace83351b0c082275ed35796d0b930084690f88ac91e219699a8eb"} err="failed to get container status \"97d9851802ace83351b0c082275ed35796d0b930084690f88ac91e219699a8eb\": rpc error: code = NotFound desc = could not find container \"97d9851802ace83351b0c082275ed35796d0b930084690f88ac91e219699a8eb\": container with ID starting with 97d9851802ace83351b0c082275ed35796d0b930084690f88ac91e219699a8eb not found: ID does not exist" Nov 02 17:48:05.670521 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.670486 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1efe214f-8d3b-4902-a82f-a5d0be9229b4-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:05.670521 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.670514 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efe214f-8d3b-4902-a82f-a5d0be9229b4-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:05.670521 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.670525 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9sjl\" (UniqueName: \"kubernetes.io/projected/1efe214f-8d3b-4902-a82f-a5d0be9229b4-kube-api-access-k9sjl\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:05.670521 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.670535 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1efe214f-8d3b-4902-a82f-a5d0be9229b4-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:05.890073 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.890038 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr"] Nov 02 17:48:05.894122 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:05.894092 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2129b9-66d879b765-d5khr"] Nov 02 17:48:06.013871 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.013813 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:06.263111 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.263072 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:06.335119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.335087 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1efe214f-8d3b-4902-a82f-a5d0be9229b4" path="/var/lib/kubelet/pods/1efe214f-8d3b-4902-a82f-a5d0be9229b4/volumes" Nov 02 17:48:06.335756 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.335732 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="253360ac-a72b-426e-8b02-77a9190e3962" path="/var/lib/kubelet/pods/253360ac-a72b-426e-8b02-77a9190e3962/volumes" Nov 02 17:48:06.342854 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.342818 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb" podUID="f2215f0b-29c5-4daf-b19e-898d6d41647c" containerName="payload-processing" containerID="cri-o://642eec8f8951c950a4fc442bb0b9168dbecc252455e034bf90205019013e7551" gracePeriod=25 Nov 02 17:48:06.357584 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.357556 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4" podUID="33b74e58-f777-4511-a68f-8c39d1eac863" containerName="payload-pre-processing" containerID="cri-o://bd91421e39b69230b77908169b19ec3370fec383edc4c935ba94c2b77685e127" gracePeriod=25 Nov 02 17:48:06.416563 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.416532 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:06.583981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.577558 5549 generic.go:356] "Generic (PLEG): container finished" podID="f2215f0b-29c5-4daf-b19e-898d6d41647c" containerID="642eec8f8951c950a4fc442bb0b9168dbecc252455e034bf90205019013e7551" exitCode=0 Nov 02 17:48:06.583981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.577621 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb" event={"ID":"f2215f0b-29c5-4daf-b19e-898d6d41647c","Type":"ContainerDied","Data":"642eec8f8951c950a4fc442bb0b9168dbecc252455e034bf90205019013e7551"} Nov 02 17:48:06.583981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.580855 5549 generic.go:356] "Generic (PLEG): container finished" podID="33b74e58-f777-4511-a68f-8c39d1eac863" containerID="bd91421e39b69230b77908169b19ec3370fec383edc4c935ba94c2b77685e127" exitCode=0 Nov 02 17:48:06.583981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.580894 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4" event={"ID":"33b74e58-f777-4511-a68f-8c39d1eac863","Type":"ContainerDied","Data":"bd91421e39b69230b77908169b19ec3370fec383edc4c935ba94c2b77685e127"} Nov 02 17:48:06.634269 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.634240 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb" Nov 02 17:48:06.693858 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.693833 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4" Nov 02 17:48:06.797354 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.797271 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2215f0b-29c5-4daf-b19e-898d6d41647c-kube-api-access-9lckw\" (UniqueName: \"kubernetes.io/projected/f2215f0b-29c5-4daf-b19e-898d6d41647c-kube-api-access-9lckw\") pod \"f2215f0b-29c5-4daf-b19e-898d6d41647c\" (UID: \"f2215f0b-29c5-4daf-b19e-898d6d41647c\") " Nov 02 17:48:06.797354 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.797315 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2215f0b-29c5-4daf-b19e-898d6d41647c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2215f0b-29c5-4daf-b19e-898d6d41647c-plugins-config-volume\") pod \"f2215f0b-29c5-4daf-b19e-898d6d41647c\" (UID: \"f2215f0b-29c5-4daf-b19e-898d6d41647c\") " Nov 02 17:48:06.797584 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.797359 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b74e58-f777-4511-a68f-8c39d1eac863-kube-api-access-8rzfj\" (UniqueName: \"kubernetes.io/projected/33b74e58-f777-4511-a68f-8c39d1eac863-kube-api-access-8rzfj\") pod \"33b74e58-f777-4511-a68f-8c39d1eac863\" (UID: \"33b74e58-f777-4511-a68f-8c39d1eac863\") " Nov 02 17:48:06.797584 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.797523 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b74e58-f777-4511-a68f-8c39d1eac863-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b74e58-f777-4511-a68f-8c39d1eac863-plugins-config-volume\") pod \"33b74e58-f777-4511-a68f-8c39d1eac863\" (UID: \"33b74e58-f777-4511-a68f-8c39d1eac863\") " Nov 02 17:48:06.797922 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.797888 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2215f0b-29c5-4daf-b19e-898d6d41647c-plugins-config-volume" pod "f2215f0b-29c5-4daf-b19e-898d6d41647c" (UID: "f2215f0b-29c5-4daf-b19e-898d6d41647c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:06.798054 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.797967 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b74e58-f777-4511-a68f-8c39d1eac863-plugins-config-volume" pod "33b74e58-f777-4511-a68f-8c39d1eac863" (UID: "33b74e58-f777-4511-a68f-8c39d1eac863"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:06.798262 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.798237 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2215f0b-29c5-4daf-b19e-898d6d41647c-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:06.798357 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.798267 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b74e58-f777-4511-a68f-8c39d1eac863-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:06.799643 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.799621 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2215f0b-29c5-4daf-b19e-898d6d41647c-kube-api-access-9lckw" pod "f2215f0b-29c5-4daf-b19e-898d6d41647c" (UID: "f2215f0b-29c5-4daf-b19e-898d6d41647c"). InnerVolumeSpecName "kube-api-access-9lckw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:06.799708 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.799654 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b74e58-f777-4511-a68f-8c39d1eac863-kube-api-access-8rzfj" pod "33b74e58-f777-4511-a68f-8c39d1eac863" (UID: "33b74e58-f777-4511-a68f-8c39d1eac863"). InnerVolumeSpecName "kube-api-access-8rzfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:06.872969 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.872929 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:06.899721 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.899694 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lckw\" (UniqueName: \"kubernetes.io/projected/f2215f0b-29c5-4daf-b19e-898d6d41647c-kube-api-access-9lckw\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:06.899721 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:06.899720 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rzfj\" (UniqueName: \"kubernetes.io/projected/33b74e58-f777-4511-a68f-8c39d1eac863-kube-api-access-8rzfj\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:07.017982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.016700 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:07.357976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.356523 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:07.591065 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.591033 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4" Nov 02 17:48:07.591239 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.591043 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4" event={"ID":"33b74e58-f777-4511-a68f-8c39d1eac863","Type":"ContainerDied","Data":"6c7853d5da2c454c6107023d7f9724879a8e839058b2f8695acb58ba4ce16a1a"} Nov 02 17:48:07.591239 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.591174 5549 scope.go:122] "RemoveContainer" containerID="bd91421e39b69230b77908169b19ec3370fec383edc4c935ba94c2b77685e127" Nov 02 17:48:07.593172 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.593153 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb" Nov 02 17:48:07.593299 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.593192 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb" event={"ID":"f2215f0b-29c5-4daf-b19e-898d6d41647c","Type":"ContainerDied","Data":"97c11ee6e5be22da3b2f9e10b8442e95cf7e070c2a902aa50af87c35d188715e"} Nov 02 17:48:07.602720 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.602599 5549 scope.go:122] "RemoveContainer" containerID="642eec8f8951c950a4fc442bb0b9168dbecc252455e034bf90205019013e7551" Nov 02 17:48:07.617082 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.617057 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4"] Nov 02 17:48:07.621888 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.621863 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2129b9-f569b5cc5-86jp4"] Nov 02 17:48:07.632851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.632825 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb"] Nov 02 17:48:07.635839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.635819 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2129b9-7485dc875f-qhrcb"] Nov 02 17:48:07.820210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.820171 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn"] Nov 02 17:48:07.820848 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.820834 5549 cpu_manager.go:414] "containerMap: removing container" podUID="33b74e58-f777-4511-a68f-8c39d1eac863" containerName="payload-pre-processing" Nov 02 17:48:07.820891 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.820849 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b74e58-f777-4511-a68f-8c39d1eac863" containerName="payload-pre-processing" Nov 02 17:48:07.820891 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.820870 5549 cpu_manager.go:414] "containerMap: removing container" podUID="f2215f0b-29c5-4daf-b19e-898d6d41647c" containerName="payload-processing" Nov 02 17:48:07.820891 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.820876 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2215f0b-29c5-4daf-b19e-898d6d41647c" containerName="payload-processing" Nov 02 17:48:07.820996 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.820894 5549 cpu_manager.go:414] "containerMap: removing container" podUID="1efe214f-8d3b-4902-a82f-a5d0be9229b4" containerName="maas-api" Nov 02 17:48:07.820996 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.820899 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1efe214f-8d3b-4902-a82f-a5d0be9229b4" containerName="maas-api" Nov 02 17:48:07.820996 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.820917 5549 cpu_manager.go:414] "containerMap: removing container" podUID="253360ac-a72b-426e-8b02-77a9190e3962" containerName="payload-processing" Nov 02 17:48:07.820996 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.820923 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253360ac-a72b-426e-8b02-77a9190e3962" containerName="payload-processing" Nov 02 17:48:07.821159 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.821022 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b74e58-f777-4511-a68f-8c39d1eac863" containerName="payload-pre-processing" Nov 02 17:48:07.821159 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.821033 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2215f0b-29c5-4daf-b19e-898d6d41647c" containerName="payload-processing" Nov 02 17:48:07.821159 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.821042 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1efe214f-8d3b-4902-a82f-a5d0be9229b4" containerName="maas-api" Nov 02 17:48:07.821159 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.821051 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="253360ac-a72b-426e-8b02-77a9190e3962" containerName="payload-processing" Nov 02 17:48:07.826077 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.826059 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" Nov 02 17:48:07.833309 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.833281 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn"] Nov 02 17:48:07.835704 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.834044 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-11f0a3-serving-cert\"" Nov 02 17:48:07.922794 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.922711 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b56136-a25b-4705-bcd7-55181db1a642-openshift-service-ca\") pod \"maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn\" (UID: \"d4b56136-a25b-4705-bcd7-55181db1a642\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" Nov 02 17:48:07.922929 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.922804 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbkhd\" (UniqueName: \"kubernetes.io/projected/d4b56136-a25b-4705-bcd7-55181db1a642-kube-api-access-zbkhd\") pod \"maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn\" (UID: \"d4b56136-a25b-4705-bcd7-55181db1a642\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" Nov 02 17:48:07.922929 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.922857 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b56136-a25b-4705-bcd7-55181db1a642-metrics-certs\") pod \"maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn\" (UID: \"d4b56136-a25b-4705-bcd7-55181db1a642\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" Nov 02 17:48:07.923199 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.923177 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b56136-a25b-4705-bcd7-55181db1a642-maas-api-tls\") pod \"maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn\" (UID: \"d4b56136-a25b-4705-bcd7-55181db1a642\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" Nov 02 17:48:07.965386 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:07.965351 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:08.024704 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:08.024668 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b56136-a25b-4705-bcd7-55181db1a642-maas-api-tls\") pod \"maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn\" (UID: \"d4b56136-a25b-4705-bcd7-55181db1a642\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" Nov 02 17:48:08.025127 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:08.024759 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b56136-a25b-4705-bcd7-55181db1a642-openshift-service-ca\") pod \"maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn\" (UID: \"d4b56136-a25b-4705-bcd7-55181db1a642\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" Nov 02 17:48:08.025127 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:08.024786 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbkhd\" (UniqueName: \"kubernetes.io/projected/d4b56136-a25b-4705-bcd7-55181db1a642-kube-api-access-zbkhd\") pod \"maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn\" (UID: \"d4b56136-a25b-4705-bcd7-55181db1a642\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" Nov 02 17:48:08.025127 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:08.024815 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b56136-a25b-4705-bcd7-55181db1a642-metrics-certs\") pod \"maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn\" (UID: \"d4b56136-a25b-4705-bcd7-55181db1a642\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" Nov 02 17:48:08.025454 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:08.025435 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b56136-a25b-4705-bcd7-55181db1a642-openshift-service-ca\") pod \"maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn\" (UID: \"d4b56136-a25b-4705-bcd7-55181db1a642\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" Nov 02 17:48:08.027504 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:08.027479 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b56136-a25b-4705-bcd7-55181db1a642-metrics-certs\") pod \"maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn\" (UID: \"d4b56136-a25b-4705-bcd7-55181db1a642\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" Nov 02 17:48:08.027627 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:08.027577 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b56136-a25b-4705-bcd7-55181db1a642-maas-api-tls\") pod \"maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn\" (UID: \"d4b56136-a25b-4705-bcd7-55181db1a642\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" Nov 02 17:48:08.033295 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:08.033276 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbkhd\" (UniqueName: \"kubernetes.io/projected/d4b56136-a25b-4705-bcd7-55181db1a642-kube-api-access-zbkhd\") pod \"maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn\" (UID: \"d4b56136-a25b-4705-bcd7-55181db1a642\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" Nov 02 17:48:08.161988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:08.159425 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" Nov 02 17:48:08.169983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:08.167418 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:08.323078 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:08.323037 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b74e58-f777-4511-a68f-8c39d1eac863" path="/var/lib/kubelet/pods/33b74e58-f777-4511-a68f-8c39d1eac863/volumes" Nov 02 17:48:08.323545 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:08.323523 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2215f0b-29c5-4daf-b19e-898d6d41647c" path="/var/lib/kubelet/pods/f2215f0b-29c5-4daf-b19e-898d6d41647c/volumes" Nov 02 17:48:08.497977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:08.494660 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn"] Nov 02 17:48:08.674333 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:08.674291 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" event={"ID":"d4b56136-a25b-4705-bcd7-55181db1a642","Type":"ContainerStarted","Data":"5eaf664625afa82d4a1a22f83c5c34cd813be079de8e0a3ec0dd24fbb0010bf8"} Nov 02 17:48:08.906049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:08.898630 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:09.558126 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:09.558087 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:09.683914 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:09.683876 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" event={"ID":"d4b56136-a25b-4705-bcd7-55181db1a642","Type":"ContainerStarted","Data":"55dcd23d4491a2474df953d255dd1433c816d8b9f36a0553584dabf65b8f817a"} Nov 02 17:48:09.684111 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:09.684082 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" Nov 02 17:48:09.699119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:09.699088 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:09.705201 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:09.705157 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" podStartSLOduration=2.226060586 podStartE2EDuration="2.705142946s" totalImagesPullingTime="479.08236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:09.701413612 +0000 UTC m=+861.489919179" watchObservedRunningTime="2026-11-02 17:48:09.705142946 +0000 UTC m=+861.493648475" Nov 02 17:48:09.846893 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:09.846796 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:10.056974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:10.048889 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:10.396072 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:10.396038 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:10.733985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:10.732453 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:11.497100 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:11.497053 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:11.645100 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:11.645068 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:48:12.416415 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:12.416373 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:13.581976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:13.581520 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:14.572942 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:14.572902 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:15.449850 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:15.449814 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:15.568319 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:15.568288 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk" Nov 02 17:48:15.568494 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:15.568479 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj" Nov 02 17:48:15.696404 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:15.696372 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" Nov 02 17:48:16.325456 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:16.325418 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:17.307025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:17.306989 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:17.588292 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:48:17.588215 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:48:18.158586 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:18.158545 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:18.846229 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:18.846133 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:19.650935 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:19.650898 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:20.426932 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:20.426894 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:21.151991 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:21.151910 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:21.912614 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:21.912532 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:22.062309 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:22.062257 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:22.604842 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:22.604805 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7347-a5f1c6"] Nov 02 17:48:22.610190 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:22.610171 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-a5f1c6" Nov 02 17:48:22.613699 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:22.613676 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zf9kl\"" Nov 02 17:48:22.615709 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:22.615682 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7347-a5f1c6"] Nov 02 17:48:22.676063 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:22.675634 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:22.788269 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:22.788239 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz7cj\" (UniqueName: \"kubernetes.io/projected/6007cc48-4388-43f3-9be0-abaf1eb37845-kube-api-access-fz7cj\") pod \"test-curl-7347-a5f1c6\" (UID: \"6007cc48-4388-43f3-9be0-abaf1eb37845\") " pod="opendatahub/test-curl-7347-a5f1c6" Nov 02 17:48:22.890197 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:22.890112 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz7cj\" (UniqueName: \"kubernetes.io/projected/6007cc48-4388-43f3-9be0-abaf1eb37845-kube-api-access-fz7cj\") pod \"test-curl-7347-a5f1c6\" (UID: \"6007cc48-4388-43f3-9be0-abaf1eb37845\") " pod="opendatahub/test-curl-7347-a5f1c6" Nov 02 17:48:22.898411 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:22.898384 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz7cj\" (UniqueName: \"kubernetes.io/projected/6007cc48-4388-43f3-9be0-abaf1eb37845-kube-api-access-fz7cj\") pod \"test-curl-7347-a5f1c6\" (UID: \"6007cc48-4388-43f3-9be0-abaf1eb37845\") " pod="opendatahub/test-curl-7347-a5f1c6" Nov 02 17:48:22.951194 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:22.951147 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-a5f1c6" Nov 02 17:48:23.236536 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:23.236505 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7347-a5f1c6"] Nov 02 17:48:23.237396 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:48:23.237367 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6007cc48_4388_43f3_9be0_abaf1eb37845.slice/crio-24f421241d6026e33f40a8717e9da81a311d0a72cea9ccbe1e4c086a9bccd899 WatchSource:0}: Error finding container 24f421241d6026e33f40a8717e9da81a311d0a72cea9ccbe1e4c086a9bccd899: Status 404 returned error can't find the container with id 24f421241d6026e33f40a8717e9da81a311d0a72cea9ccbe1e4c086a9bccd899 Nov 02 17:48:23.312485 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:23.312428 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:23.448283 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:23.448251 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:23.978770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:23.978732 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7347-a5f1c6" event={"ID":"6007cc48-4388-43f3-9be0-abaf1eb37845","Type":"ContainerStarted","Data":"24f421241d6026e33f40a8717e9da81a311d0a72cea9ccbe1e4c086a9bccd899"} Nov 02 17:48:24.004371 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:24.004335 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:24.735132 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:24.735092 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:25.425155 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:25.425118 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:25.994536 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:25.994500 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7347-a5f1c6" event={"ID":"6007cc48-4388-43f3-9be0-abaf1eb37845","Type":"ContainerStarted","Data":"ab97e2666d22d423b1a66104ed4030b12d920cd20efb0a36029cdd2ccb39fd49"} Nov 02 17:48:26.013559 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:26.013506 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7347-a5f1c6" podStartSLOduration=2.012057293 podStartE2EDuration="4.013492451s" totalImagesPullingTime="2.001435158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:26.010795414 +0000 UTC m=+877.799300942" watchObservedRunningTime="2026-11-02 17:48:26.013492451 +0000 UTC m=+877.801997978" Nov 02 17:48:26.250612 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:26.250527 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:26.727855 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:26.727817 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7347-a5f1c6"] Nov 02 17:48:26.731452 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:26.731420 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7347-a5f1c6"] Nov 02 17:48:26.919656 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:26.919623 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:27.810046 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:27.805190 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:28.049315 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:28.049276 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7347-a5f1c6" podUID="6007cc48-4388-43f3-9be0-abaf1eb37845" containerName="test-curl-7347-a5f1c6" containerID="cri-o://ab97e2666d22d423b1a66104ed4030b12d920cd20efb0a36029cdd2ccb39fd49" gracePeriod=2 Nov 02 17:48:28.051938 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:48:28.051897 5549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-a5f1c6\" is forbidden: User \"system:node:ip-10-0-142-133.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-133.us-east-2.compute.internal' and this object" podUID="6007cc48-4388-43f3-9be0-abaf1eb37845" pod="opendatahub/test-curl-7347-a5f1c6" Nov 02 17:48:28.319839 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:48:28.319808 5549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-a5f1c6\" is forbidden: User \"system:node:ip-10-0-142-133.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-133.us-east-2.compute.internal' and this object" podUID="6007cc48-4388-43f3-9be0-abaf1eb37845" pod="opendatahub/test-curl-7347-a5f1c6" Nov 02 17:48:28.610852 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:28.610765 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:29.383856 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:29.380234 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:29.525129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:29.525099 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z"] Nov 02 17:48:29.546430 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:29.546398 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h"] Nov 02 17:48:29.650641 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:29.650521 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:29.815557 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:29.815518 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:30.114582 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:30.114551 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:30.220720 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:30.220693 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-a5f1c6" Nov 02 17:48:30.371933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:30.371838 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6007cc48-4388-43f3-9be0-abaf1eb37845-kube-api-access-fz7cj\" (UniqueName: \"kubernetes.io/projected/6007cc48-4388-43f3-9be0-abaf1eb37845-kube-api-access-fz7cj\") pod \"6007cc48-4388-43f3-9be0-abaf1eb37845\" (UID: \"6007cc48-4388-43f3-9be0-abaf1eb37845\") " Nov 02 17:48:30.392021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:30.388929 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:30.404580 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:30.395148 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6007cc48-4388-43f3-9be0-abaf1eb37845-kube-api-access-fz7cj" pod "6007cc48-4388-43f3-9be0-abaf1eb37845" (UID: "6007cc48-4388-43f3-9be0-abaf1eb37845"). InnerVolumeSpecName "kube-api-access-fz7cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:30.474320 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:30.474286 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz7cj\" (UniqueName: \"kubernetes.io/projected/6007cc48-4388-43f3-9be0-abaf1eb37845-kube-api-access-fz7cj\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:31.070446 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:31.070411 5549 generic.go:356] "Generic (PLEG): container finished" podID="6007cc48-4388-43f3-9be0-abaf1eb37845" containerID="ab97e2666d22d423b1a66104ed4030b12d920cd20efb0a36029cdd2ccb39fd49" exitCode=137 Nov 02 17:48:31.070628 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:31.070478 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-a5f1c6" Nov 02 17:48:31.070628 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:31.070521 5549 scope.go:122] "RemoveContainer" containerID="ab97e2666d22d423b1a66104ed4030b12d920cd20efb0a36029cdd2ccb39fd49" Nov 02 17:48:31.084607 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:31.084584 5549 scope.go:122] "RemoveContainer" containerID="ab97e2666d22d423b1a66104ed4030b12d920cd20efb0a36029cdd2ccb39fd49" Nov 02 17:48:31.085123 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:48:31.085099 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab97e2666d22d423b1a66104ed4030b12d920cd20efb0a36029cdd2ccb39fd49\": container with ID starting with ab97e2666d22d423b1a66104ed4030b12d920cd20efb0a36029cdd2ccb39fd49 not found: ID does not exist" containerID="ab97e2666d22d423b1a66104ed4030b12d920cd20efb0a36029cdd2ccb39fd49" Nov 02 17:48:31.085197 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:31.085135 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab97e2666d22d423b1a66104ed4030b12d920cd20efb0a36029cdd2ccb39fd49"} err="failed to get container status \"ab97e2666d22d423b1a66104ed4030b12d920cd20efb0a36029cdd2ccb39fd49\": rpc error: code = NotFound desc = could not find container \"ab97e2666d22d423b1a66104ed4030b12d920cd20efb0a36029cdd2ccb39fd49\": container with ID starting with ab97e2666d22d423b1a66104ed4030b12d920cd20efb0a36029cdd2ccb39fd49 not found: ID does not exist" Nov 02 17:48:31.162976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:31.132546 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:31.844887 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:31.844837 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:32.320601 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:32.320566 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6007cc48-4388-43f3-9be0-abaf1eb37845" path="/var/lib/kubelet/pods/6007cc48-4388-43f3-9be0-abaf1eb37845/volumes" Nov 02 17:48:32.642024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:32.630805 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:33.571291 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:33.566746 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:34.231479 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:34.231440 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:34.525496 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:34.525411 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z" podUID="17f7e4cd-9c02-4b92-899b-a5201c88d254" containerName="payload-processing" containerID="cri-o://31088770e2772069e123584675d38714055bae50973e5a2b60703ed1958aad62" gracePeriod=25 Nov 02 17:48:34.547043 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:34.547006 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h" podUID="4812a18c-24c8-487f-a327-7670d532a18c" containerName="payload-pre-processing" containerID="cri-o://7f28c669798c563f1609878b8df6e7d87328446e49862648a0107eea86359bdd" gracePeriod=25 Nov 02 17:48:34.849171 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:34.849131 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z" Nov 02 17:48:34.880861 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:34.880833 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h" Nov 02 17:48:34.954546 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:34.954517 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4812a18c-24c8-487f-a327-7670d532a18c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4812a18c-24c8-487f-a327-7670d532a18c-plugins-config-volume\") pod \"4812a18c-24c8-487f-a327-7670d532a18c\" (UID: \"4812a18c-24c8-487f-a327-7670d532a18c\") " Nov 02 17:48:34.954721 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:34.954641 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4812a18c-24c8-487f-a327-7670d532a18c-kube-api-access-wbff2\" (UniqueName: \"kubernetes.io/projected/4812a18c-24c8-487f-a327-7670d532a18c-kube-api-access-wbff2\") pod \"4812a18c-24c8-487f-a327-7670d532a18c\" (UID: \"4812a18c-24c8-487f-a327-7670d532a18c\") " Nov 02 17:48:34.954804 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:34.954748 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f7e4cd-9c02-4b92-899b-a5201c88d254-kube-api-access-mhhd2\" (UniqueName: \"kubernetes.io/projected/17f7e4cd-9c02-4b92-899b-a5201c88d254-kube-api-access-mhhd2\") pod \"17f7e4cd-9c02-4b92-899b-a5201c88d254\" (UID: \"17f7e4cd-9c02-4b92-899b-a5201c88d254\") " Nov 02 17:48:34.954926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:34.954905 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f7e4cd-9c02-4b92-899b-a5201c88d254-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f7e4cd-9c02-4b92-899b-a5201c88d254-plugins-config-volume\") pod \"17f7e4cd-9c02-4b92-899b-a5201c88d254\" (UID: \"17f7e4cd-9c02-4b92-899b-a5201c88d254\") " Nov 02 17:48:34.955031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:34.954964 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4812a18c-24c8-487f-a327-7670d532a18c-plugins-config-volume" pod "4812a18c-24c8-487f-a327-7670d532a18c" (UID: "4812a18c-24c8-487f-a327-7670d532a18c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:34.955319 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:34.955289 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f7e4cd-9c02-4b92-899b-a5201c88d254-plugins-config-volume" pod "17f7e4cd-9c02-4b92-899b-a5201c88d254" (UID: "17f7e4cd-9c02-4b92-899b-a5201c88d254"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:34.955546 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:34.955522 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4812a18c-24c8-487f-a327-7670d532a18c-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:34.956943 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:34.956922 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4812a18c-24c8-487f-a327-7670d532a18c-kube-api-access-wbff2" pod "4812a18c-24c8-487f-a327-7670d532a18c" (UID: "4812a18c-24c8-487f-a327-7670d532a18c"). InnerVolumeSpecName "kube-api-access-wbff2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:34.957115 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:34.957090 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f7e4cd-9c02-4b92-899b-a5201c88d254-kube-api-access-mhhd2" pod "17f7e4cd-9c02-4b92-899b-a5201c88d254" (UID: "17f7e4cd-9c02-4b92-899b-a5201c88d254"). InnerVolumeSpecName "kube-api-access-mhhd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:35.004423 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.004385 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:35.063023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.057212 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f7e4cd-9c02-4b92-899b-a5201c88d254-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:35.063023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.057255 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbff2\" (UniqueName: \"kubernetes.io/projected/4812a18c-24c8-487f-a327-7670d532a18c-kube-api-access-wbff2\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:35.063023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.057272 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhhd2\" (UniqueName: \"kubernetes.io/projected/17f7e4cd-9c02-4b92-899b-a5201c88d254-kube-api-access-mhhd2\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:35.151978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.150798 5549 generic.go:356] "Generic (PLEG): container finished" podID="17f7e4cd-9c02-4b92-899b-a5201c88d254" containerID="31088770e2772069e123584675d38714055bae50973e5a2b60703ed1958aad62" exitCode=0 Nov 02 17:48:35.151978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.150997 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z" event={"ID":"17f7e4cd-9c02-4b92-899b-a5201c88d254","Type":"ContainerDied","Data":"31088770e2772069e123584675d38714055bae50973e5a2b60703ed1958aad62"} Nov 02 17:48:35.151978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.151085 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z" event={"ID":"17f7e4cd-9c02-4b92-899b-a5201c88d254","Type":"ContainerDied","Data":"d40be8c66bf30b34e71f59038c5a45477af37406483868fed6e3d13d34da44ea"} Nov 02 17:48:35.151978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.151114 5549 scope.go:122] "RemoveContainer" containerID="31088770e2772069e123584675d38714055bae50973e5a2b60703ed1958aad62" Nov 02 17:48:35.151978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.151277 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z" Nov 02 17:48:35.186992 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.185445 5549 generic.go:356] "Generic (PLEG): container finished" podID="4812a18c-24c8-487f-a327-7670d532a18c" containerID="7f28c669798c563f1609878b8df6e7d87328446e49862648a0107eea86359bdd" exitCode=0 Nov 02 17:48:35.186992 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.185535 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h" event={"ID":"4812a18c-24c8-487f-a327-7670d532a18c","Type":"ContainerDied","Data":"7f28c669798c563f1609878b8df6e7d87328446e49862648a0107eea86359bdd"} Nov 02 17:48:35.186992 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.185592 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h" event={"ID":"4812a18c-24c8-487f-a327-7670d532a18c","Type":"ContainerDied","Data":"9b30557bbe55e3023efffb860efca5d2e00dc3a4034547cc839b79d056c68fa2"} Nov 02 17:48:35.186992 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.185685 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h" Nov 02 17:48:35.235441 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.232053 5549 scope.go:122] "RemoveContainer" containerID="31088770e2772069e123584675d38714055bae50973e5a2b60703ed1958aad62" Nov 02 17:48:35.235441 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:48:35.235235 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31088770e2772069e123584675d38714055bae50973e5a2b60703ed1958aad62\": container with ID starting with 31088770e2772069e123584675d38714055bae50973e5a2b60703ed1958aad62 not found: ID does not exist" containerID="31088770e2772069e123584675d38714055bae50973e5a2b60703ed1958aad62" Nov 02 17:48:35.235441 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.235296 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31088770e2772069e123584675d38714055bae50973e5a2b60703ed1958aad62"} err="failed to get container status \"31088770e2772069e123584675d38714055bae50973e5a2b60703ed1958aad62\": rpc error: code = NotFound desc = could not find container \"31088770e2772069e123584675d38714055bae50973e5a2b60703ed1958aad62\": container with ID starting with 31088770e2772069e123584675d38714055bae50973e5a2b60703ed1958aad62 not found: ID does not exist" Nov 02 17:48:35.235441 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.235323 5549 scope.go:122] "RemoveContainer" containerID="7f28c669798c563f1609878b8df6e7d87328446e49862648a0107eea86359bdd" Nov 02 17:48:35.246175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.246147 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h"] Nov 02 17:48:35.249011 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.248986 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a976bc-fc44d487f-r8c2h"] Nov 02 17:48:35.253147 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.253124 5549 scope.go:122] "RemoveContainer" containerID="7f28c669798c563f1609878b8df6e7d87328446e49862648a0107eea86359bdd" Nov 02 17:48:35.253592 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:48:35.253559 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f28c669798c563f1609878b8df6e7d87328446e49862648a0107eea86359bdd\": container with ID starting with 7f28c669798c563f1609878b8df6e7d87328446e49862648a0107eea86359bdd not found: ID does not exist" containerID="7f28c669798c563f1609878b8df6e7d87328446e49862648a0107eea86359bdd" Nov 02 17:48:35.253695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.253604 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f28c669798c563f1609878b8df6e7d87328446e49862648a0107eea86359bdd"} err="failed to get container status \"7f28c669798c563f1609878b8df6e7d87328446e49862648a0107eea86359bdd\": rpc error: code = NotFound desc = could not find container \"7f28c669798c563f1609878b8df6e7d87328446e49862648a0107eea86359bdd\": container with ID starting with 7f28c669798c563f1609878b8df6e7d87328446e49862648a0107eea86359bdd not found: ID does not exist" Nov 02 17:48:35.255563 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.255481 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z"] Nov 02 17:48:35.260546 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.260522 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a976bc-7b74756c79-mjx4z"] Nov 02 17:48:35.834056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:35.827045 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:36.322713 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:36.322666 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17f7e4cd-9c02-4b92-899b-a5201c88d254" path="/var/lib/kubelet/pods/17f7e4cd-9c02-4b92-899b-a5201c88d254/volumes" Nov 02 17:48:36.323385 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:36.323361 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4812a18c-24c8-487f-a327-7670d532a18c" path="/var/lib/kubelet/pods/4812a18c-24c8-487f-a327-7670d532a18c/volumes" Nov 02 17:48:36.463742 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:36.463700 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb"] Nov 02 17:48:36.464330 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:36.464288 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" podUID="8277c127-0ecb-4839-b899-5145144dafa9" containerName="maas-api" containerID="cri-o://114912530c2efeb31cdd9d68dc641b12c9ecb49ac6d2f5e4a539db3ba5aa93f9" gracePeriod=30 Nov 02 17:48:36.553821 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:36.553774 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:37.314481 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.314440 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:37.619022 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.595480 5549 generic.go:356] "Generic (PLEG): container finished" podID="8277c127-0ecb-4839-b899-5145144dafa9" containerID="114912530c2efeb31cdd9d68dc641b12c9ecb49ac6d2f5e4a539db3ba5aa93f9" exitCode=0 Nov 02 17:48:37.619022 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.595710 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" event={"ID":"8277c127-0ecb-4839-b899-5145144dafa9","Type":"ContainerDied","Data":"114912530c2efeb31cdd9d68dc641b12c9ecb49ac6d2f5e4a539db3ba5aa93f9"} Nov 02 17:48:37.653975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.653158 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:48:37.710687 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.703401 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:37.713292 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.713261 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8277c127-0ecb-4839-b899-5145144dafa9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8277c127-0ecb-4839-b899-5145144dafa9-maas-api-tls\") pod \"8277c127-0ecb-4839-b899-5145144dafa9\" (UID: \"8277c127-0ecb-4839-b899-5145144dafa9\") " Nov 02 17:48:37.715209 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.715181 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8277c127-0ecb-4839-b899-5145144dafa9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8277c127-0ecb-4839-b899-5145144dafa9-metrics-certs\") pod \"8277c127-0ecb-4839-b899-5145144dafa9\" (UID: \"8277c127-0ecb-4839-b899-5145144dafa9\") " Nov 02 17:48:37.715489 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.715467 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8277c127-0ecb-4839-b899-5145144dafa9-kube-api-access-zhfxk\" (UniqueName: \"kubernetes.io/projected/8277c127-0ecb-4839-b899-5145144dafa9-kube-api-access-zhfxk\") pod \"8277c127-0ecb-4839-b899-5145144dafa9\" (UID: \"8277c127-0ecb-4839-b899-5145144dafa9\") " Nov 02 17:48:37.718437 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.718414 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8277c127-0ecb-4839-b899-5145144dafa9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8277c127-0ecb-4839-b899-5145144dafa9-openshift-service-ca\") pod \"8277c127-0ecb-4839-b899-5145144dafa9\" (UID: \"8277c127-0ecb-4839-b899-5145144dafa9\") " Nov 02 17:48:37.719894 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.719868 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8277c127-0ecb-4839-b899-5145144dafa9-openshift-service-ca" pod "8277c127-0ecb-4839-b899-5145144dafa9" (UID: "8277c127-0ecb-4839-b899-5145144dafa9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:37.725331 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.725306 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8277c127-0ecb-4839-b899-5145144dafa9-metrics-certs" pod "8277c127-0ecb-4839-b899-5145144dafa9" (UID: "8277c127-0ecb-4839-b899-5145144dafa9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:48:37.730981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.730929 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp"] Nov 02 17:48:37.732186 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.732143 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8277c127-0ecb-4839-b899-5145144dafa9-maas-api-tls" pod "8277c127-0ecb-4839-b899-5145144dafa9" (UID: "8277c127-0ecb-4839-b899-5145144dafa9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:48:37.754341 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.753418 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8277c127-0ecb-4839-b899-5145144dafa9-kube-api-access-zhfxk" pod "8277c127-0ecb-4839-b899-5145144dafa9" (UID: "8277c127-0ecb-4839-b899-5145144dafa9"). InnerVolumeSpecName "kube-api-access-zhfxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:37.764580 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.762033 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j"] Nov 02 17:48:37.830482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.830392 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8277c127-0ecb-4839-b899-5145144dafa9-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:37.830482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.830433 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8277c127-0ecb-4839-b899-5145144dafa9-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:37.830482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.830447 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8277c127-0ecb-4839-b899-5145144dafa9-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:37.830482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:37.830460 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhfxk\" (UniqueName: \"kubernetes.io/projected/8277c127-0ecb-4839-b899-5145144dafa9-kube-api-access-zhfxk\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:38.127034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:38.103662 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:38.258998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:38.257564 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:38.500979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:38.500282 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:38.736019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:38.719519 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" Nov 02 17:48:38.736019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:38.719413 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb" event={"ID":"8277c127-0ecb-4839-b899-5145144dafa9","Type":"ContainerDied","Data":"f3affda9a4655f389968d622279bad930bc5838daafccd946910fc7c16975a67"} Nov 02 17:48:38.736019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:38.719667 5549 scope.go:122] "RemoveContainer" containerID="114912530c2efeb31cdd9d68dc641b12c9ecb49ac6d2f5e4a539db3ba5aa93f9" Nov 02 17:48:38.776239 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:38.775049 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb"] Nov 02 17:48:38.776783 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:38.776758 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a976bc-84b755767-9bzcb"] Nov 02 17:48:39.368075 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:39.362595 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:40.224178 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:40.224142 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:40.332004 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:40.331968 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8277c127-0ecb-4839-b899-5145144dafa9" path="/var/lib/kubelet/pods/8277c127-0ecb-4839-b899-5145144dafa9/volumes" Nov 02 17:48:41.204173 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.204142 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:41.366540 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.366492 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:41.648114 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.648028 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k"] Nov 02 17:48:41.648766 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.648744 5549 cpu_manager.go:414] "containerMap: removing container" podUID="17f7e4cd-9c02-4b92-899b-a5201c88d254" containerName="payload-processing" Nov 02 17:48:41.648766 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.648761 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f7e4cd-9c02-4b92-899b-a5201c88d254" containerName="payload-processing" Nov 02 17:48:41.648988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.648779 5549 cpu_manager.go:414] "containerMap: removing container" podUID="8277c127-0ecb-4839-b899-5145144dafa9" containerName="maas-api" Nov 02 17:48:41.648988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.648785 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8277c127-0ecb-4839-b899-5145144dafa9" containerName="maas-api" Nov 02 17:48:41.648988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.648801 5549 cpu_manager.go:414] "containerMap: removing container" podUID="4812a18c-24c8-487f-a327-7670d532a18c" containerName="payload-pre-processing" Nov 02 17:48:41.648988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.648806 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4812a18c-24c8-487f-a327-7670d532a18c" containerName="payload-pre-processing" Nov 02 17:48:41.648988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.648815 5549 cpu_manager.go:414] "containerMap: removing container" podUID="6007cc48-4388-43f3-9be0-abaf1eb37845" containerName="test-curl-7347-a5f1c6" Nov 02 17:48:41.648988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.648820 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6007cc48-4388-43f3-9be0-abaf1eb37845" containerName="test-curl-7347-a5f1c6" Nov 02 17:48:41.648988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.648941 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f7e4cd-9c02-4b92-899b-a5201c88d254" containerName="payload-processing" Nov 02 17:48:41.648988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.648968 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8277c127-0ecb-4839-b899-5145144dafa9" containerName="maas-api" Nov 02 17:48:41.648988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.648977 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="4812a18c-24c8-487f-a327-7670d532a18c" containerName="payload-pre-processing" Nov 02 17:48:41.648988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.648984 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6007cc48-4388-43f3-9be0-abaf1eb37845" containerName="test-curl-7347-a5f1c6" Nov 02 17:48:41.655932 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.655909 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.663287 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.663243 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4c9298-openshift-default-dockercfg-rqzvt\"" Nov 02 17:48:41.665230 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.665187 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k"] Nov 02 17:48:41.775718 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.775681 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:41.803833 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.799002 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a56a6b8e-233b-4256-b47f-48a323ce8f55-istiod-ca-cert\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.803833 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.799084 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-credential-socket\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.803833 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.799126 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-workload-certs\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.803833 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.799263 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-workload-socket\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.803833 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.799316 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-envoy\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.803833 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.799397 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tbdr\" (UniqueName: \"kubernetes.io/projected/a56a6b8e-233b-4256-b47f-48a323ce8f55-kube-api-access-5tbdr\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.803833 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.799542 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-token\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.803833 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.799587 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-podinfo\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.803833 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.799614 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-data\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.907065 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.906966 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-workload-socket\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.908408 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.908385 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-envoy\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.909015 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.908944 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tbdr\" (UniqueName: \"kubernetes.io/projected/a56a6b8e-233b-4256-b47f-48a323ce8f55-kube-api-access-5tbdr\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.909277 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.909262 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-token\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.909382 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.909371 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-podinfo\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.909474 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.909463 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-data\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.909581 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.909571 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a56a6b8e-233b-4256-b47f-48a323ce8f55-istiod-ca-cert\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.909703 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.909692 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-credential-socket\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.909799 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.909787 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-workload-certs\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.919015 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.910197 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-workload-certs\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.920056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.908302 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-workload-socket\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.920203 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.914822 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a56a6b8e-233b-4256-b47f-48a323ce8f55-istiod-ca-cert\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.920300 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.915086 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-data\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.920388 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.915312 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-credential-socket\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.938979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.931481 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:41.958541 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.942784 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-token\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.958541 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.948025 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tbdr\" (UniqueName: \"kubernetes.io/projected/a56a6b8e-233b-4256-b47f-48a323ce8f55-kube-api-access-5tbdr\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.964343 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.961642 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-podinfo\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.964794 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.964769 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-envoy\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:41.985125 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:41.973070 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:42.133148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.133110 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:42.341167 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.341134 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:42.368998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.364035 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k"] Nov 02 17:48:42.374904 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.374854 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:48:42.375068 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.374935 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:48:42.375068 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.374998 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:48:42.722280 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.722244 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx"] Nov 02 17:48:42.727129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.727099 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:42.731421 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.731384 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" podUID="53edf2f7-a11d-456f-820d-2727ca8bdada" containerName="payload-processing" containerID="cri-o://f8714b24f0f5d968ff049acc4487c6dcb7ba60dafd26170ccfef397a5738515a" gracePeriod=25 Nov 02 17:48:42.737614 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.737588 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d1fc77-openshift-default-dockercfg-hgpcc\"" Nov 02 17:48:42.742394 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.742363 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx"] Nov 02 17:48:42.765196 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.765148 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" podUID="c629cfcd-ff99-4340-ae7f-dc417b197b43" containerName="payload-pre-processing" containerID="cri-o://998011eaa59211380d0b820133ece3ca38aae5a650fa4578cb9136dd93c684fb" gracePeriod=25 Nov 02 17:48:42.878000 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.866947 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:42.974025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.973067 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2"] Nov 02 17:48:42.989019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.929504 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-credential-socket\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:42.989019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.988633 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-envoy\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:42.989019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.988916 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qcnc\" (UniqueName: \"kubernetes.io/projected/57b8639c-8c58-4d9a-81c9-f5c262a11884-kube-api-access-2qcnc\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:42.989342 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.989065 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-data\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:42.989342 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.989199 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-podinfo\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.003035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.989866 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-token\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.003035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.990034 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b8639c-8c58-4d9a-81c9-f5c262a11884-istiod-ca-cert\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.003035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.990097 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-workload-certs\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.003035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:42.994599 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-workload-socket\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.044019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.038661 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" event={"ID":"a56a6b8e-233b-4256-b47f-48a323ce8f55","Type":"ContainerStarted","Data":"485d45edf1e797dcee8144203100e279ccda4ed50a42bde120abfb90bfda8207"} Nov 02 17:48:43.044019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.038711 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" event={"ID":"a56a6b8e-233b-4256-b47f-48a323ce8f55","Type":"ContainerStarted","Data":"95d146050b28ba4fbf5a7780a82c010436b0574d6e2d0c1ff12d63b45934badb"} Nov 02 17:48:43.044019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.038816 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.044019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.038837 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2"] Nov 02 17:48:43.096659 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.096613 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-workload-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.096817 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.096678 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-credential-socket\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.096817 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.096725 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-envoy\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.096891 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.096855 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qcnc\" (UniqueName: \"kubernetes.io/projected/57b8639c-8c58-4d9a-81c9-f5c262a11884-kube-api-access-2qcnc\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.096923 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.096897 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-data\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.097011 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.096934 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istiod-ca-cert\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.097011 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.096982 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-podinfo\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.097094 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.097022 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-credential-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.097094 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.097054 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-envoy\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.097094 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.097084 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-data\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.097196 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.097176 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-podinfo\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.097233 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.097215 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bmt7\" (UniqueName: \"kubernetes.io/projected/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-kube-api-access-9bmt7\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.097283 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.097269 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-token\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.097325 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.097310 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b8639c-8c58-4d9a-81c9-f5c262a11884-istiod-ca-cert\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.097380 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.097367 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-workload-certs\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.097442 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.097429 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-token\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.097535 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.097520 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-workload-socket\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.097604 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.097591 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-workload-certs\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.098150 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.098127 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-credential-socket\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.112912 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.112878 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-data\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.116175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.116148 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b8639c-8c58-4d9a-81c9-f5c262a11884-istiod-ca-cert\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.116648 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.116627 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-workload-certs\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.117196 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.117172 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-workload-socket\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.118824 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.118773 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" podStartSLOduration=2.118756834 podStartE2EDuration="2.118756834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:43.063084982 +0000 UTC m=+894.851590514" watchObservedRunningTime="2026-11-02 17:48:43.118756834 +0000 UTC m=+894.907262366" Nov 02 17:48:43.120789 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.120752 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-envoy\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.126525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.126485 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a976bc-openshift-default-dockercfg-qwjsm\"" Nov 02 17:48:43.133930 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.129469 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-podinfo\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.133930 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.132385 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-token\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.135283 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.135247 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qcnc\" (UniqueName: \"kubernetes.io/projected/57b8639c-8c58-4d9a-81c9-f5c262a11884-kube-api-access-2qcnc\") pod \"e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.199379 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.199341 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-workload-certs\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.199610 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.199407 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-workload-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.199610 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.199526 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istiod-ca-cert\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.199610 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.199586 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-credential-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.199824 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.199621 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-envoy\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.199824 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.199676 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-data\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.199824 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.199767 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-podinfo\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.199824 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.199800 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bmt7\" (UniqueName: \"kubernetes.io/projected/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-kube-api-access-9bmt7\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.199824 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.199811 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-workload-certs\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.200130 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.199899 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-workload-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.200130 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.199916 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-token\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.200302 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.200276 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istiod-ca-cert\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.200424 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.200406 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-credential-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.200499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.200471 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-data\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.202442 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.202416 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-envoy\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.202596 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.202577 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-podinfo\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.209243 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.209222 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bmt7\" (UniqueName: \"kubernetes.io/projected/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-kube-api-access-9bmt7\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.209243 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.209239 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-token\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.356597 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.356458 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:43.365978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.363471 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:43.375478 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.373342 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:43.524212 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.524186 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:43.727370 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.727346 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx"] Nov 02 17:48:43.727606 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:48:43.727584 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b8639c_8c58_4d9a_81c9_f5c262a11884.slice/crio-53e8008bdf29215124704071efc34afce5c2fb9666035b870b2fc4b1f319d5af WatchSource:0}: Error finding container 53e8008bdf29215124704071efc34afce5c2fb9666035b870b2fc4b1f319d5af: Status 404 returned error can't find the container with id 53e8008bdf29215124704071efc34afce5c2fb9666035b870b2fc4b1f319d5af Nov 02 17:48:43.730398 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:48:43.730374 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfed89fe_b4e4_41b2_82b0_2ba0d79469de.slice/crio-74c6aba24745b74248a459911959cf406085d215cd012e10de30fce4260583ac WatchSource:0}: Error finding container 74c6aba24745b74248a459911959cf406085d215cd012e10de30fce4260583ac: Status 404 returned error can't find the container with id 74c6aba24745b74248a459911959cf406085d215cd012e10de30fce4260583ac Nov 02 17:48:43.730604 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.730573 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:48:43.730667 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.730654 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:48:43.730702 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.730690 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:48:43.731623 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.731600 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2"] Nov 02 17:48:43.732818 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.732788 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:48:43.732905 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.732851 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:48:43.732905 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.732889 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:48:43.986363 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.986197 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:43.992069 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:43.992040 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:44.027438 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:44.027392 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" event={"ID":"57b8639c-8c58-4d9a-81c9-f5c262a11884","Type":"ContainerStarted","Data":"08be164b6776ced2d5d1e0c324929647ec7b401de830adb35ab57c5396edff71"} Nov 02 17:48:44.027616 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:44.027465 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" event={"ID":"57b8639c-8c58-4d9a-81c9-f5c262a11884","Type":"ContainerStarted","Data":"53e8008bdf29215124704071efc34afce5c2fb9666035b870b2fc4b1f319d5af"} Nov 02 17:48:44.029551 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:44.029520 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" event={"ID":"cfed89fe-b4e4-41b2-82b0-2ba0d79469de","Type":"ContainerStarted","Data":"b024bd8c3dec58978906753eee86be208dc78fa2e790052279be5351e69c1014"} Nov 02 17:48:44.029690 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:44.029572 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" event={"ID":"cfed89fe-b4e4-41b2-82b0-2ba0d79469de","Type":"ContainerStarted","Data":"74c6aba24745b74248a459911959cf406085d215cd012e10de30fce4260583ac"} Nov 02 17:48:44.029923 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:44.029903 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:44.030891 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:44.030865 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:48:44.059850 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:44.059792 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" podStartSLOduration=2.059774782 podStartE2EDuration="2.059774782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:44.056515995 +0000 UTC m=+895.845021524" watchObservedRunningTime="2026-11-02 17:48:44.059774782 +0000 UTC m=+895.848280311" Nov 02 17:48:44.115019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:44.114943 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" podStartSLOduration=2.114924052 podStartE2EDuration="2.114924052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:44.11206204 +0000 UTC m=+895.900567567" watchObservedRunningTime="2026-11-02 17:48:44.114924052 +0000 UTC m=+895.903429604" Nov 02 17:48:44.295026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:44.290188 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:44.376098 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:44.376064 5549 patch_prober.go:29] interesting pod/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 17:48:44.376477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:44.376121 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" podUID="cfed89fe-b4e4-41b2-82b0-2ba0d79469de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 17:48:44.376614 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:44.376578 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:44.376663 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:44.376635 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:44.382024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:44.381981 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:44.605011 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:44.604881 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" podUID="53edf2f7-a11d-456f-820d-2727ca8bdada" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:48:44.605727 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:44.605697 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" podUID="c629cfcd-ff99-4340-ae7f-dc417b197b43" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:48:45.037364 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:45.037232 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:45.038312 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:45.038287 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:48:45.374315 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:45.374242 5549 patch_prober.go:29] interesting pod/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 17:48:45.374315 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:45.374295 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" podUID="cfed89fe-b4e4-41b2-82b0-2ba0d79469de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 17:48:45.521966 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:45.521921 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:45.693034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:45.687896 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:45.854977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:45.848669 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:46.331978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:46.321640 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:46.399980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:46.398758 5549 patch_prober.go:29] interesting pod/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 17:48:46.399980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:46.398817 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" podUID="cfed89fe-b4e4-41b2-82b0-2ba0d79469de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 17:48:46.614612 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:46.614521 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2"] Nov 02 17:48:46.706998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:46.706729 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:46.866731 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:46.866634 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:47.065589 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:47.065544 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" podUID="cfed89fe-b4e4-41b2-82b0-2ba0d79469de" containerName="istio-proxy" containerID="cri-o://b024bd8c3dec58978906753eee86be208dc78fa2e790052279be5351e69c1014" gracePeriod=30 Nov 02 17:48:47.215531 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:47.215494 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:47.252546 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:47.252516 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw"] Nov 02 17:48:47.257068 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:47.257049 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw" Nov 02 17:48:47.266556 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:47.266533 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4c9298\"" Nov 02 17:48:47.270736 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:47.270716 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw"] Nov 02 17:48:47.339788 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:47.339757 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2mv7\" (UniqueName: \"kubernetes.io/projected/6428db64-abdc-4a92-8826-016c79a52bd1-kube-api-access-t2mv7\") pod \"payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw\" (UID: \"6428db64-abdc-4a92-8826-016c79a52bd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw" Nov 02 17:48:47.339996 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:47.339923 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6428db64-abdc-4a92-8826-016c79a52bd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw\" (UID: \"6428db64-abdc-4a92-8826-016c79a52bd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw" Nov 02 17:48:47.481031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:47.432691 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:47.481031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:47.459113 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6428db64-abdc-4a92-8826-016c79a52bd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw\" (UID: \"6428db64-abdc-4a92-8826-016c79a52bd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw" Nov 02 17:48:47.481031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:47.459466 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2mv7\" (UniqueName: \"kubernetes.io/projected/6428db64-abdc-4a92-8826-016c79a52bd1-kube-api-access-t2mv7\") pod \"payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw\" (UID: \"6428db64-abdc-4a92-8826-016c79a52bd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw" Nov 02 17:48:47.481031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:47.461096 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6428db64-abdc-4a92-8826-016c79a52bd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw\" (UID: \"6428db64-abdc-4a92-8826-016c79a52bd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw" Nov 02 17:48:47.542977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:47.542404 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2mv7\" (UniqueName: \"kubernetes.io/projected/6428db64-abdc-4a92-8826-016c79a52bd1-kube-api-access-t2mv7\") pod \"payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw\" (UID: \"6428db64-abdc-4a92-8826-016c79a52bd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw" Nov 02 17:48:47.587976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:47.587681 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw" Nov 02 17:48:47.643894 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:47.643665 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:47.738272 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:48:47.738166 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:48:48.003315 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.002142 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw"] Nov 02 17:48:48.074350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.074319 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw" event={"ID":"6428db64-abdc-4a92-8826-016c79a52bd1","Type":"ContainerStarted","Data":"02b40b747a7aee2c3353dc1aa053a70753cef7ea17dfc49e912a878d7fe9d831"} Nov 02 17:48:48.141129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.140999 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:48.408086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.407873 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:48.409835 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.409811 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp"] Nov 02 17:48:48.418372 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.418347 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp" Nov 02 17:48:48.425743 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.423280 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp"] Nov 02 17:48:48.425743 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.423498 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d1fc77-dockercfg-2rkst\"" Nov 02 17:48:48.425743 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.425133 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d1fc77\"" Nov 02 17:48:48.440429 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.440396 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr"] Nov 02 17:48:48.447074 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.446764 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr" Nov 02 17:48:48.456819 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.456020 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr"] Nov 02 17:48:48.569976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.518122 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhfsw\" (UniqueName: \"kubernetes.io/projected/3544d84a-2276-4ce3-bda4-3d9c66ee96f6-kube-api-access-nhfsw\") pod \"payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp\" (UID: \"3544d84a-2276-4ce3-bda4-3d9c66ee96f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp" Nov 02 17:48:48.569976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.518309 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3544d84a-2276-4ce3-bda4-3d9c66ee96f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp\" (UID: \"3544d84a-2276-4ce3-bda4-3d9c66ee96f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp" Nov 02 17:48:48.616344 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.614164 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:48.623988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.623868 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfc5a8b-de29-498d-b1ba-e4d4671b82fd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr\" (UID: \"8bfc5a8b-de29-498d-b1ba-e4d4671b82fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr" Nov 02 17:48:48.646991 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.635088 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhfsw\" (UniqueName: \"kubernetes.io/projected/3544d84a-2276-4ce3-bda4-3d9c66ee96f6-kube-api-access-nhfsw\") pod \"payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp\" (UID: \"3544d84a-2276-4ce3-bda4-3d9c66ee96f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp" Nov 02 17:48:48.646991 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.635180 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7wkb\" (UniqueName: \"kubernetes.io/projected/8bfc5a8b-de29-498d-b1ba-e4d4671b82fd-kube-api-access-m7wkb\") pod \"payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr\" (UID: \"8bfc5a8b-de29-498d-b1ba-e4d4671b82fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr" Nov 02 17:48:48.646991 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.635336 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3544d84a-2276-4ce3-bda4-3d9c66ee96f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp\" (UID: \"3544d84a-2276-4ce3-bda4-3d9c66ee96f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp" Nov 02 17:48:48.646991 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.636398 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3544d84a-2276-4ce3-bda4-3d9c66ee96f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp\" (UID: \"3544d84a-2276-4ce3-bda4-3d9c66ee96f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp" Nov 02 17:48:48.684002 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.683904 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhfsw\" (UniqueName: \"kubernetes.io/projected/3544d84a-2276-4ce3-bda4-3d9c66ee96f6-kube-api-access-nhfsw\") pod \"payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp\" (UID: \"3544d84a-2276-4ce3-bda4-3d9c66ee96f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp" Nov 02 17:48:48.737042 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.736997 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfc5a8b-de29-498d-b1ba-e4d4671b82fd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr\" (UID: \"8bfc5a8b-de29-498d-b1ba-e4d4671b82fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr" Nov 02 17:48:48.737883 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.737847 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7wkb\" (UniqueName: \"kubernetes.io/projected/8bfc5a8b-de29-498d-b1ba-e4d4671b82fd-kube-api-access-m7wkb\") pod \"payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr\" (UID: \"8bfc5a8b-de29-498d-b1ba-e4d4671b82fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr" Nov 02 17:48:48.749988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.739870 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfc5a8b-de29-498d-b1ba-e4d4671b82fd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr\" (UID: \"8bfc5a8b-de29-498d-b1ba-e4d4671b82fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr" Nov 02 17:48:48.750234 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.747205 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp" Nov 02 17:48:48.774042 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.771074 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7wkb\" (UniqueName: \"kubernetes.io/projected/8bfc5a8b-de29-498d-b1ba-e4d4671b82fd-kube-api-access-m7wkb\") pod \"payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr\" (UID: \"8bfc5a8b-de29-498d-b1ba-e4d4671b82fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr" Nov 02 17:48:48.774042 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.771429 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr" Nov 02 17:48:48.934643 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.934496 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 17:48:48.941812 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:48.941745 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 17:48:49.117137 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:49.114994 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw" event={"ID":"6428db64-abdc-4a92-8826-016c79a52bd1","Type":"ContainerStarted","Data":"8904b189e9f80553a39c1193dcbf649fe0e06719c582c72a63b2c8cf01069cf2"} Nov 02 17:48:49.117137 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:49.116079 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw" Nov 02 17:48:49.141425 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:49.141383 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw" podStartSLOduration=2.141370729 podStartE2EDuration="2.141370729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:49.137310903 +0000 UTC m=+900.925816433" watchObservedRunningTime="2026-11-02 17:48:49.141370729 +0000 UTC m=+900.929876279" Nov 02 17:48:49.212805 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:49.212774 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp"] Nov 02 17:48:49.242793 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:49.242757 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:49.373292 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:49.368304 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr"] Nov 02 17:48:50.046999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:50.040373 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:50.146680 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:50.146631 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr" event={"ID":"8bfc5a8b-de29-498d-b1ba-e4d4671b82fd","Type":"ContainerStarted","Data":"1218967aa9745e76f664d3f56d01411b355ffd863a449bbdd5bd3842b648566b"} Nov 02 17:48:50.146890 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:50.146696 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr" event={"ID":"8bfc5a8b-de29-498d-b1ba-e4d4671b82fd","Type":"ContainerStarted","Data":"357eeb520f19693f6361fbba74adc436cf851ddbae4e5fb2c50ca57fd1e924d6"} Nov 02 17:48:50.147068 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:50.147043 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr" Nov 02 17:48:50.148739 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:50.148714 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp" event={"ID":"3544d84a-2276-4ce3-bda4-3d9c66ee96f6","Type":"ContainerStarted","Data":"08c3d68ef1d1db83486976ab47d28f1ff875dd2dac2e92e9e41834754df2be45"} Nov 02 17:48:50.148851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:50.148760 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp" event={"ID":"3544d84a-2276-4ce3-bda4-3d9c66ee96f6","Type":"ContainerStarted","Data":"b7f505aa9be71953e280265d8b72b8629305519f0cbc06a4ec6863fab23d1336"} Nov 02 17:48:50.148851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:50.148820 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp" Nov 02 17:48:50.171863 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:50.171810 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr" podStartSLOduration=2.171796422 podStartE2EDuration="2.171796422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:50.171427437 +0000 UTC m=+901.959932978" watchObservedRunningTime="2026-11-02 17:48:50.171796422 +0000 UTC m=+901.960301950" Nov 02 17:48:50.192667 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:50.192616 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp" podStartSLOduration=2.192602877 podStartE2EDuration="2.192602877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:50.191745083 +0000 UTC m=+901.980250612" watchObservedRunningTime="2026-11-02 17:48:50.192602877 +0000 UTC m=+901.981108405" Nov 02 17:48:50.722131 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:50.722091 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:51.457974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:51.452716 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:52.029931 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.029894 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl"] Nov 02 17:48:52.035697 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.035665 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" Nov 02 17:48:52.051160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.051122 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4c9298-serving-cert\"" Nov 02 17:48:52.056014 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.055987 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl"] Nov 02 17:48:52.124945 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.124904 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf2kb\" (UniqueName: \"kubernetes.io/projected/316538fa-77cd-4d89-8844-0be9cb16cc5c-kube-api-access-nf2kb\") pod \"maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl\" (UID: \"316538fa-77cd-4d89-8844-0be9cb16cc5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" Nov 02 17:48:52.125156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.125139 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/316538fa-77cd-4d89-8844-0be9cb16cc5c-metrics-certs\") pod \"maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl\" (UID: \"316538fa-77cd-4d89-8844-0be9cb16cc5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" Nov 02 17:48:52.125220 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.125185 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316538fa-77cd-4d89-8844-0be9cb16cc5c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl\" (UID: \"316538fa-77cd-4d89-8844-0be9cb16cc5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" Nov 02 17:48:52.125279 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.125251 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316538fa-77cd-4d89-8844-0be9cb16cc5c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl\" (UID: \"316538fa-77cd-4d89-8844-0be9cb16cc5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" Nov 02 17:48:52.136211 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.136176 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:52.244984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.239011 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf2kb\" (UniqueName: \"kubernetes.io/projected/316538fa-77cd-4d89-8844-0be9cb16cc5c-kube-api-access-nf2kb\") pod \"maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl\" (UID: \"316538fa-77cd-4d89-8844-0be9cb16cc5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" Nov 02 17:48:52.244984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.239239 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/316538fa-77cd-4d89-8844-0be9cb16cc5c-metrics-certs\") pod \"maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl\" (UID: \"316538fa-77cd-4d89-8844-0be9cb16cc5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" Nov 02 17:48:52.244984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.239301 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316538fa-77cd-4d89-8844-0be9cb16cc5c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl\" (UID: \"316538fa-77cd-4d89-8844-0be9cb16cc5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" Nov 02 17:48:52.244984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.239358 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316538fa-77cd-4d89-8844-0be9cb16cc5c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl\" (UID: \"316538fa-77cd-4d89-8844-0be9cb16cc5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" Nov 02 17:48:52.244984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.240892 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316538fa-77cd-4d89-8844-0be9cb16cc5c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl\" (UID: \"316538fa-77cd-4d89-8844-0be9cb16cc5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" Nov 02 17:48:52.264041 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.255200 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf2kb\" (UniqueName: \"kubernetes.io/projected/316538fa-77cd-4d89-8844-0be9cb16cc5c-kube-api-access-nf2kb\") pod \"maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl\" (UID: \"316538fa-77cd-4d89-8844-0be9cb16cc5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" Nov 02 17:48:52.265217 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.265186 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/316538fa-77cd-4d89-8844-0be9cb16cc5c-metrics-certs\") pod \"maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl\" (UID: \"316538fa-77cd-4d89-8844-0be9cb16cc5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" Nov 02 17:48:52.277006 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.275563 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316538fa-77cd-4d89-8844-0be9cb16cc5c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl\" (UID: \"316538fa-77cd-4d89-8844-0be9cb16cc5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" Nov 02 17:48:52.315123 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.308123 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:52.352153 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.352122 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" Nov 02 17:48:52.637990 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.637945 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl"] Nov 02 17:48:52.756160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.753608 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj"] Nov 02 17:48:52.767356 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.765760 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk"] Nov 02 17:48:52.859690 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.859645 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:52.998795 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:52.998763 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:53.176984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.172003 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r"] Nov 02 17:48:53.211516 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.201627 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r"] Nov 02 17:48:53.211516 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.201769 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:48:53.247973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.241375 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d1fc77-serving-cert\"" Nov 02 17:48:53.340260 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.337987 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:53.475101 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.475052 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r\" (UID: \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:48:53.475463 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.475427 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r\" (UID: \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:48:53.476023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.475914 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-metrics-certs\") pod \"maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r\" (UID: \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:48:53.476115 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.476059 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnhwv\" (UniqueName: \"kubernetes.io/projected/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-kube-api-access-cnhwv\") pod \"maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r\" (UID: \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:48:53.502996 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.481231 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" event={"ID":"316538fa-77cd-4d89-8844-0be9cb16cc5c","Type":"ContainerStarted","Data":"fd0a2ba40ea06783b8c7fdbacf1f1c9956997eeebf38d502b975f8c4ddb8c601"} Nov 02 17:48:53.506975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.505064 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:53.577395 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.577365 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnhwv\" (UniqueName: \"kubernetes.io/projected/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-kube-api-access-cnhwv\") pod \"maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r\" (UID: \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:48:53.577523 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.577461 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r\" (UID: \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:48:53.577572 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.577529 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r\" (UID: \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:48:53.577656 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:48:53.577634 5549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-serving-cert: secret "maas-api-e2e-ipp-a-d1fc77-serving-cert" not found Nov 02 17:48:53.577727 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:48:53.577714 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-maas-api-tls podName:e9154805-a9a0-4a01-b44a-1cbb6a3cd88c nodeName:}" failed. No retries permitted until 2026-11-02 17:48:54.077691779 +0000 UTC m=+905.866197285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-maas-api-tls") pod "maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" (UID: "e9154805-a9a0-4a01-b44a-1cbb6a3cd88c") : secret "maas-api-e2e-ipp-a-d1fc77-serving-cert" not found Nov 02 17:48:53.577808 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.577790 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-metrics-certs\") pod \"maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r\" (UID: \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:48:53.578168 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.578146 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r\" (UID: \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:48:53.580263 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.580242 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-metrics-certs\") pod \"maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r\" (UID: \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:48:53.589630 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.589607 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnhwv\" (UniqueName: \"kubernetes.io/projected/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-kube-api-access-cnhwv\") pod \"maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r\" (UID: \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:48:53.832061 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:53.832011 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:54.089771 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:54.087096 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r\" (UID: \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:48:54.107914 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:54.107873 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r\" (UID: \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:48:54.120985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:54.118780 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:48:54.450367 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:54.444623 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:54.511795 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:54.511759 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" event={"ID":"316538fa-77cd-4d89-8844-0be9cb16cc5c","Type":"ContainerStarted","Data":"77a3ab5605a7de74d0b4c5336057b88baaf2f5eec17b61d593a394d5e999741f"} Nov 02 17:48:54.512094 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:54.512076 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" Nov 02 17:48:54.520777 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:54.520750 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:48:54.546135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:54.546095 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r"] Nov 02 17:48:54.546749 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:54.546707 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" podStartSLOduration=1.69035214 podStartE2EDuration="2.546695546s" totalImagesPullingTime="856.343406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:54.541013773 +0000 UTC m=+906.329519295" watchObservedRunningTime="2026-11-02 17:48:54.546695546 +0000 UTC m=+906.335201096" Nov 02 17:48:54.596506 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:54.596475 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" podUID="c629cfcd-ff99-4340-ae7f-dc417b197b43" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:48:54.596605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:54.596519 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" podUID="53edf2f7-a11d-456f-820d-2727ca8bdada" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:48:55.137976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.132297 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:55.271883 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.271789 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:55.525841 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.524228 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" event={"ID":"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c","Type":"ContainerStarted","Data":"90a03347fe25c43a52b3e5029dc06a50798bd4dabac426541f2182a1e31f2561"} Nov 02 17:48:55.525841 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.524295 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" event={"ID":"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c","Type":"ContainerStarted","Data":"52ba38c1766e7b948ba128dd2cdaf15dfd7d87a3215516aee8bc05685652d580"} Nov 02 17:48:55.525841 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.525049 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:48:55.526903 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.526879 5549 generic.go:356] "Generic (PLEG): container finished" podID="cfed89fe-b4e4-41b2-82b0-2ba0d79469de" containerID="b024bd8c3dec58978906753eee86be208dc78fa2e790052279be5351e69c1014" exitCode=0 Nov 02 17:48:55.527182 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.527119 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" event={"ID":"cfed89fe-b4e4-41b2-82b0-2ba0d79469de","Type":"ContainerDied","Data":"b024bd8c3dec58978906753eee86be208dc78fa2e790052279be5351e69c1014"} Nov 02 17:48:55.529541 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:48:55.529505 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfed89fe_b4e4_41b2_82b0_2ba0d79469de.slice/crio-b024bd8c3dec58978906753eee86be208dc78fa2e790052279be5351e69c1014.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfed89fe_b4e4_41b2_82b0_2ba0d79469de.slice/crio-conmon-b024bd8c3dec58978906753eee86be208dc78fa2e790052279be5351e69c1014.scope\": RecentStats: unable to find data in memory cache]" Nov 02 17:48:55.545944 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.545902 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" podStartSLOduration=2.098112241 podStartE2EDuration="2.545889489s" totalImagesPullingTime="447.777248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:55.545439437 +0000 UTC m=+907.333945015" watchObservedRunningTime="2026-11-02 17:48:55.545889489 +0000 UTC m=+907.334395016" Nov 02 17:48:55.651112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.651088 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:55.833593 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.833498 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-data\") pod \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " Nov 02 17:48:55.833593 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.833564 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-credential-socket\") pod \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " Nov 02 17:48:55.833844 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.833655 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-podinfo\") pod \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " Nov 02 17:48:55.833844 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.833771 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istiod-ca-cert\") pod \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " Nov 02 17:48:55.833844 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.833837 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-kube-api-access-9bmt7\" (UniqueName: \"kubernetes.io/projected/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-kube-api-access-9bmt7\") pod \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " Nov 02 17:48:55.834023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.833882 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-workload-socket\") pod \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " Nov 02 17:48:55.834077 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.834028 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-workload-certs\") pod \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " Nov 02 17:48:55.834135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.834105 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-envoy\") pod \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " Nov 02 17:48:55.848079 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.847337 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istiod-ca-cert" pod "cfed89fe-b4e4-41b2-82b0-2ba0d79469de" (UID: "cfed89fe-b4e4-41b2-82b0-2ba0d79469de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:55.848079 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.847564 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-data" pod "cfed89fe-b4e4-41b2-82b0-2ba0d79469de" (UID: "cfed89fe-b4e4-41b2-82b0-2ba0d79469de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:55.848079 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.847705 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-credential-socket" pod "cfed89fe-b4e4-41b2-82b0-2ba0d79469de" (UID: "cfed89fe-b4e4-41b2-82b0-2ba0d79469de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:55.863989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.858262 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-workload-socket" pod "cfed89fe-b4e4-41b2-82b0-2ba0d79469de" (UID: "cfed89fe-b4e4-41b2-82b0-2ba0d79469de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:55.882987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.867249 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-token\" (UniqueName: \"kubernetes.io/projected/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-token\") pod \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\" (UID: \"cfed89fe-b4e4-41b2-82b0-2ba0d79469de\") " Nov 02 17:48:55.882987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.867547 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-workload-certs" pod "cfed89fe-b4e4-41b2-82b0-2ba0d79469de" (UID: "cfed89fe-b4e4-41b2-82b0-2ba0d79469de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:55.882987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.868081 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:55.882987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.868104 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:55.882987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.868121 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:55.882987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.868134 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:55.882987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.868146 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:55.890808 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.890768 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:55.938032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.917482 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-token" pod "cfed89fe-b4e4-41b2-82b0-2ba0d79469de" (UID: "cfed89fe-b4e4-41b2-82b0-2ba0d79469de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:55.938032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.917650 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-envoy" pod "cfed89fe-b4e4-41b2-82b0-2ba0d79469de" (UID: "cfed89fe-b4e4-41b2-82b0-2ba0d79469de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:55.938032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.921272 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-kube-api-access-9bmt7" pod "cfed89fe-b4e4-41b2-82b0-2ba0d79469de" (UID: "cfed89fe-b4e4-41b2-82b0-2ba0d79469de"). InnerVolumeSpecName "kube-api-access-9bmt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:55.938032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.933062 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-podinfo" pod "cfed89fe-b4e4-41b2-82b0-2ba0d79469de" (UID: "cfed89fe-b4e4-41b2-82b0-2ba0d79469de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:48:56.020994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.986113 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:56.020994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.986156 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:56.020994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.986174 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bmt7\" (UniqueName: \"kubernetes.io/projected/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-kube-api-access-9bmt7\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:56.020994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:55.986191 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfed89fe-b4e4-41b2-82b0-2ba0d79469de-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:56.537216 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:56.537184 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" Nov 02 17:48:56.537642 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:56.537214 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2" event={"ID":"cfed89fe-b4e4-41b2-82b0-2ba0d79469de","Type":"ContainerDied","Data":"74c6aba24745b74248a459911959cf406085d215cd012e10de30fce4260583ac"} Nov 02 17:48:56.537642 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:56.537276 5549 scope.go:122] "RemoveContainer" containerID="b024bd8c3dec58978906753eee86be208dc78fa2e790052279be5351e69c1014" Nov 02 17:48:56.559836 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:56.559807 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2"] Nov 02 17:48:56.563297 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:56.563275 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-rv6p2"] Nov 02 17:48:56.777096 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:56.770832 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:57.520981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:57.520631 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:57.754154 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:57.754085 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj" podUID="6b826c10-7d3c-45e0-ae91-b3c47bb14b33" containerName="payload-processing" containerID="cri-o://0c85cadfa580454e577c2dfc85182b4f21afacdd4b0affdbe2aedcbb025724b7" gracePeriod=25 Nov 02 17:48:57.822376 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:57.822333 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk" podUID="bb09e15a-379c-42d4-85cf-a70b727dfcb4" containerName="payload-pre-processing" containerID="cri-o://f60be58566207744d8eb705a58e696943477eb26c95bb7993fa4c12a06dfb8cc" gracePeriod=25 Nov 02 17:48:57.976079 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:57.973593 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq"] Nov 02 17:48:57.976079 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:57.974330 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" podUID="5e5f44cf-7b31-4a72-a1fa-99fa7efb2217" containerName="maas-api" containerID="cri-o://84e2d9add46e1b34c9e01506eca65f1e7b953fd71cebb78f33be1b4edd9a9852" gracePeriod=30 Nov 02 17:48:58.071410 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.071322 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:58.250654 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.228783 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:58.405305 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.403823 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfed89fe-b4e4-41b2-82b0-2ba0d79469de" path="/var/lib/kubelet/pods/cfed89fe-b4e4-41b2-82b0-2ba0d79469de/volumes" Nov 02 17:48:58.418003 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.406249 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk" Nov 02 17:48:58.418003 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.408655 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:58.428315 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.428276 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj" Nov 02 17:48:58.461435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.461404 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb09e15a-379c-42d4-85cf-a70b727dfcb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb09e15a-379c-42d4-85cf-a70b727dfcb4-plugins-config-volume\") pod \"bb09e15a-379c-42d4-85cf-a70b727dfcb4\" (UID: \"bb09e15a-379c-42d4-85cf-a70b727dfcb4\") " Nov 02 17:48:58.461599 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.461523 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb09e15a-379c-42d4-85cf-a70b727dfcb4-kube-api-access-tvfdf\" (UniqueName: \"kubernetes.io/projected/bb09e15a-379c-42d4-85cf-a70b727dfcb4-kube-api-access-tvfdf\") pod \"bb09e15a-379c-42d4-85cf-a70b727dfcb4\" (UID: \"bb09e15a-379c-42d4-85cf-a70b727dfcb4\") " Nov 02 17:48:58.461711 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.461690 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b826c10-7d3c-45e0-ae91-b3c47bb14b33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b826c10-7d3c-45e0-ae91-b3c47bb14b33-plugins-config-volume\") pod \"6b826c10-7d3c-45e0-ae91-b3c47bb14b33\" (UID: \"6b826c10-7d3c-45e0-ae91-b3c47bb14b33\") " Nov 02 17:48:58.461817 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.461792 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b826c10-7d3c-45e0-ae91-b3c47bb14b33-kube-api-access-vxpq7\" (UniqueName: \"kubernetes.io/projected/6b826c10-7d3c-45e0-ae91-b3c47bb14b33-kube-api-access-vxpq7\") pod \"6b826c10-7d3c-45e0-ae91-b3c47bb14b33\" (UID: \"6b826c10-7d3c-45e0-ae91-b3c47bb14b33\") " Nov 02 17:48:58.462378 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.462228 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb09e15a-379c-42d4-85cf-a70b727dfcb4-plugins-config-volume" pod "bb09e15a-379c-42d4-85cf-a70b727dfcb4" (UID: "bb09e15a-379c-42d4-85cf-a70b727dfcb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:58.462509 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.462429 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b826c10-7d3c-45e0-ae91-b3c47bb14b33-plugins-config-volume" pod "6b826c10-7d3c-45e0-ae91-b3c47bb14b33" (UID: "6b826c10-7d3c-45e0-ae91-b3c47bb14b33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:58.465413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.465380 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb09e15a-379c-42d4-85cf-a70b727dfcb4-kube-api-access-tvfdf" pod "bb09e15a-379c-42d4-85cf-a70b727dfcb4" (UID: "bb09e15a-379c-42d4-85cf-a70b727dfcb4"). InnerVolumeSpecName "kube-api-access-tvfdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:58.466140 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.466114 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b826c10-7d3c-45e0-ae91-b3c47bb14b33-kube-api-access-vxpq7" pod "6b826c10-7d3c-45e0-ae91-b3c47bb14b33" (UID: "6b826c10-7d3c-45e0-ae91-b3c47bb14b33"). InnerVolumeSpecName "kube-api-access-vxpq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:58.565326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.565287 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b826c10-7d3c-45e0-ae91-b3c47bb14b33-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:58.570815 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.570779 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxpq7\" (UniqueName: \"kubernetes.io/projected/6b826c10-7d3c-45e0-ae91-b3c47bb14b33-kube-api-access-vxpq7\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:58.571049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.571033 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb09e15a-379c-42d4-85cf-a70b727dfcb4-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:58.571179 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.571165 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvfdf\" (UniqueName: \"kubernetes.io/projected/bb09e15a-379c-42d4-85cf-a70b727dfcb4-kube-api-access-tvfdf\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:58.590063 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.590007 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:58.630500 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.630465 5549 generic.go:356] "Generic (PLEG): container finished" podID="6b826c10-7d3c-45e0-ae91-b3c47bb14b33" containerID="0c85cadfa580454e577c2dfc85182b4f21afacdd4b0affdbe2aedcbb025724b7" exitCode=0 Nov 02 17:48:58.630688 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.630667 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj" Nov 02 17:48:58.630766 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.630667 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj" event={"ID":"6b826c10-7d3c-45e0-ae91-b3c47bb14b33","Type":"ContainerDied","Data":"0c85cadfa580454e577c2dfc85182b4f21afacdd4b0affdbe2aedcbb025724b7"} Nov 02 17:48:58.630826 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.630808 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj" event={"ID":"6b826c10-7d3c-45e0-ae91-b3c47bb14b33","Type":"ContainerDied","Data":"7e5a4755dde82fc844fd554c8f12b15be2371cbe22ae9c4c1ccd01e1c4bfb800"} Nov 02 17:48:58.630889 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.630850 5549 scope.go:122] "RemoveContainer" containerID="0c85cadfa580454e577c2dfc85182b4f21afacdd4b0affdbe2aedcbb025724b7" Nov 02 17:48:58.634634 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.634606 5549 generic.go:356] "Generic (PLEG): container finished" podID="5e5f44cf-7b31-4a72-a1fa-99fa7efb2217" containerID="84e2d9add46e1b34c9e01506eca65f1e7b953fd71cebb78f33be1b4edd9a9852" exitCode=0 Nov 02 17:48:58.634847 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.634755 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" event={"ID":"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217","Type":"ContainerDied","Data":"84e2d9add46e1b34c9e01506eca65f1e7b953fd71cebb78f33be1b4edd9a9852"} Nov 02 17:48:58.636842 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.636820 5549 generic.go:356] "Generic (PLEG): container finished" podID="bb09e15a-379c-42d4-85cf-a70b727dfcb4" containerID="f60be58566207744d8eb705a58e696943477eb26c95bb7993fa4c12a06dfb8cc" exitCode=0 Nov 02 17:48:58.637007 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.636880 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk" event={"ID":"bb09e15a-379c-42d4-85cf-a70b727dfcb4","Type":"ContainerDied","Data":"f60be58566207744d8eb705a58e696943477eb26c95bb7993fa4c12a06dfb8cc"} Nov 02 17:48:58.637165 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.637148 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk" event={"ID":"bb09e15a-379c-42d4-85cf-a70b727dfcb4","Type":"ContainerDied","Data":"ede8404e5dcb53741fd76957645ba87b32f57fe7575865521a5993c77ddffb35"} Nov 02 17:48:58.637282 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.636936 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk" Nov 02 17:48:58.639294 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.639268 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" Nov 02 17:48:58.645633 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.645610 5549 scope.go:122] "RemoveContainer" containerID="0c85cadfa580454e577c2dfc85182b4f21afacdd4b0affdbe2aedcbb025724b7" Nov 02 17:48:58.646221 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:48:58.646192 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c85cadfa580454e577c2dfc85182b4f21afacdd4b0affdbe2aedcbb025724b7\": container with ID starting with 0c85cadfa580454e577c2dfc85182b4f21afacdd4b0affdbe2aedcbb025724b7 not found: ID does not exist" containerID="0c85cadfa580454e577c2dfc85182b4f21afacdd4b0affdbe2aedcbb025724b7" Nov 02 17:48:58.646346 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.646234 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c85cadfa580454e577c2dfc85182b4f21afacdd4b0affdbe2aedcbb025724b7"} err="failed to get container status \"0c85cadfa580454e577c2dfc85182b4f21afacdd4b0affdbe2aedcbb025724b7\": rpc error: code = NotFound desc = could not find container \"0c85cadfa580454e577c2dfc85182b4f21afacdd4b0affdbe2aedcbb025724b7\": container with ID starting with 0c85cadfa580454e577c2dfc85182b4f21afacdd4b0affdbe2aedcbb025724b7 not found: ID does not exist" Nov 02 17:48:58.646346 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.646257 5549 scope.go:122] "RemoveContainer" containerID="f60be58566207744d8eb705a58e696943477eb26c95bb7993fa4c12a06dfb8cc" Nov 02 17:48:58.658039 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.658014 5549 scope.go:122] "RemoveContainer" containerID="f60be58566207744d8eb705a58e696943477eb26c95bb7993fa4c12a06dfb8cc" Nov 02 17:48:58.658299 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:48:58.658278 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f60be58566207744d8eb705a58e696943477eb26c95bb7993fa4c12a06dfb8cc\": container with ID starting with f60be58566207744d8eb705a58e696943477eb26c95bb7993fa4c12a06dfb8cc not found: ID does not exist" containerID="f60be58566207744d8eb705a58e696943477eb26c95bb7993fa4c12a06dfb8cc" Nov 02 17:48:58.658390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.658303 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f60be58566207744d8eb705a58e696943477eb26c95bb7993fa4c12a06dfb8cc"} err="failed to get container status \"f60be58566207744d8eb705a58e696943477eb26c95bb7993fa4c12a06dfb8cc\": rpc error: code = NotFound desc = could not find container \"f60be58566207744d8eb705a58e696943477eb26c95bb7993fa4c12a06dfb8cc\": container with ID starting with f60be58566207744d8eb705a58e696943477eb26c95bb7993fa4c12a06dfb8cc not found: ID does not exist" Nov 02 17:48:58.659522 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.659499 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj"] Nov 02 17:48:58.663679 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.663657 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-11f0a3-956d967fc-5jjsj"] Nov 02 17:48:58.672447 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.672427 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-metrics-certs\") pod \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\" (UID: \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\") " Nov 02 17:48:58.672636 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.672623 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-openshift-service-ca\") pod \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\" (UID: \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\") " Nov 02 17:48:58.672734 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.672723 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-maas-api-tls\") pod \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\" (UID: \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\") " Nov 02 17:48:58.672795 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.672783 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-kube-api-access-kmvd6\" (UniqueName: \"kubernetes.io/projected/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-kube-api-access-kmvd6\") pod \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\" (UID: \"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217\") " Nov 02 17:48:58.673054 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.673018 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-openshift-service-ca" pod "5e5f44cf-7b31-4a72-a1fa-99fa7efb2217" (UID: "5e5f44cf-7b31-4a72-a1fa-99fa7efb2217"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:58.673520 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.673496 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:58.674630 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.674600 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-maas-api-tls" pod "5e5f44cf-7b31-4a72-a1fa-99fa7efb2217" (UID: "5e5f44cf-7b31-4a72-a1fa-99fa7efb2217"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:48:58.674876 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.674847 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-metrics-certs" pod "5e5f44cf-7b31-4a72-a1fa-99fa7efb2217" (UID: "5e5f44cf-7b31-4a72-a1fa-99fa7efb2217"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:48:58.675132 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.675114 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-kube-api-access-kmvd6" pod "5e5f44cf-7b31-4a72-a1fa-99fa7efb2217" (UID: "5e5f44cf-7b31-4a72-a1fa-99fa7efb2217"). InnerVolumeSpecName "kube-api-access-kmvd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:58.696599 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.696566 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk"] Nov 02 17:48:58.701832 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.701802 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-11f0a3-c967574b7-s95gk"] Nov 02 17:48:58.775629 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.775593 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:58.775999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.775630 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:58.775999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:58.775676 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmvd6\" (UniqueName: \"kubernetes.io/projected/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217-kube-api-access-kmvd6\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:59.095360 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.095326 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn"] Nov 02 17:48:59.095608 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.095578 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" podUID="d4b56136-a25b-4705-bcd7-55181db1a642" containerName="maas-api" containerID="cri-o://55dcd23d4491a2474df953d255dd1433c816d8b9f36a0553584dabf65b8f817a" gracePeriod=30 Nov 02 17:48:59.191983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.191936 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:59.600398 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.600367 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" Nov 02 17:48:59.646230 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.646195 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" Nov 02 17:48:59.646388 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.646205 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq" event={"ID":"5e5f44cf-7b31-4a72-a1fa-99fa7efb2217","Type":"ContainerDied","Data":"73cc343b8b401a3d80cd821291b2529dfb59a1d4bd2cfcdb4c9025940221bf71"} Nov 02 17:48:59.646388 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.646340 5549 scope.go:122] "RemoveContainer" containerID="84e2d9add46e1b34c9e01506eca65f1e7b953fd71cebb78f33be1b4edd9a9852" Nov 02 17:48:59.656562 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.656537 5549 generic.go:356] "Generic (PLEG): container finished" podID="d4b56136-a25b-4705-bcd7-55181db1a642" containerID="55dcd23d4491a2474df953d255dd1433c816d8b9f36a0553584dabf65b8f817a" exitCode=0 Nov 02 17:48:59.656685 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.656572 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" event={"ID":"d4b56136-a25b-4705-bcd7-55181db1a642","Type":"ContainerDied","Data":"55dcd23d4491a2474df953d255dd1433c816d8b9f36a0553584dabf65b8f817a"} Nov 02 17:48:59.656685 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.656586 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" Nov 02 17:48:59.656685 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.656606 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn" event={"ID":"d4b56136-a25b-4705-bcd7-55181db1a642","Type":"ContainerDied","Data":"5eaf664625afa82d4a1a22f83c5c34cd813be079de8e0a3ec0dd24fbb0010bf8"} Nov 02 17:48:59.658188 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.658173 5549 scope.go:122] "RemoveContainer" containerID="55dcd23d4491a2474df953d255dd1433c816d8b9f36a0553584dabf65b8f817a" Nov 02 17:48:59.668603 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.668577 5549 scope.go:122] "RemoveContainer" containerID="55dcd23d4491a2474df953d255dd1433c816d8b9f36a0553584dabf65b8f817a" Nov 02 17:48:59.668903 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:48:59.668883 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55dcd23d4491a2474df953d255dd1433c816d8b9f36a0553584dabf65b8f817a\": container with ID starting with 55dcd23d4491a2474df953d255dd1433c816d8b9f36a0553584dabf65b8f817a not found: ID does not exist" containerID="55dcd23d4491a2474df953d255dd1433c816d8b9f36a0553584dabf65b8f817a" Nov 02 17:48:59.669015 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.668910 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55dcd23d4491a2474df953d255dd1433c816d8b9f36a0553584dabf65b8f817a"} err="failed to get container status \"55dcd23d4491a2474df953d255dd1433c816d8b9f36a0553584dabf65b8f817a\": rpc error: code = NotFound desc = could not find container \"55dcd23d4491a2474df953d255dd1433c816d8b9f36a0553584dabf65b8f817a\": container with ID starting with 55dcd23d4491a2474df953d255dd1433c816d8b9f36a0553584dabf65b8f817a not found: ID does not exist" Nov 02 17:48:59.674182 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.674147 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq"] Nov 02 17:48:59.678131 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.678111 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-k9vbq"] Nov 02 17:48:59.702724 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.702700 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:59.713410 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.713390 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4b56136-a25b-4705-bcd7-55181db1a642-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b56136-a25b-4705-bcd7-55181db1a642-maas-api-tls\") pod \"d4b56136-a25b-4705-bcd7-55181db1a642\" (UID: \"d4b56136-a25b-4705-bcd7-55181db1a642\") " Nov 02 17:48:59.713612 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.713597 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4b56136-a25b-4705-bcd7-55181db1a642-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b56136-a25b-4705-bcd7-55181db1a642-openshift-service-ca\") pod \"d4b56136-a25b-4705-bcd7-55181db1a642\" (UID: \"d4b56136-a25b-4705-bcd7-55181db1a642\") " Nov 02 17:48:59.713678 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.713630 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4b56136-a25b-4705-bcd7-55181db1a642-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b56136-a25b-4705-bcd7-55181db1a642-metrics-certs\") pod \"d4b56136-a25b-4705-bcd7-55181db1a642\" (UID: \"d4b56136-a25b-4705-bcd7-55181db1a642\") " Nov 02 17:48:59.713735 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.713679 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b56136-a25b-4705-bcd7-55181db1a642-kube-api-access-zbkhd\" (UniqueName: \"kubernetes.io/projected/d4b56136-a25b-4705-bcd7-55181db1a642-kube-api-access-zbkhd\") pod \"d4b56136-a25b-4705-bcd7-55181db1a642\" (UID: \"d4b56136-a25b-4705-bcd7-55181db1a642\") " Nov 02 17:48:59.713930 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.713907 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b56136-a25b-4705-bcd7-55181db1a642-openshift-service-ca" pod "d4b56136-a25b-4705-bcd7-55181db1a642" (UID: "d4b56136-a25b-4705-bcd7-55181db1a642"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:59.714570 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.714541 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b56136-a25b-4705-bcd7-55181db1a642-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:59.715712 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.715690 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b56136-a25b-4705-bcd7-55181db1a642-maas-api-tls" pod "d4b56136-a25b-4705-bcd7-55181db1a642" (UID: "d4b56136-a25b-4705-bcd7-55181db1a642"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:48:59.715851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.715828 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b56136-a25b-4705-bcd7-55181db1a642-metrics-certs" pod "d4b56136-a25b-4705-bcd7-55181db1a642" (UID: "d4b56136-a25b-4705-bcd7-55181db1a642"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:48:59.715900 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.715854 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b56136-a25b-4705-bcd7-55181db1a642-kube-api-access-zbkhd" pod "d4b56136-a25b-4705-bcd7-55181db1a642" (UID: "d4b56136-a25b-4705-bcd7-55181db1a642"). InnerVolumeSpecName "kube-api-access-zbkhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:59.819611 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.818157 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b56136-a25b-4705-bcd7-55181db1a642-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:59.819611 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.818205 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbkhd\" (UniqueName: \"kubernetes.io/projected/d4b56136-a25b-4705-bcd7-55181db1a642-kube-api-access-zbkhd\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:59.819611 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.818220 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b56136-a25b-4705-bcd7-55181db1a642-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:59.981346 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.981315 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn"] Nov 02 17:48:59.985402 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:48:59.985373 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-11f0a3-c844b6cb8-c5ptn"] Nov 02 17:49:00.320912 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:00.320832 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5f44cf-7b31-4a72-a1fa-99fa7efb2217" path="/var/lib/kubelet/pods/5e5f44cf-7b31-4a72-a1fa-99fa7efb2217/volumes" Nov 02 17:49:00.321319 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:00.321302 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b826c10-7d3c-45e0-ae91-b3c47bb14b33" path="/var/lib/kubelet/pods/6b826c10-7d3c-45e0-ae91-b3c47bb14b33/volumes" Nov 02 17:49:00.321693 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:00.321679 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb09e15a-379c-42d4-85cf-a70b727dfcb4" path="/var/lib/kubelet/pods/bb09e15a-379c-42d4-85cf-a70b727dfcb4/volumes" Nov 02 17:49:00.322397 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:00.322382 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b56136-a25b-4705-bcd7-55181db1a642" path="/var/lib/kubelet/pods/d4b56136-a25b-4705-bcd7-55181db1a642/volumes" Nov 02 17:49:00.438978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:00.436017 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:00.533303 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:00.533275 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" Nov 02 17:49:01.160241 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:01.160207 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw" Nov 02 17:49:01.160641 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:01.160374 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr" Nov 02 17:49:01.160641 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:01.160428 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp" Nov 02 17:49:01.223586 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:01.223538 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:01.543690 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:01.543634 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:49:01.981010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:01.980836 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:02.136305 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:02.136269 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:02.425845 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:02.425710 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:02.608469 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:02.608357 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:02.759993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:02.759946 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:03.083057 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.069111 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:03.325999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.325200 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" Nov 02 17:49:03.378202 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.378176 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c629cfcd-ff99-4340-ae7f-dc417b197b43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c629cfcd-ff99-4340-ae7f-dc417b197b43-plugins-config-volume\") pod \"c629cfcd-ff99-4340-ae7f-dc417b197b43\" (UID: \"c629cfcd-ff99-4340-ae7f-dc417b197b43\") " Nov 02 17:49:03.378482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.378462 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c629cfcd-ff99-4340-ae7f-dc417b197b43-kube-api-access-49jvq\" (UniqueName: \"kubernetes.io/projected/c629cfcd-ff99-4340-ae7f-dc417b197b43-kube-api-access-49jvq\") pod \"c629cfcd-ff99-4340-ae7f-dc417b197b43\" (UID: \"c629cfcd-ff99-4340-ae7f-dc417b197b43\") " Nov 02 17:49:03.378565 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.378548 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c629cfcd-ff99-4340-ae7f-dc417b197b43-plugins-config-volume" pod "c629cfcd-ff99-4340-ae7f-dc417b197b43" (UID: "c629cfcd-ff99-4340-ae7f-dc417b197b43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:49:03.379026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.379006 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c629cfcd-ff99-4340-ae7f-dc417b197b43-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:03.380646 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.380620 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c629cfcd-ff99-4340-ae7f-dc417b197b43-kube-api-access-49jvq" pod "c629cfcd-ff99-4340-ae7f-dc417b197b43" (UID: "c629cfcd-ff99-4340-ae7f-dc417b197b43"). InnerVolumeSpecName "kube-api-access-49jvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:03.384146 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.384098 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" Nov 02 17:49:03.479436 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.479402 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53edf2f7-a11d-456f-820d-2727ca8bdada-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53edf2f7-a11d-456f-820d-2727ca8bdada-plugins-config-volume\") pod \"53edf2f7-a11d-456f-820d-2727ca8bdada\" (UID: \"53edf2f7-a11d-456f-820d-2727ca8bdada\") " Nov 02 17:49:03.479893 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.479568 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53edf2f7-a11d-456f-820d-2727ca8bdada-kube-api-access-mrbqp\" (UniqueName: \"kubernetes.io/projected/53edf2f7-a11d-456f-820d-2727ca8bdada-kube-api-access-mrbqp\") pod \"53edf2f7-a11d-456f-820d-2727ca8bdada\" (UID: \"53edf2f7-a11d-456f-820d-2727ca8bdada\") " Nov 02 17:49:03.479893 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.479854 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53edf2f7-a11d-456f-820d-2727ca8bdada-plugins-config-volume" pod "53edf2f7-a11d-456f-820d-2727ca8bdada" (UID: "53edf2f7-a11d-456f-820d-2727ca8bdada"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:49:03.480191 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.480171 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53edf2f7-a11d-456f-820d-2727ca8bdada-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:03.480240 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.480196 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49jvq\" (UniqueName: \"kubernetes.io/projected/c629cfcd-ff99-4340-ae7f-dc417b197b43-kube-api-access-49jvq\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:03.481551 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.481530 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53edf2f7-a11d-456f-820d-2727ca8bdada-kube-api-access-mrbqp" pod "53edf2f7-a11d-456f-820d-2727ca8bdada" (UID: "53edf2f7-a11d-456f-820d-2727ca8bdada"). InnerVolumeSpecName "kube-api-access-mrbqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:03.593974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.590291 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrbqp\" (UniqueName: \"kubernetes.io/projected/53edf2f7-a11d-456f-820d-2727ca8bdada-kube-api-access-mrbqp\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:03.740980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.732913 5549 generic.go:356] "Generic (PLEG): container finished" podID="c629cfcd-ff99-4340-ae7f-dc417b197b43" containerID="998011eaa59211380d0b820133ece3ca38aae5a650fa4578cb9136dd93c684fb" exitCode=0 Nov 02 17:49:03.740980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.733147 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" event={"ID":"c629cfcd-ff99-4340-ae7f-dc417b197b43","Type":"ContainerDied","Data":"998011eaa59211380d0b820133ece3ca38aae5a650fa4578cb9136dd93c684fb"} Nov 02 17:49:03.740980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.733235 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" event={"ID":"c629cfcd-ff99-4340-ae7f-dc417b197b43","Type":"ContainerDied","Data":"bcc29d7ddca3b57b6500089972f1e344c8e96da8ce761fd7968a5c9a9c8a380f"} Nov 02 17:49:03.740980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.733274 5549 scope.go:122] "RemoveContainer" containerID="998011eaa59211380d0b820133ece3ca38aae5a650fa4578cb9136dd93c684fb" Nov 02 17:49:03.740980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.733471 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j" Nov 02 17:49:03.745998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.745429 5549 generic.go:356] "Generic (PLEG): container finished" podID="53edf2f7-a11d-456f-820d-2727ca8bdada" containerID="f8714b24f0f5d968ff049acc4487c6dcb7ba60dafd26170ccfef397a5738515a" exitCode=0 Nov 02 17:49:03.745998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.745631 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" Nov 02 17:49:03.764378 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.761972 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" event={"ID":"53edf2f7-a11d-456f-820d-2727ca8bdada","Type":"ContainerDied","Data":"f8714b24f0f5d968ff049acc4487c6dcb7ba60dafd26170ccfef397a5738515a"} Nov 02 17:49:03.764378 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.762051 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp" event={"ID":"53edf2f7-a11d-456f-820d-2727ca8bdada","Type":"ContainerDied","Data":"a4febf12ab8d9e0e6f4eb04b463c6557ca975b0c900ea6403ad292a9c4872788"} Nov 02 17:49:03.782566 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.782534 5549 scope.go:122] "RemoveContainer" containerID="998011eaa59211380d0b820133ece3ca38aae5a650fa4578cb9136dd93c684fb" Nov 02 17:49:03.784004 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:49:03.783933 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998011eaa59211380d0b820133ece3ca38aae5a650fa4578cb9136dd93c684fb\": container with ID starting with 998011eaa59211380d0b820133ece3ca38aae5a650fa4578cb9136dd93c684fb not found: ID does not exist" containerID="998011eaa59211380d0b820133ece3ca38aae5a650fa4578cb9136dd93c684fb" Nov 02 17:49:03.784233 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.783993 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998011eaa59211380d0b820133ece3ca38aae5a650fa4578cb9136dd93c684fb"} err="failed to get container status \"998011eaa59211380d0b820133ece3ca38aae5a650fa4578cb9136dd93c684fb\": rpc error: code = NotFound desc = could not find container \"998011eaa59211380d0b820133ece3ca38aae5a650fa4578cb9136dd93c684fb\": container with ID starting with 998011eaa59211380d0b820133ece3ca38aae5a650fa4578cb9136dd93c684fb not found: ID does not exist" Nov 02 17:49:03.784233 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.784213 5549 scope.go:122] "RemoveContainer" containerID="f8714b24f0f5d968ff049acc4487c6dcb7ba60dafd26170ccfef397a5738515a" Nov 02 17:49:03.807993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.807947 5549 scope.go:122] "RemoveContainer" containerID="f8714b24f0f5d968ff049acc4487c6dcb7ba60dafd26170ccfef397a5738515a" Nov 02 17:49:03.808374 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:49:03.808347 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8714b24f0f5d968ff049acc4487c6dcb7ba60dafd26170ccfef397a5738515a\": container with ID starting with f8714b24f0f5d968ff049acc4487c6dcb7ba60dafd26170ccfef397a5738515a not found: ID does not exist" containerID="f8714b24f0f5d968ff049acc4487c6dcb7ba60dafd26170ccfef397a5738515a" Nov 02 17:49:03.808417 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.808387 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8714b24f0f5d968ff049acc4487c6dcb7ba60dafd26170ccfef397a5738515a"} err="failed to get container status \"f8714b24f0f5d968ff049acc4487c6dcb7ba60dafd26170ccfef397a5738515a\": rpc error: code = NotFound desc = could not find container \"f8714b24f0f5d968ff049acc4487c6dcb7ba60dafd26170ccfef397a5738515a\": container with ID starting with f8714b24f0f5d968ff049acc4487c6dcb7ba60dafd26170ccfef397a5738515a not found: ID does not exist" Nov 02 17:49:03.818031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.818003 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j"] Nov 02 17:49:03.822195 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.822172 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-rzl8j"] Nov 02 17:49:03.834665 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.834631 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp"] Nov 02 17:49:03.837618 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:03.837598 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-gp7kp"] Nov 02 17:49:04.007247 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:04.007155 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:04.156271 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:04.156233 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:04.321099 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:04.321019 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53edf2f7-a11d-456f-820d-2727ca8bdada" path="/var/lib/kubelet/pods/53edf2f7-a11d-456f-820d-2727ca8bdada/volumes" Nov 02 17:49:04.321616 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:04.321590 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c629cfcd-ff99-4340-ae7f-dc417b197b43" path="/var/lib/kubelet/pods/c629cfcd-ff99-4340-ae7f-dc417b197b43/volumes" Nov 02 17:49:04.446123 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:04.446086 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:04.643386 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:04.643300 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:05.103978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:05.096220 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:05.661209 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:05.661130 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:05.970056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:05.970019 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:06.107498 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.107461 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:06.380124 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380046 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg"] Nov 02 17:49:06.380677 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380664 5549 cpu_manager.go:414] "containerMap: removing container" podUID="5e5f44cf-7b31-4a72-a1fa-99fa7efb2217" containerName="maas-api" Nov 02 17:49:06.380716 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380679 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5f44cf-7b31-4a72-a1fa-99fa7efb2217" containerName="maas-api" Nov 02 17:49:06.380716 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380706 5549 cpu_manager.go:414] "containerMap: removing container" podUID="d4b56136-a25b-4705-bcd7-55181db1a642" containerName="maas-api" Nov 02 17:49:06.380716 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380712 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b56136-a25b-4705-bcd7-55181db1a642" containerName="maas-api" Nov 02 17:49:06.380817 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380725 5549 cpu_manager.go:414] "containerMap: removing container" podUID="6b826c10-7d3c-45e0-ae91-b3c47bb14b33" containerName="payload-processing" Nov 02 17:49:06.380817 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380730 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b826c10-7d3c-45e0-ae91-b3c47bb14b33" containerName="payload-processing" Nov 02 17:49:06.380817 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380736 5549 cpu_manager.go:414] "containerMap: removing container" podUID="bb09e15a-379c-42d4-85cf-a70b727dfcb4" containerName="payload-pre-processing" Nov 02 17:49:06.380817 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380742 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb09e15a-379c-42d4-85cf-a70b727dfcb4" containerName="payload-pre-processing" Nov 02 17:49:06.380817 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380753 5549 cpu_manager.go:414] "containerMap: removing container" podUID="c629cfcd-ff99-4340-ae7f-dc417b197b43" containerName="payload-pre-processing" Nov 02 17:49:06.380817 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380757 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c629cfcd-ff99-4340-ae7f-dc417b197b43" containerName="payload-pre-processing" Nov 02 17:49:06.380817 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380772 5549 cpu_manager.go:414] "containerMap: removing container" podUID="cfed89fe-b4e4-41b2-82b0-2ba0d79469de" containerName="istio-proxy" Nov 02 17:49:06.380817 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380776 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfed89fe-b4e4-41b2-82b0-2ba0d79469de" containerName="istio-proxy" Nov 02 17:49:06.380817 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380788 5549 cpu_manager.go:414] "containerMap: removing container" podUID="53edf2f7-a11d-456f-820d-2727ca8bdada" containerName="payload-processing" Nov 02 17:49:06.380817 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380793 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53edf2f7-a11d-456f-820d-2727ca8bdada" containerName="payload-processing" Nov 02 17:49:06.381110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380884 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b56136-a25b-4705-bcd7-55181db1a642" containerName="maas-api" Nov 02 17:49:06.381110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380893 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b826c10-7d3c-45e0-ae91-b3c47bb14b33" containerName="payload-processing" Nov 02 17:49:06.381110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380899 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb09e15a-379c-42d4-85cf-a70b727dfcb4" containerName="payload-pre-processing" Nov 02 17:49:06.381110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380908 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c629cfcd-ff99-4340-ae7f-dc417b197b43" containerName="payload-pre-processing" Nov 02 17:49:06.381110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380915 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfed89fe-b4e4-41b2-82b0-2ba0d79469de" containerName="istio-proxy" Nov 02 17:49:06.381110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380922 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="53edf2f7-a11d-456f-820d-2727ca8bdada" containerName="payload-processing" Nov 02 17:49:06.381110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.380929 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5f44cf-7b31-4a72-a1fa-99fa7efb2217" containerName="maas-api" Nov 02 17:49:06.384585 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.384566 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg" Nov 02 17:49:06.392077 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.392052 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg"] Nov 02 17:49:06.394487 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.394463 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-da5f07e6-dockercfg-24wqk\"" Nov 02 17:49:06.394606 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.394556 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-da5f07e6\"" Nov 02 17:49:06.395927 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.395899 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:06.397887 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.397863 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88"] Nov 02 17:49:06.402732 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.402711 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88" Nov 02 17:49:06.421222 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.421191 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88"] Nov 02 17:49:06.512975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.510973 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4l5g\" (UniqueName: \"kubernetes.io/projected/0fa6aa51-6ff3-4de7-b322-e3eafe126eb5-kube-api-access-p4l5g\") pod \"payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88\" (UID: \"0fa6aa51-6ff3-4de7-b322-e3eafe126eb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88" Nov 02 17:49:06.538975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.532450 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:06.538975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.534916 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d1d1ae-9504-4f22-9f9d-3cda90298edb-plugins-config-volume\") pod \"payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg\" (UID: \"37d1d1ae-9504-4f22-9f9d-3cda90298edb\") " pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg" Nov 02 17:49:06.538975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.535135 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa6aa51-6ff3-4de7-b322-e3eafe126eb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88\" (UID: \"0fa6aa51-6ff3-4de7-b322-e3eafe126eb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88" Nov 02 17:49:06.538975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.535195 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q6xz\" (UniqueName: \"kubernetes.io/projected/37d1d1ae-9504-4f22-9f9d-3cda90298edb-kube-api-access-6q6xz\") pod \"payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg\" (UID: \"37d1d1ae-9504-4f22-9f9d-3cda90298edb\") " pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg" Nov 02 17:49:06.645021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.642758 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4l5g\" (UniqueName: \"kubernetes.io/projected/0fa6aa51-6ff3-4de7-b322-e3eafe126eb5-kube-api-access-p4l5g\") pod \"payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88\" (UID: \"0fa6aa51-6ff3-4de7-b322-e3eafe126eb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88" Nov 02 17:49:06.645021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.643488 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d1d1ae-9504-4f22-9f9d-3cda90298edb-plugins-config-volume\") pod \"payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg\" (UID: \"37d1d1ae-9504-4f22-9f9d-3cda90298edb\") " pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg" Nov 02 17:49:06.645021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.645010 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa6aa51-6ff3-4de7-b322-e3eafe126eb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88\" (UID: \"0fa6aa51-6ff3-4de7-b322-e3eafe126eb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88" Nov 02 17:49:06.645313 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.645157 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa6aa51-6ff3-4de7-b322-e3eafe126eb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88\" (UID: \"0fa6aa51-6ff3-4de7-b322-e3eafe126eb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88" Nov 02 17:49:06.645372 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.645317 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6xz\" (UniqueName: \"kubernetes.io/projected/37d1d1ae-9504-4f22-9f9d-3cda90298edb-kube-api-access-6q6xz\") pod \"payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg\" (UID: \"37d1d1ae-9504-4f22-9f9d-3cda90298edb\") " pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg" Nov 02 17:49:06.658975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.646241 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d1d1ae-9504-4f22-9f9d-3cda90298edb-plugins-config-volume\") pod \"payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg\" (UID: \"37d1d1ae-9504-4f22-9f9d-3cda90298edb\") " pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg" Nov 02 17:49:06.680025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.679990 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4l5g\" (UniqueName: \"kubernetes.io/projected/0fa6aa51-6ff3-4de7-b322-e3eafe126eb5-kube-api-access-p4l5g\") pod \"payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88\" (UID: \"0fa6aa51-6ff3-4de7-b322-e3eafe126eb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88" Nov 02 17:49:06.689979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.683782 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q6xz\" (UniqueName: \"kubernetes.io/projected/37d1d1ae-9504-4f22-9f9d-3cda90298edb-kube-api-access-6q6xz\") pod \"payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg\" (UID: \"37d1d1ae-9504-4f22-9f9d-3cda90298edb\") " pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg" Nov 02 17:49:06.705109 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.704279 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg" Nov 02 17:49:06.741007 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:06.729698 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88" Nov 02 17:49:07.151414 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:07.144848 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:07.151414 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:07.144933 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg"] Nov 02 17:49:07.151414 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:07.149830 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88"] Nov 02 17:49:07.158054 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:49:07.158009 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa6aa51_6ff3_4de7_b322_e3eafe126eb5.slice/crio-d9ae485748f9445d6ba883784d7486b851043a0be0ba6900cb46ae67ee1deaac WatchSource:0}: Error finding container d9ae485748f9445d6ba883784d7486b851043a0be0ba6900cb46ae67ee1deaac: Status 404 returned error can't find the container with id d9ae485748f9445d6ba883784d7486b851043a0be0ba6900cb46ae67ee1deaac Nov 02 17:49:07.921973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:07.919698 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg" event={"ID":"37d1d1ae-9504-4f22-9f9d-3cda90298edb","Type":"ContainerStarted","Data":"0926c6fd8c357f2cb404b8f689d0f4d68dcb1ff0d2a91b43d8ede84c37d71c48"} Nov 02 17:49:07.921973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:07.919765 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg" event={"ID":"37d1d1ae-9504-4f22-9f9d-3cda90298edb","Type":"ContainerStarted","Data":"d24ff16af3065039070e1720eeac3721fc2337c00c5466f99752a6e33d50fa6e"} Nov 02 17:49:07.932983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:07.930030 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg" Nov 02 17:49:07.933206 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:07.933166 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88" event={"ID":"0fa6aa51-6ff3-4de7-b322-e3eafe126eb5","Type":"ContainerStarted","Data":"5482cbe0c0b9877b1ae1dfe3da8df8cea4ca1047ff6bf57684f13214eb89f9b5"} Nov 02 17:49:07.950809 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:07.950771 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88" event={"ID":"0fa6aa51-6ff3-4de7-b322-e3eafe126eb5","Type":"ContainerStarted","Data":"d9ae485748f9445d6ba883784d7486b851043a0be0ba6900cb46ae67ee1deaac"} Nov 02 17:49:07.951092 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:07.951073 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88" Nov 02 17:49:08.090385 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:08.090322 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg" podStartSLOduration=2.090301535 podStartE2EDuration="2.090301535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:07.970389059 +0000 UTC m=+919.758894582" watchObservedRunningTime="2026-11-02 17:49:08.090301535 +0000 UTC m=+919.878807064" Nov 02 17:49:08.111083 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:08.111031 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88" podStartSLOduration=2.111016782 podStartE2EDuration="2.111016782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:08.107884915 +0000 UTC m=+919.896390442" watchObservedRunningTime="2026-11-02 17:49:08.111016782 +0000 UTC m=+919.899522310" Nov 02 17:49:08.142414 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:08.142384 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:08.326182 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:08.325542 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:09.216462 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:09.216414 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:09.709062 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:09.709029 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:09.734796 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:09.734762 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq"] Nov 02 17:49:09.740790 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:09.740751 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq" Nov 02 17:49:09.747507 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:09.747482 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-82b305-dockercfg-4lk4f\"" Nov 02 17:49:09.748355 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:09.748331 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq"] Nov 02 17:49:09.788156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:09.788125 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-82b305\"" Nov 02 17:49:09.861420 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:09.861379 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:09.880805 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:09.876023 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705e2065-14fd-430c-9114-941204515fce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq\" (UID: \"705e2065-14fd-430c-9114-941204515fce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq" Nov 02 17:49:09.880805 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:09.876357 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvg88\" (UniqueName: \"kubernetes.io/projected/705e2065-14fd-430c-9114-941204515fce-kube-api-access-rvg88\") pod \"payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq\" (UID: \"705e2065-14fd-430c-9114-941204515fce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq" Nov 02 17:49:09.997307 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:09.980191 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705e2065-14fd-430c-9114-941204515fce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq\" (UID: \"705e2065-14fd-430c-9114-941204515fce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq" Nov 02 17:49:09.997307 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:09.980414 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvg88\" (UniqueName: \"kubernetes.io/projected/705e2065-14fd-430c-9114-941204515fce-kube-api-access-rvg88\") pod \"payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq\" (UID: \"705e2065-14fd-430c-9114-941204515fce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq" Nov 02 17:49:09.997307 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:09.981771 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705e2065-14fd-430c-9114-941204515fce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq\" (UID: \"705e2065-14fd-430c-9114-941204515fce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq" Nov 02 17:49:10.052840 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:10.052796 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvg88\" (UniqueName: \"kubernetes.io/projected/705e2065-14fd-430c-9114-941204515fce-kube-api-access-rvg88\") pod \"payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq\" (UID: \"705e2065-14fd-430c-9114-941204515fce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq" Nov 02 17:49:10.063978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:10.061415 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq" Nov 02 17:49:10.355870 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:10.355749 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:10.414268 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:10.414236 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq"] Nov 02 17:49:10.414846 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:49:10.414814 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705e2065_14fd_430c_9114_941204515fce.slice/crio-581fc81641621306ed21c8eb59511185c2822141b5c642dadeeacaa78e3cbec0 WatchSource:0}: Error finding container 581fc81641621306ed21c8eb59511185c2822141b5c642dadeeacaa78e3cbec0: Status 404 returned error can't find the container with id 581fc81641621306ed21c8eb59511185c2822141b5c642dadeeacaa78e3cbec0 Nov 02 17:49:10.771743 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:10.771706 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw"] Nov 02 17:49:11.020987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:11.020919 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq" event={"ID":"705e2065-14fd-430c-9114-941204515fce","Type":"ContainerStarted","Data":"bee38e7398309abbb3d6bcd9a3ff8e1c55356fcc7459233efde4f70297b52097"} Nov 02 17:49:11.021169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:11.021024 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq" event={"ID":"705e2065-14fd-430c-9114-941204515fce","Type":"ContainerStarted","Data":"581fc81641621306ed21c8eb59511185c2822141b5c642dadeeacaa78e3cbec0"} Nov 02 17:49:11.021233 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:11.021157 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq" Nov 02 17:49:11.043771 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:11.043674 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq" podStartSLOduration=2.043660384 podStartE2EDuration="2.043660384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:11.041098736 +0000 UTC m=+922.829604263" watchObservedRunningTime="2026-11-02 17:49:11.043660384 +0000 UTC m=+922.832165911" Nov 02 17:49:11.678236 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:11.678198 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr"] Nov 02 17:49:11.697696 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:11.697651 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp"] Nov 02 17:49:11.911289 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:11.911248 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:12.080402 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:12.080366 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:12.215912 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:12.215265 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:12.541016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:12.536041 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:13.300728 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.300634 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk"] Nov 02 17:49:13.305772 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.305746 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" Nov 02 17:49:13.314719 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.314701 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-da5f07e6-serving-cert\"" Nov 02 17:49:13.324552 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.324526 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk"] Nov 02 17:49:13.383315 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.383281 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/177815d2-33da-445e-9084-4134446547bd-maas-api-tls\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk\" (UID: \"177815d2-33da-445e-9084-4134446547bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" Nov 02 17:49:13.383529 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.383504 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/177815d2-33da-445e-9084-4134446547bd-openshift-service-ca\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk\" (UID: \"177815d2-33da-445e-9084-4134446547bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" Nov 02 17:49:13.383574 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.383544 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g789k\" (UniqueName: \"kubernetes.io/projected/177815d2-33da-445e-9084-4134446547bd-kube-api-access-g789k\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk\" (UID: \"177815d2-33da-445e-9084-4134446547bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" Nov 02 17:49:13.383665 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.383650 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/177815d2-33da-445e-9084-4134446547bd-metrics-certs\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk\" (UID: \"177815d2-33da-445e-9084-4134446547bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" Nov 02 17:49:13.427525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.427495 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:13.486123 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.485297 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/177815d2-33da-445e-9084-4134446547bd-metrics-certs\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk\" (UID: \"177815d2-33da-445e-9084-4134446547bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" Nov 02 17:49:13.486123 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.485464 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/177815d2-33da-445e-9084-4134446547bd-maas-api-tls\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk\" (UID: \"177815d2-33da-445e-9084-4134446547bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" Nov 02 17:49:13.486123 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.485680 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/177815d2-33da-445e-9084-4134446547bd-openshift-service-ca\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk\" (UID: \"177815d2-33da-445e-9084-4134446547bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" Nov 02 17:49:13.486123 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.485726 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g789k\" (UniqueName: \"kubernetes.io/projected/177815d2-33da-445e-9084-4134446547bd-kube-api-access-g789k\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk\" (UID: \"177815d2-33da-445e-9084-4134446547bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" Nov 02 17:49:13.487927 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.487877 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/177815d2-33da-445e-9084-4134446547bd-openshift-service-ca\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk\" (UID: \"177815d2-33da-445e-9084-4134446547bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" Nov 02 17:49:13.492568 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.492533 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/177815d2-33da-445e-9084-4134446547bd-maas-api-tls\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk\" (UID: \"177815d2-33da-445e-9084-4134446547bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" Nov 02 17:49:13.530738 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.530706 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/177815d2-33da-445e-9084-4134446547bd-metrics-certs\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk\" (UID: \"177815d2-33da-445e-9084-4134446547bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" Nov 02 17:49:13.540977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.540695 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g789k\" (UniqueName: \"kubernetes.io/projected/177815d2-33da-445e-9084-4134446547bd-kube-api-access-g789k\") pod \"maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk\" (UID: \"177815d2-33da-445e-9084-4134446547bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" Nov 02 17:49:13.591946 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.589432 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:13.620897 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.620865 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" Nov 02 17:49:13.899240 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:13.899191 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk"] Nov 02 17:49:13.933868 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:49:13.914123 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177815d2_33da_445e_9084_4134446547bd.slice/crio-e9d0aa8ec5d588f97493a2d9523d3f85c114aef7517b2efb5461d6a4faa89ccb WatchSource:0}: Error finding container e9d0aa8ec5d588f97493a2d9523d3f85c114aef7517b2efb5461d6a4faa89ccb: Status 404 returned error can't find the container with id e9d0aa8ec5d588f97493a2d9523d3f85c114aef7517b2efb5461d6a4faa89ccb Nov 02 17:49:14.108714 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:14.108674 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" event={"ID":"177815d2-33da-445e-9084-4134446547bd","Type":"ContainerStarted","Data":"e9d0aa8ec5d588f97493a2d9523d3f85c114aef7517b2efb5461d6a4faa89ccb"} Nov 02 17:49:14.219646 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:14.219604 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:15.014393 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:15.009475 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:15.119601 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:15.119565 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" event={"ID":"177815d2-33da-445e-9084-4134446547bd","Type":"ContainerStarted","Data":"4e1251835da89da5a9fddeb25225b12d803c4890ffd996fb71a2fa66df963de2"} Nov 02 17:49:15.119791 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:15.119649 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" Nov 02 17:49:15.153862 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:15.153808 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" podStartSLOduration=1.707593341 podStartE2EDuration="2.153793311s" totalImagesPullingTime="446.19997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:15.150268813 +0000 UTC m=+926.938774367" watchObservedRunningTime="2026-11-02 17:49:15.153793311 +0000 UTC m=+926.942298840" Nov 02 17:49:15.571693 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:15.570836 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:15.750622 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:15.745424 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:15.777977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:15.772141 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw" podUID="6428db64-abdc-4a92-8826-016c79a52bd1" containerName="payload-pre-processing" containerID="cri-o://8904b189e9f80553a39c1193dcbf649fe0e06719c582c72a63b2c8cf01069cf2" gracePeriod=25 Nov 02 17:49:16.045476 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.040739 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94"] Nov 02 17:49:16.085236 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.060083 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" Nov 02 17:49:16.085236 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.082582 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-82b305-serving-cert\"" Nov 02 17:49:16.115902 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.115855 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:16.132885 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.132846 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpvn9\" (UniqueName: \"kubernetes.io/projected/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-kube-api-access-gpvn9\") pod \"maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94\" (UID: \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" Nov 02 17:49:16.133107 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.133065 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94\" (UID: \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" Nov 02 17:49:16.133268 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.133248 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-metrics-certs\") pod \"maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94\" (UID: \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" Nov 02 17:49:16.133348 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.133307 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94\" (UID: \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" Nov 02 17:49:16.158851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.150929 5549 generic.go:356] "Generic (PLEG): container finished" podID="6428db64-abdc-4a92-8826-016c79a52bd1" containerID="8904b189e9f80553a39c1193dcbf649fe0e06719c582c72a63b2c8cf01069cf2" exitCode=0 Nov 02 17:49:16.158851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.152075 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw" event={"ID":"6428db64-abdc-4a92-8826-016c79a52bd1","Type":"ContainerDied","Data":"8904b189e9f80553a39c1193dcbf649fe0e06719c582c72a63b2c8cf01069cf2"} Nov 02 17:49:16.187360 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.185058 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94"] Nov 02 17:49:16.291576 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.291518 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:16.293199 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.293156 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-metrics-certs\") pod \"maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94\" (UID: \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" Nov 02 17:49:16.311107 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.311068 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94\" (UID: \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" Nov 02 17:49:16.338113 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.338075 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpvn9\" (UniqueName: \"kubernetes.io/projected/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-kube-api-access-gpvn9\") pod \"maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94\" (UID: \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" Nov 02 17:49:16.338918 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.338887 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94\" (UID: \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" Nov 02 17:49:16.356004 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.341815 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94\" (UID: \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" Nov 02 17:49:16.362115 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.345980 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-metrics-certs\") pod \"maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94\" (UID: \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" Nov 02 17:49:16.380980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.358930 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94\" (UID: \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" Nov 02 17:49:16.402607 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.402431 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpvn9\" (UniqueName: \"kubernetes.io/projected/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-kube-api-access-gpvn9\") pod \"maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94\" (UID: \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" Nov 02 17:49:16.413993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.410128 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" Nov 02 17:49:16.434784 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.434758 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:16.537563 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.537530 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw" Nov 02 17:49:16.648562 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.648481 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6428db64-abdc-4a92-8826-016c79a52bd1-kube-api-access-t2mv7\" (UniqueName: \"kubernetes.io/projected/6428db64-abdc-4a92-8826-016c79a52bd1-kube-api-access-t2mv7\") pod \"6428db64-abdc-4a92-8826-016c79a52bd1\" (UID: \"6428db64-abdc-4a92-8826-016c79a52bd1\") " Nov 02 17:49:16.648730 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.648578 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6428db64-abdc-4a92-8826-016c79a52bd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6428db64-abdc-4a92-8826-016c79a52bd1-plugins-config-volume\") pod \"6428db64-abdc-4a92-8826-016c79a52bd1\" (UID: \"6428db64-abdc-4a92-8826-016c79a52bd1\") " Nov 02 17:49:16.649027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.649000 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6428db64-abdc-4a92-8826-016c79a52bd1-plugins-config-volume" pod "6428db64-abdc-4a92-8826-016c79a52bd1" (UID: "6428db64-abdc-4a92-8826-016c79a52bd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:49:16.649226 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.649210 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6428db64-abdc-4a92-8826-016c79a52bd1-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:16.650553 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.650528 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6428db64-abdc-4a92-8826-016c79a52bd1-kube-api-access-t2mv7" pod "6428db64-abdc-4a92-8826-016c79a52bd1" (UID: "6428db64-abdc-4a92-8826-016c79a52bd1"). InnerVolumeSpecName "kube-api-access-t2mv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:16.715155 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.714979 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp" podUID="3544d84a-2276-4ce3-bda4-3d9c66ee96f6" containerName="payload-pre-processing" containerID="cri-o://08c3d68ef1d1db83486976ab47d28f1ff875dd2dac2e92e9e41834754df2be45" gracePeriod=25 Nov 02 17:49:16.715450 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.715375 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr" podUID="8bfc5a8b-de29-498d-b1ba-e4d4671b82fd" containerName="payload-processing" containerID="cri-o://1218967aa9745e76f664d3f56d01411b355ffd863a449bbdd5bd3842b648566b" gracePeriod=25 Nov 02 17:49:16.773040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.767220 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2mv7\" (UniqueName: \"kubernetes.io/projected/6428db64-abdc-4a92-8826-016c79a52bd1-kube-api-access-t2mv7\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:16.792262 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:16.792214 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94"] Nov 02 17:49:16.856801 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:49:16.852156 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ad8c31_06b1_4604_a74c_92f6b0483fa7.slice/crio-9bba22506f602b87cbc572fa29e4a04cd24633c04a47f8f2b50a061f3a1b638c WatchSource:0}: Error finding container 9bba22506f602b87cbc572fa29e4a04cd24633c04a47f8f2b50a061f3a1b638c: Status 404 returned error can't find the container with id 9bba22506f602b87cbc572fa29e4a04cd24633c04a47f8f2b50a061f3a1b638c Nov 02 17:49:17.178054 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.177428 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp" Nov 02 17:49:17.247793 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.247752 5549 generic.go:356] "Generic (PLEG): container finished" podID="8bfc5a8b-de29-498d-b1ba-e4d4671b82fd" containerID="1218967aa9745e76f664d3f56d01411b355ffd863a449bbdd5bd3842b648566b" exitCode=0 Nov 02 17:49:17.248246 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.247832 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr" event={"ID":"8bfc5a8b-de29-498d-b1ba-e4d4671b82fd","Type":"ContainerDied","Data":"1218967aa9745e76f664d3f56d01411b355ffd863a449bbdd5bd3842b648566b"} Nov 02 17:49:17.254268 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.254241 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp" event={"ID":"3544d84a-2276-4ce3-bda4-3d9c66ee96f6","Type":"ContainerDied","Data":"08c3d68ef1d1db83486976ab47d28f1ff875dd2dac2e92e9e41834754df2be45"} Nov 02 17:49:17.255071 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.251926 5549 generic.go:356] "Generic (PLEG): container finished" podID="3544d84a-2276-4ce3-bda4-3d9c66ee96f6" containerID="08c3d68ef1d1db83486976ab47d28f1ff875dd2dac2e92e9e41834754df2be45" exitCode=0 Nov 02 17:49:17.255071 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.252058 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp" Nov 02 17:49:17.255071 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.254911 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp" event={"ID":"3544d84a-2276-4ce3-bda4-3d9c66ee96f6","Type":"ContainerDied","Data":"b7f505aa9be71953e280265d8b72b8629305519f0cbc06a4ec6863fab23d1336"} Nov 02 17:49:17.255071 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.249644 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:17.260698 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.254474 5549 scope.go:122] "RemoveContainer" containerID="08c3d68ef1d1db83486976ab47d28f1ff875dd2dac2e92e9e41834754df2be45" Nov 02 17:49:17.286880 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.286840 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw" event={"ID":"6428db64-abdc-4a92-8826-016c79a52bd1","Type":"ContainerDied","Data":"02b40b747a7aee2c3353dc1aa053a70753cef7ea17dfc49e912a878d7fe9d831"} Nov 02 17:49:17.287074 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.287004 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw" Nov 02 17:49:17.288803 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.288779 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" event={"ID":"a8ad8c31-06b1-4604-a74c-92f6b0483fa7","Type":"ContainerStarted","Data":"9bba22506f602b87cbc572fa29e4a04cd24633c04a47f8f2b50a061f3a1b638c"} Nov 02 17:49:17.299884 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.299867 5549 scope.go:122] "RemoveContainer" containerID="08c3d68ef1d1db83486976ab47d28f1ff875dd2dac2e92e9e41834754df2be45" Nov 02 17:49:17.300219 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:49:17.300194 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c3d68ef1d1db83486976ab47d28f1ff875dd2dac2e92e9e41834754df2be45\": container with ID starting with 08c3d68ef1d1db83486976ab47d28f1ff875dd2dac2e92e9e41834754df2be45 not found: ID does not exist" containerID="08c3d68ef1d1db83486976ab47d28f1ff875dd2dac2e92e9e41834754df2be45" Nov 02 17:49:17.300313 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.300230 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c3d68ef1d1db83486976ab47d28f1ff875dd2dac2e92e9e41834754df2be45"} err="failed to get container status \"08c3d68ef1d1db83486976ab47d28f1ff875dd2dac2e92e9e41834754df2be45\": rpc error: code = NotFound desc = could not find container \"08c3d68ef1d1db83486976ab47d28f1ff875dd2dac2e92e9e41834754df2be45\": container with ID starting with 08c3d68ef1d1db83486976ab47d28f1ff875dd2dac2e92e9e41834754df2be45 not found: ID does not exist" Nov 02 17:49:17.300313 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.300252 5549 scope.go:122] "RemoveContainer" containerID="8904b189e9f80553a39c1193dcbf649fe0e06719c582c72a63b2c8cf01069cf2" Nov 02 17:49:17.303513 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.303489 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr" Nov 02 17:49:17.315776 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.315719 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw"] Nov 02 17:49:17.319466 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.319442 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-6579fd96d7-4hdhw"] Nov 02 17:49:17.326666 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.326631 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3544d84a-2276-4ce3-bda4-3d9c66ee96f6-kube-api-access-nhfsw\" (UniqueName: \"kubernetes.io/projected/3544d84a-2276-4ce3-bda4-3d9c66ee96f6-kube-api-access-nhfsw\") pod \"3544d84a-2276-4ce3-bda4-3d9c66ee96f6\" (UID: \"3544d84a-2276-4ce3-bda4-3d9c66ee96f6\") " Nov 02 17:49:17.326822 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.326804 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3544d84a-2276-4ce3-bda4-3d9c66ee96f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3544d84a-2276-4ce3-bda4-3d9c66ee96f6-plugins-config-volume\") pod \"3544d84a-2276-4ce3-bda4-3d9c66ee96f6\" (UID: \"3544d84a-2276-4ce3-bda4-3d9c66ee96f6\") " Nov 02 17:49:17.327851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.327815 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3544d84a-2276-4ce3-bda4-3d9c66ee96f6-plugins-config-volume" pod "3544d84a-2276-4ce3-bda4-3d9c66ee96f6" (UID: "3544d84a-2276-4ce3-bda4-3d9c66ee96f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:49:17.329916 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.329882 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3544d84a-2276-4ce3-bda4-3d9c66ee96f6-kube-api-access-nhfsw" pod "3544d84a-2276-4ce3-bda4-3d9c66ee96f6" (UID: "3544d84a-2276-4ce3-bda4-3d9c66ee96f6"). InnerVolumeSpecName "kube-api-access-nhfsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:17.428490 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.428458 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bfc5a8b-de29-498d-b1ba-e4d4671b82fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfc5a8b-de29-498d-b1ba-e4d4671b82fd-plugins-config-volume\") pod \"8bfc5a8b-de29-498d-b1ba-e4d4671b82fd\" (UID: \"8bfc5a8b-de29-498d-b1ba-e4d4671b82fd\") " Nov 02 17:49:17.428580 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.428562 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bfc5a8b-de29-498d-b1ba-e4d4671b82fd-kube-api-access-m7wkb\" (UniqueName: \"kubernetes.io/projected/8bfc5a8b-de29-498d-b1ba-e4d4671b82fd-kube-api-access-m7wkb\") pod \"8bfc5a8b-de29-498d-b1ba-e4d4671b82fd\" (UID: \"8bfc5a8b-de29-498d-b1ba-e4d4671b82fd\") " Nov 02 17:49:17.428913 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.428891 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bfc5a8b-de29-498d-b1ba-e4d4671b82fd-plugins-config-volume" pod "8bfc5a8b-de29-498d-b1ba-e4d4671b82fd" (UID: "8bfc5a8b-de29-498d-b1ba-e4d4671b82fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:49:17.429465 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.429445 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhfsw\" (UniqueName: \"kubernetes.io/projected/3544d84a-2276-4ce3-bda4-3d9c66ee96f6-kube-api-access-nhfsw\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:17.430011 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.429991 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfc5a8b-de29-498d-b1ba-e4d4671b82fd-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:17.430105 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.430018 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3544d84a-2276-4ce3-bda4-3d9c66ee96f6-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:17.431705 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.431684 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfc5a8b-de29-498d-b1ba-e4d4671b82fd-kube-api-access-m7wkb" pod "8bfc5a8b-de29-498d-b1ba-e4d4671b82fd" (UID: "8bfc5a8b-de29-498d-b1ba-e4d4671b82fd"). InnerVolumeSpecName "kube-api-access-m7wkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:17.531309 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.531278 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7wkb\" (UniqueName: \"kubernetes.io/projected/8bfc5a8b-de29-498d-b1ba-e4d4671b82fd-kube-api-access-m7wkb\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:17.611803 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.611709 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp"] Nov 02 17:49:17.616300 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.616270 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7987b86bb7-jhrnp"] Nov 02 17:49:17.620364 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:49:17.620335 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:49:17.676042 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.676000 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:17.829294 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:17.829252 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:18.334977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:18.334325 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:18.473707 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:18.473678 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr" Nov 02 17:49:18.477679 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:18.477651 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3544d84a-2276-4ce3-bda4-3d9c66ee96f6" path="/var/lib/kubelet/pods/3544d84a-2276-4ce3-bda4-3d9c66ee96f6/volumes" Nov 02 17:49:18.478144 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:18.478128 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6428db64-abdc-4a92-8826-016c79a52bd1" path="/var/lib/kubelet/pods/6428db64-abdc-4a92-8826-016c79a52bd1/volumes" Nov 02 17:49:18.478643 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:18.478548 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr" event={"ID":"8bfc5a8b-de29-498d-b1ba-e4d4671b82fd","Type":"ContainerDied","Data":"357eeb520f19693f6361fbba74adc436cf851ddbae4e5fb2c50ca57fd1e924d6"} Nov 02 17:49:18.478643 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:18.478592 5549 scope.go:122] "RemoveContainer" containerID="1218967aa9745e76f664d3f56d01411b355ffd863a449bbdd5bd3842b648566b" Nov 02 17:49:18.511978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:18.488498 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" event={"ID":"a8ad8c31-06b1-4604-a74c-92f6b0483fa7","Type":"ContainerStarted","Data":"3d0530ad80beedac5ad7d6261736b1dd9408a0707eb1591764ecb6b4d22019d4"} Nov 02 17:49:18.514937 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:18.514910 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" Nov 02 17:49:18.549140 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:18.549065 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" podStartSLOduration=3.05100461 podStartE2EDuration="3.549052356s" totalImagesPullingTime="498.047746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:18.546485644 +0000 UTC m=+930.334991172" watchObservedRunningTime="2026-11-02 17:49:18.549052356 +0000 UTC m=+930.337557883" Nov 02 17:49:18.564201 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:18.564160 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr"] Nov 02 17:49:18.567939 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:18.567904 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-55f986b45-fp7zr"] Nov 02 17:49:19.115490 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:19.115455 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:19.842979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:19.840626 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:20.028825 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:20.028791 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88" Nov 02 17:49:20.030733 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:20.030711 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg" Nov 02 17:49:20.344741 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:20.344706 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bfc5a8b-de29-498d-b1ba-e4d4671b82fd" path="/var/lib/kubelet/pods/8bfc5a8b-de29-498d-b1ba-e4d4671b82fd/volumes" Nov 02 17:49:20.540975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:20.537571 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:21.161443 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:21.161405 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" Nov 02 17:49:21.242904 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:21.242846 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:21.958671 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:21.958628 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:22.109245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:22.109196 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq" Nov 02 17:49:22.142824 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:22.142785 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:23.099975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:23.091412 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:23.954802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:23.933813 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:24.589504 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:24.574246 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" Nov 02 17:49:24.957742 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:24.957592 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:25.574977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:25.571058 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:26.439976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:26.439826 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:27.127810 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:27.127774 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:27.807883 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:27.807834 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:27.845021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:27.844980 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:49:28.165627 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.165523 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq"] Nov 02 17:49:28.280169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.280132 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:28.457031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.452164 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:28.611982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.611902 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:28.673045 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.673011 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7347-e38d82"] Nov 02 17:49:28.674119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.674098 5549 cpu_manager.go:414] "containerMap: removing container" podUID="3544d84a-2276-4ce3-bda4-3d9c66ee96f6" containerName="payload-pre-processing" Nov 02 17:49:28.674172 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.674123 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3544d84a-2276-4ce3-bda4-3d9c66ee96f6" containerName="payload-pre-processing" Nov 02 17:49:28.674172 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.674157 5549 cpu_manager.go:414] "containerMap: removing container" podUID="6428db64-abdc-4a92-8826-016c79a52bd1" containerName="payload-pre-processing" Nov 02 17:49:28.674172 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.674165 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6428db64-abdc-4a92-8826-016c79a52bd1" containerName="payload-pre-processing" Nov 02 17:49:28.674272 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.674204 5549 cpu_manager.go:414] "containerMap: removing container" podUID="8bfc5a8b-de29-498d-b1ba-e4d4671b82fd" containerName="payload-processing" Nov 02 17:49:28.674272 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.674213 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bfc5a8b-de29-498d-b1ba-e4d4671b82fd" containerName="payload-processing" Nov 02 17:49:28.674429 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.674413 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3544d84a-2276-4ce3-bda4-3d9c66ee96f6" containerName="payload-pre-processing" Nov 02 17:49:28.674464 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.674443 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6428db64-abdc-4a92-8826-016c79a52bd1" containerName="payload-pre-processing" Nov 02 17:49:28.674464 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.674459 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bfc5a8b-de29-498d-b1ba-e4d4671b82fd" containerName="payload-processing" Nov 02 17:49:28.680275 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.680110 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-e38d82" Nov 02 17:49:28.681738 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.681706 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7347-e38d82"] Nov 02 17:49:28.686211 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.686184 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zf9kl\"" Nov 02 17:49:28.798609 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.798583 5549 generic.go:356] "Generic (PLEG): container finished" podID="a56a6b8e-233b-4256-b47f-48a323ce8f55" containerID="485d45edf1e797dcee8144203100e279ccda4ed50a42bde120abfb90bfda8207" exitCode=0 Nov 02 17:49:28.799488 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.798651 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" event={"ID":"a56a6b8e-233b-4256-b47f-48a323ce8f55","Type":"ContainerDied","Data":"485d45edf1e797dcee8144203100e279ccda4ed50a42bde120abfb90bfda8207"} Nov 02 17:49:28.799488 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.799151 5549 scope.go:122] "RemoveContainer" containerID="485d45edf1e797dcee8144203100e279ccda4ed50a42bde120abfb90bfda8207" Nov 02 17:49:28.801059 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.801028 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:49:28.801141 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.801107 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:49:28.801179 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.801143 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:49:28.808729 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.808705 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dppjl\" (UniqueName: \"kubernetes.io/projected/1332770a-2f7c-4685-9aa2-2bace1e8034c-kube-api-access-dppjl\") pod \"test-curl-7347-e38d82\" (UID: \"1332770a-2f7c-4685-9aa2-2bace1e8034c\") " pod="opendatahub/test-curl-7347-e38d82" Nov 02 17:49:28.867291 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.843847 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:28.946356 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:28.943623 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dppjl\" (UniqueName: \"kubernetes.io/projected/1332770a-2f7c-4685-9aa2-2bace1e8034c-kube-api-access-dppjl\") pod \"test-curl-7347-e38d82\" (UID: \"1332770a-2f7c-4685-9aa2-2bace1e8034c\") " pod="opendatahub/test-curl-7347-e38d82" Nov 02 17:49:29.031976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:29.030311 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:49:29.166330 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:29.161619 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dppjl\" (UniqueName: \"kubernetes.io/projected/1332770a-2f7c-4685-9aa2-2bace1e8034c-kube-api-access-dppjl\") pod \"test-curl-7347-e38d82\" (UID: \"1332770a-2f7c-4685-9aa2-2bace1e8034c\") " pod="opendatahub/test-curl-7347-e38d82" Nov 02 17:49:29.302010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:29.295457 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-e38d82" Nov 02 17:49:29.635008 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:29.634880 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:29.717221 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:29.717190 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7347-e38d82"] Nov 02 17:49:29.717852 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:49:29.717827 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1332770a_2f7c_4685_9aa2_2bace1e8034c.slice/crio-b7c6142492269692b7cfb963baf2b8f1577118181ebae47b660504011c0b5f6e WatchSource:0}: Error finding container b7c6142492269692b7cfb963baf2b8f1577118181ebae47b660504011c0b5f6e: Status 404 returned error can't find the container with id b7c6142492269692b7cfb963baf2b8f1577118181ebae47b660504011c0b5f6e Nov 02 17:49:29.811791 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:29.811757 5549 generic.go:356] "Generic (PLEG): container finished" podID="57b8639c-8c58-4d9a-81c9-f5c262a11884" containerID="08be164b6776ced2d5d1e0c324929647ec7b401de830adb35ab57c5396edff71" exitCode=0 Nov 02 17:49:29.812225 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:29.811841 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" event={"ID":"57b8639c-8c58-4d9a-81c9-f5c262a11884","Type":"ContainerDied","Data":"08be164b6776ced2d5d1e0c324929647ec7b401de830adb35ab57c5396edff71"} Nov 02 17:49:29.812469 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:29.812436 5549 scope.go:122] "RemoveContainer" containerID="08be164b6776ced2d5d1e0c324929647ec7b401de830adb35ab57c5396edff71" Nov 02 17:49:29.814280 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:29.814250 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:49:29.814375 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:29.814338 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:49:29.814428 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:29.814386 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:49:29.815018 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:29.814991 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" event={"ID":"a56a6b8e-233b-4256-b47f-48a323ce8f55","Type":"ContainerStarted","Data":"6b435e4d67a67a5d850dbda594960f74658c4d0e0e02c997672db7d7f641662f"} Nov 02 17:49:29.816624 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:29.816600 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7347-e38d82" event={"ID":"1332770a-2f7c-4685-9aa2-2bace1e8034c","Type":"ContainerStarted","Data":"c654423aaf3e217bdf7ed9013a179ef4e51e936c4dd6dc4cd2f9d50c54a857c5"} Nov 02 17:49:29.816727 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:29.816636 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7347-e38d82" event={"ID":"1332770a-2f7c-4685-9aa2-2bace1e8034c","Type":"ContainerStarted","Data":"b7c6142492269692b7cfb963baf2b8f1577118181ebae47b660504011c0b5f6e"} Nov 02 17:49:29.947974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:29.945084 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7347-e38d82" podStartSLOduration=1.945066044 podStartE2EDuration="1.945066044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:29.884445469 +0000 UTC m=+941.672950993" watchObservedRunningTime="2026-11-02 17:49:29.945066044 +0000 UTC m=+941.733571576" Nov 02 17:49:30.008611 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:30.007996 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:49:30.036986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:30.036871 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:49:30.503568 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:30.503528 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:30.752978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:30.749677 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7347-e38d82"] Nov 02 17:49:30.752978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:30.749841 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7347-e38d82"] Nov 02 17:49:30.841714 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:30.841623 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" event={"ID":"57b8639c-8c58-4d9a-81c9-f5c262a11884","Type":"ContainerStarted","Data":"01c0f7c604eb311a0a9a65be09805f73baf58f862bda570b43cb812149d2d95e"} Nov 02 17:49:31.017978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:31.014105 5549 patch_prober.go:29] interesting pod/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:49:31.017978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:31.014176 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" podUID="a56a6b8e-233b-4256-b47f-48a323ce8f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 17:49:31.202989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:31.202941 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:31.341876 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:31.341842 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:31.365615 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:31.365581 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:49:31.376126 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:31.376089 5549 patch_prober.go:29] interesting pod/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:49:31.376423 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:31.376390 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" podUID="57b8639c-8c58-4d9a-81c9-f5c262a11884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 17:49:31.847974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:31.847912 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7347-e38d82" podUID="1332770a-2f7c-4685-9aa2-2bace1e8034c" containerName="test-curl-7347-e38d82" containerID="cri-o://c654423aaf3e217bdf7ed9013a179ef4e51e936c4dd6dc4cd2f9d50c54a857c5" gracePeriod=2 Nov 02 17:49:31.850505 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:49:31.850477 5549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-e38d82\" is forbidden: User \"system:node:ip-10-0-142-133.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-133.us-east-2.compute.internal' and this object" podUID="1332770a-2f7c-4685-9aa2-2bace1e8034c" pod="opendatahub/test-curl-7347-e38d82" Nov 02 17:49:31.916109 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:31.915436 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:31.994596 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:31.994555 5549 patch_prober.go:29] interesting pod/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:49:31.996807 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:31.996771 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" podUID="a56a6b8e-233b-4256-b47f-48a323ce8f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 17:49:32.364436 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:32.364402 5549 patch_prober.go:29] interesting pod/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:49:32.364595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:32.364449 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" podUID="57b8639c-8c58-4d9a-81c9-f5c262a11884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 17:49:32.749637 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:32.749607 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:32.889377 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:32.889342 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:32.988977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:32.986480 5549 patch_prober.go:29] interesting pod/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:49:32.988977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:32.986537 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" podUID="a56a6b8e-233b-4256-b47f-48a323ce8f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 17:49:33.067409 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.067326 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:33.166089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.166012 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq" podUID="705e2065-14fd-430c-9114-941204515fce" containerName="payload-pre-processing" containerID="cri-o://bee38e7398309abbb3d6bcd9a3ff8e1c55356fcc7459233efde4f70297b52097" gracePeriod=25 Nov 02 17:49:33.365339 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.365199 5549 patch_prober.go:29] interesting pod/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:49:33.365339 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.365259 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" podUID="57b8639c-8c58-4d9a-81c9-f5c262a11884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 17:49:33.456865 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.456829 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:33.540552 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.539768 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf"] Nov 02 17:49:33.541230 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.540913 5549 cpu_manager.go:414] "containerMap: removing container" podUID="1332770a-2f7c-4685-9aa2-2bace1e8034c" containerName="test-curl-7347-e38d82" Nov 02 17:49:33.541230 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.540938 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1332770a-2f7c-4685-9aa2-2bace1e8034c" containerName="test-curl-7347-e38d82" Nov 02 17:49:33.546338 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.541778 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1332770a-2f7c-4685-9aa2-2bace1e8034c" containerName="test-curl-7347-e38d82" Nov 02 17:49:33.546338 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.545039 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" Nov 02 17:49:33.552385 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.552354 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf"] Nov 02 17:49:33.612038 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.611275 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-82b305-dockercfg-bc26l\"" Nov 02 17:49:33.630533 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.630498 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:33.654106 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.654066 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w4dz\" (UniqueName: \"kubernetes.io/projected/89796f12-ff97-4206-af09-e8cf8e06c681-kube-api-access-4w4dz\") pod \"payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf\" (UID: \"89796f12-ff97-4206-af09-e8cf8e06c681\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" Nov 02 17:49:33.661928 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.661499 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89796f12-ff97-4206-af09-e8cf8e06c681-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf\" (UID: \"89796f12-ff97-4206-af09-e8cf8e06c681\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" Nov 02 17:49:33.768074 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.767386 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w4dz\" (UniqueName: \"kubernetes.io/projected/89796f12-ff97-4206-af09-e8cf8e06c681-kube-api-access-4w4dz\") pod \"payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf\" (UID: \"89796f12-ff97-4206-af09-e8cf8e06c681\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" Nov 02 17:49:33.768074 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.767576 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89796f12-ff97-4206-af09-e8cf8e06c681-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf\" (UID: \"89796f12-ff97-4206-af09-e8cf8e06c681\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" Nov 02 17:49:33.774039 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.768711 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89796f12-ff97-4206-af09-e8cf8e06c681-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf\" (UID: \"89796f12-ff97-4206-af09-e8cf8e06c681\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" Nov 02 17:49:33.819978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.819236 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq" Nov 02 17:49:33.996458 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.995086 5549 patch_prober.go:29] interesting pod/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:49:33.996458 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:33.995142 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" podUID="a56a6b8e-233b-4256-b47f-48a323ce8f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 17:49:34.035702 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.022596 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705e2065-14fd-430c-9114-941204515fce-kube-api-access-rvg88\" (UniqueName: \"kubernetes.io/projected/705e2065-14fd-430c-9114-941204515fce-kube-api-access-rvg88\") pod \"705e2065-14fd-430c-9114-941204515fce\" (UID: \"705e2065-14fd-430c-9114-941204515fce\") " Nov 02 17:49:34.035702 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.022674 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/705e2065-14fd-430c-9114-941204515fce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705e2065-14fd-430c-9114-941204515fce-plugins-config-volume\") pod \"705e2065-14fd-430c-9114-941204515fce\" (UID: \"705e2065-14fd-430c-9114-941204515fce\") " Nov 02 17:49:34.035702 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.035610 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705e2065-14fd-430c-9114-941204515fce-plugins-config-volume" pod "705e2065-14fd-430c-9114-941204515fce" (UID: "705e2065-14fd-430c-9114-941204515fce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:49:34.116998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.077679 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w4dz\" (UniqueName: \"kubernetes.io/projected/89796f12-ff97-4206-af09-e8cf8e06c681-kube-api-access-4w4dz\") pod \"payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf\" (UID: \"89796f12-ff97-4206-af09-e8cf8e06c681\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" Nov 02 17:49:34.134654 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.134616 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:34.135369 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.135344 5549 generic.go:356] "Generic (PLEG): container finished" podID="705e2065-14fd-430c-9114-941204515fce" containerID="bee38e7398309abbb3d6bcd9a3ff8e1c55356fcc7459233efde4f70297b52097" exitCode=0 Nov 02 17:49:34.135630 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.135612 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq" event={"ID":"705e2065-14fd-430c-9114-941204515fce","Type":"ContainerDied","Data":"bee38e7398309abbb3d6bcd9a3ff8e1c55356fcc7459233efde4f70297b52097"} Nov 02 17:49:34.135779 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.135764 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq" event={"ID":"705e2065-14fd-430c-9114-941204515fce","Type":"ContainerDied","Data":"581fc81641621306ed21c8eb59511185c2822141b5c642dadeeacaa78e3cbec0"} Nov 02 17:49:34.135890 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.135878 5549 scope.go:122] "RemoveContainer" containerID="bee38e7398309abbb3d6bcd9a3ff8e1c55356fcc7459233efde4f70297b52097" Nov 02 17:49:34.139979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.135895 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705e2065-14fd-430c-9114-941204515fce-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:34.209556 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.140087 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq" Nov 02 17:49:34.209556 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.173320 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" Nov 02 17:49:34.248021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.216218 5549 scope.go:122] "RemoveContainer" containerID="bee38e7398309abbb3d6bcd9a3ff8e1c55356fcc7459233efde4f70297b52097" Nov 02 17:49:34.290012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.287153 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705e2065-14fd-430c-9114-941204515fce-kube-api-access-rvg88" pod "705e2065-14fd-430c-9114-941204515fce" (UID: "705e2065-14fd-430c-9114-941204515fce"). InnerVolumeSpecName "kube-api-access-rvg88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:34.290012 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:49:34.288503 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bee38e7398309abbb3d6bcd9a3ff8e1c55356fcc7459233efde4f70297b52097\": container with ID starting with bee38e7398309abbb3d6bcd9a3ff8e1c55356fcc7459233efde4f70297b52097 not found: ID does not exist" containerID="bee38e7398309abbb3d6bcd9a3ff8e1c55356fcc7459233efde4f70297b52097" Nov 02 17:49:34.290012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.288543 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bee38e7398309abbb3d6bcd9a3ff8e1c55356fcc7459233efde4f70297b52097"} err="failed to get container status \"bee38e7398309abbb3d6bcd9a3ff8e1c55356fcc7459233efde4f70297b52097\": rpc error: code = NotFound desc = could not find container \"bee38e7398309abbb3d6bcd9a3ff8e1c55356fcc7459233efde4f70297b52097\": container with ID starting with bee38e7398309abbb3d6bcd9a3ff8e1c55356fcc7459233efde4f70297b52097 not found: ID does not exist" Nov 02 17:49:34.290012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.289862 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvg88\" (UniqueName: \"kubernetes.io/projected/705e2065-14fd-430c-9114-941204515fce-kube-api-access-rvg88\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:34.364902 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.364864 5549 patch_prober.go:29] interesting pod/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:49:34.365742 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.365193 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" podUID="57b8639c-8c58-4d9a-81c9-f5c262a11884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 17:49:34.555047 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.535858 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq"] Nov 02 17:49:34.566882 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.566834 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-779c546bdc-655nq"] Nov 02 17:49:34.646104 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.646073 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-e38d82" Nov 02 17:49:34.733003 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:49:34.732927 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89796f12_ff97_4206_af09_e8cf8e06c681.slice/crio-7bf52d9e8024a7c68b9bb2ab833eb2d12fa6b811f45be1ba42d6bb91664c955d WatchSource:0}: Error finding container 7bf52d9e8024a7c68b9bb2ab833eb2d12fa6b811f45be1ba42d6bb91664c955d: Status 404 returned error can't find the container with id 7bf52d9e8024a7c68b9bb2ab833eb2d12fa6b811f45be1ba42d6bb91664c955d Nov 02 17:49:34.734996 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.734930 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf"] Nov 02 17:49:34.738471 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.738448 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1332770a-2f7c-4685-9aa2-2bace1e8034c-kube-api-access-dppjl\" (UniqueName: \"kubernetes.io/projected/1332770a-2f7c-4685-9aa2-2bace1e8034c-kube-api-access-dppjl\") pod \"1332770a-2f7c-4685-9aa2-2bace1e8034c\" (UID: \"1332770a-2f7c-4685-9aa2-2bace1e8034c\") " Nov 02 17:49:34.740479 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.740455 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1332770a-2f7c-4685-9aa2-2bace1e8034c-kube-api-access-dppjl" pod "1332770a-2f7c-4685-9aa2-2bace1e8034c" (UID: "1332770a-2f7c-4685-9aa2-2bace1e8034c"). InnerVolumeSpecName "kube-api-access-dppjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:34.840497 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.840454 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dppjl\" (UniqueName: \"kubernetes.io/projected/1332770a-2f7c-4685-9aa2-2bace1e8034c-kube-api-access-dppjl\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:34.944982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.944860 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:34.997031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.992323 5549 patch_prober.go:29] interesting pod/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:49:34.997031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:34.992386 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" podUID="a56a6b8e-233b-4256-b47f-48a323ce8f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 17:49:35.162427 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:35.162143 5549 generic.go:356] "Generic (PLEG): container finished" podID="1332770a-2f7c-4685-9aa2-2bace1e8034c" containerID="c654423aaf3e217bdf7ed9013a179ef4e51e936c4dd6dc4cd2f9d50c54a857c5" exitCode=137 Nov 02 17:49:35.162427 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:35.162211 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-e38d82" Nov 02 17:49:35.162427 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:35.162312 5549 scope.go:122] "RemoveContainer" containerID="c654423aaf3e217bdf7ed9013a179ef4e51e936c4dd6dc4cd2f9d50c54a857c5" Nov 02 17:49:35.165038 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:35.164533 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" event={"ID":"89796f12-ff97-4206-af09-e8cf8e06c681","Type":"ContainerStarted","Data":"7bf52d9e8024a7c68b9bb2ab833eb2d12fa6b811f45be1ba42d6bb91664c955d"} Nov 02 17:49:35.176035 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:49:35.175899 5549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-e38d82\" is forbidden: User \"system:node:ip-10-0-142-133.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-133.us-east-2.compute.internal' and this object" podUID="1332770a-2f7c-4685-9aa2-2bace1e8034c" pod="opendatahub/test-curl-7347-e38d82" Nov 02 17:49:35.176400 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:35.176382 5549 scope.go:122] "RemoveContainer" containerID="c654423aaf3e217bdf7ed9013a179ef4e51e936c4dd6dc4cd2f9d50c54a857c5" Nov 02 17:49:35.176796 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:49:35.176747 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c654423aaf3e217bdf7ed9013a179ef4e51e936c4dd6dc4cd2f9d50c54a857c5\": container with ID starting with c654423aaf3e217bdf7ed9013a179ef4e51e936c4dd6dc4cd2f9d50c54a857c5 not found: ID does not exist" containerID="c654423aaf3e217bdf7ed9013a179ef4e51e936c4dd6dc4cd2f9d50c54a857c5" Nov 02 17:49:35.176866 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:35.176813 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c654423aaf3e217bdf7ed9013a179ef4e51e936c4dd6dc4cd2f9d50c54a857c5"} err="failed to get container status \"c654423aaf3e217bdf7ed9013a179ef4e51e936c4dd6dc4cd2f9d50c54a857c5\": rpc error: code = NotFound desc = could not find container \"c654423aaf3e217bdf7ed9013a179ef4e51e936c4dd6dc4cd2f9d50c54a857c5\": container with ID starting with c654423aaf3e217bdf7ed9013a179ef4e51e936c4dd6dc4cd2f9d50c54a857c5 not found: ID does not exist" Nov 02 17:49:35.364418 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:35.364340 5549 patch_prober.go:29] interesting pod/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:49:35.364418 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:35.364395 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" podUID="57b8639c-8c58-4d9a-81c9-f5c262a11884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 17:49:35.704573 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:35.704469 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:35.987859 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:35.987351 5549 patch_prober.go:29] interesting pod/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:49:35.987859 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:35.987415 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" podUID="a56a6b8e-233b-4256-b47f-48a323ce8f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 17:49:36.207596 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:36.207515 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:36.324728 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:36.324637 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1332770a-2f7c-4685-9aa2-2bace1e8034c" path="/var/lib/kubelet/pods/1332770a-2f7c-4685-9aa2-2bace1e8034c/volumes" Nov 02 17:49:36.325353 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:36.325327 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="705e2065-14fd-430c-9114-941204515fce" path="/var/lib/kubelet/pods/705e2065-14fd-430c-9114-941204515fce/volumes" Nov 02 17:49:36.364499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:36.364466 5549 patch_prober.go:29] interesting pod/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:49:36.364669 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:36.364529 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" podUID="57b8639c-8c58-4d9a-81c9-f5c262a11884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 17:49:36.444371 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:36.444067 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:36.990713 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:36.990682 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:49:36.991088 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:36.991063 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:49:36.991795 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:36.991777 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:49:37.099870 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:37.099780 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:37.242992 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:37.235458 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" event={"ID":"89796f12-ff97-4206-af09-e8cf8e06c681","Type":"ContainerStarted","Data":"d4fecb565292e842da6e3becfd748f26a9a83bf3ea97544fe945f9a46f850f03"} Nov 02 17:49:37.242992 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:37.235551 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" Nov 02 17:49:37.261142 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:37.261086 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" podStartSLOduration=2.243198955 podStartE2EDuration="4.261070061s" totalImagesPullingTime="2.017871106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:37.256431466 +0000 UTC m=+949.044936994" watchObservedRunningTime="2026-11-02 17:49:37.261070061 +0000 UTC m=+949.049575602" Nov 02 17:49:37.380716 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:37.380623 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:49:37.381469 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:37.381317 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:49:37.384414 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:37.384371 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:49:37.750985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:37.747728 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:37.887283 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:37.887249 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:38.052239 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.052149 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46"] Nov 02 17:49:38.053244 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.053224 5549 cpu_manager.go:414] "containerMap: removing container" podUID="705e2065-14fd-430c-9114-941204515fce" containerName="payload-pre-processing" Nov 02 17:49:38.053316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.053246 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705e2065-14fd-430c-9114-941204515fce" containerName="payload-pre-processing" Nov 02 17:49:38.053371 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.053364 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="705e2065-14fd-430c-9114-941204515fce" containerName="payload-pre-processing" Nov 02 17:49:38.057802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.057781 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.067172 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.067145 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46"] Nov 02 17:49:38.067860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.067832 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8f7957-openshift-default-dockercfg-jtn9h\"" Nov 02 17:49:38.135528 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.135492 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-workload-socket\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.135733 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.135547 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-data\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.135733 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.135690 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-podinfo\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.135872 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.135784 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-token\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.135872 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.135847 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-envoy\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.135990 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.135882 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-credential-socket\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.135990 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.135924 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-workload-certs\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.136153 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.136137 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhnc6\" (UniqueName: \"kubernetes.io/projected/36f413ea-e51a-4d63-a07f-0f1a701e6357-kube-api-access-mhnc6\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.136199 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.136170 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f413ea-e51a-4d63-a07f-0f1a701e6357-istiod-ca-cert\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.146562 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.146532 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:38.272563 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.272518 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-token\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.273156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.273129 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-envoy\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.273315 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.273297 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-credential-socket\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.273460 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.273436 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-workload-certs\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.273731 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.273713 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhnc6\" (UniqueName: \"kubernetes.io/projected/36f413ea-e51a-4d63-a07f-0f1a701e6357-kube-api-access-mhnc6\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.273860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.273843 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f413ea-e51a-4d63-a07f-0f1a701e6357-istiod-ca-cert\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.274020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.274002 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-workload-socket\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.274203 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.274185 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-data\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.274360 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.274345 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-podinfo\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.296977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.275304 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-workload-certs\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.296977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.276758 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f413ea-e51a-4d63-a07f-0f1a701e6357-istiod-ca-cert\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.296977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.280140 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-workload-socket\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.296977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.280380 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-credential-socket\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.296977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.280601 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-data\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.298292 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.298253 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-envoy\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.404320 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.404241 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-token\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.405151 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.405121 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-podinfo\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.429979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.429485 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhnc6\" (UniqueName: \"kubernetes.io/projected/36f413ea-e51a-4d63-a07f-0f1a701e6357-kube-api-access-mhnc6\") pod \"e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.601871 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.601773 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:38.695904 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.694255 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:38.769805 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.769765 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:38.920105 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:38.920066 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:39.007641 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:39.007610 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46"] Nov 02 17:49:39.007856 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:49:39.007825 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f413ea_e51a_4d63_a07f_0f1a701e6357.slice/crio-dcb56d396cfad917a59f3cb78faaf194f2a7e25c8f4916317cb8cf863296709c WatchSource:0}: Error finding container dcb56d396cfad917a59f3cb78faaf194f2a7e25c8f4916317cb8cf863296709c: Status 404 returned error can't find the container with id dcb56d396cfad917a59f3cb78faaf194f2a7e25c8f4916317cb8cf863296709c Nov 02 17:49:39.011072 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:39.011033 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:49:39.011160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:39.011118 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:49:39.011218 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:39.011161 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:49:39.217976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:39.217044 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:39.409028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:39.372820 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:39.446907 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:39.446872 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" event={"ID":"36f413ea-e51a-4d63-a07f-0f1a701e6357","Type":"ContainerStarted","Data":"a0fa576d446c3351778e82c3e6ad8b2fed1c7bc88ec3fe03e7a3ba89bd055e7c"} Nov 02 17:49:39.447629 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:39.447598 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" event={"ID":"36f413ea-e51a-4d63-a07f-0f1a701e6357","Type":"ContainerStarted","Data":"dcb56d396cfad917a59f3cb78faaf194f2a7e25c8f4916317cb8cf863296709c"} Nov 02 17:49:39.519662 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:39.519531 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" podStartSLOduration=1.519506419 podStartE2EDuration="1.519506419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:39.505190139 +0000 UTC m=+951.293695680" watchObservedRunningTime="2026-11-02 17:49:39.519506419 +0000 UTC m=+951.308011949" Nov 02 17:49:39.633599 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:39.633562 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:39.700027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:39.696099 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:40.494055 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:40.494015 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:40.705153 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:40.705108 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:41.459864 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:41.459834 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:41.460842 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:41.460819 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:49:41.535556 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:41.535485 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:41.842510 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:41.842438 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:41.999929 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:41.999893 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:42.642569 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:42.642467 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:42.812755 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:42.812720 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:42.990400 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:42.990364 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:43.092445 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:43.092410 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm"] Nov 02 17:49:43.098054 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:43.098028 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm" Nov 02 17:49:43.102811 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:43.102777 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8f7957\"" Nov 02 17:49:43.103741 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:43.103716 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8f7957-dockercfg-lb8fn\"" Nov 02 17:49:43.104377 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:43.104353 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm"] Nov 02 17:49:43.147210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:43.147178 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:43.268044 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:43.265566 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwkmz\" (UniqueName: \"kubernetes.io/projected/d9a8e32b-d358-433c-b026-8324c16bed35-kube-api-access-zwkmz\") pod \"payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm\" (UID: \"d9a8e32b-d358-433c-b026-8324c16bed35\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm" Nov 02 17:49:43.268044 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:43.265625 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a8e32b-d358-433c-b026-8324c16bed35-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm\" (UID: \"d9a8e32b-d358-433c-b026-8324c16bed35\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm" Nov 02 17:49:43.367271 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:43.367229 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwkmz\" (UniqueName: \"kubernetes.io/projected/d9a8e32b-d358-433c-b026-8324c16bed35-kube-api-access-zwkmz\") pod \"payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm\" (UID: \"d9a8e32b-d358-433c-b026-8324c16bed35\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm" Nov 02 17:49:43.367839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:43.367799 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a8e32b-d358-433c-b026-8324c16bed35-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm\" (UID: \"d9a8e32b-d358-433c-b026-8324c16bed35\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm" Nov 02 17:49:43.369060 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:43.369023 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a8e32b-d358-433c-b026-8324c16bed35-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm\" (UID: \"d9a8e32b-d358-433c-b026-8324c16bed35\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm" Nov 02 17:49:43.413044 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:43.410732 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwkmz\" (UniqueName: \"kubernetes.io/projected/d9a8e32b-d358-433c-b026-8324c16bed35-kube-api-access-zwkmz\") pod \"payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm\" (UID: \"d9a8e32b-d358-433c-b026-8324c16bed35\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm" Nov 02 17:49:43.423977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:43.420346 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm" Nov 02 17:49:43.746765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:43.746732 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm"] Nov 02 17:49:43.747613 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:49:43.747590 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a8e32b_d358_433c_b026_8324c16bed35.slice/crio-f4dd25933e2a06f14137558cee1ec9383d09e34753dc5edbdf2fb55e565a2b1f WatchSource:0}: Error finding container f4dd25933e2a06f14137558cee1ec9383d09e34753dc5edbdf2fb55e565a2b1f: Status 404 returned error can't find the container with id f4dd25933e2a06f14137558cee1ec9383d09e34753dc5edbdf2fb55e565a2b1f Nov 02 17:49:43.797800 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:43.788017 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:44.096665 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:44.096579 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:44.258975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:44.258854 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:44.501824 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:44.501785 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm" event={"ID":"d9a8e32b-d358-433c-b026-8324c16bed35","Type":"ContainerStarted","Data":"37d9ab54d593266d277b6222601fe7f83efdac51fcd212dfb3accd87f71ef094"} Nov 02 17:49:44.502059 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:44.501848 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm" event={"ID":"d9a8e32b-d358-433c-b026-8324c16bed35","Type":"ContainerStarted","Data":"f4dd25933e2a06f14137558cee1ec9383d09e34753dc5edbdf2fb55e565a2b1f"} Nov 02 17:49:44.502059 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:44.502009 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm" Nov 02 17:49:44.520876 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:44.520823 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm" podStartSLOduration=1.520810276 podStartE2EDuration="1.520810276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:44.518218715 +0000 UTC m=+956.306724255" watchObservedRunningTime="2026-11-02 17:49:44.520810276 +0000 UTC m=+956.309315804" Nov 02 17:49:44.669822 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:44.659116 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:45.545721 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:45.541598 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:46.556985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:46.552145 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:47.449875 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:47.449801 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:47.587187 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:49:47.587152 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:49:48.324111 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:48.320736 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:48.623287 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:48.623191 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:48.767407 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:48.767378 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:49.150987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:49.145573 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:49.447915 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:49.447887 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" Nov 02 17:49:49.848033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:49.847785 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:50.799123 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:50.799089 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv"] Nov 02 17:49:50.806107 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:50.806073 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" Nov 02 17:49:50.815303 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:50.815266 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv"] Nov 02 17:49:50.816315 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:50.816234 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8f7957-serving-cert\"" Nov 02 17:49:50.941317 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:50.941278 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:51.002983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:51.002003 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btv9k\" (UniqueName: \"kubernetes.io/projected/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-kube-api-access-btv9k\") pod \"maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv\" (UID: \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" Nov 02 17:49:51.002983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:51.002205 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv\" (UID: \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" Nov 02 17:49:51.002983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:51.002430 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-metrics-certs\") pod \"maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv\" (UID: \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" Nov 02 17:49:51.002983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:51.002488 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv\" (UID: \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" Nov 02 17:49:51.095368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:51.095282 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:51.103834 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:51.103798 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-metrics-certs\") pod \"maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv\" (UID: \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" Nov 02 17:49:51.104017 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:51.103853 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv\" (UID: \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" Nov 02 17:49:51.104017 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:51.103994 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btv9k\" (UniqueName: \"kubernetes.io/projected/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-kube-api-access-btv9k\") pod \"maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv\" (UID: \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" Nov 02 17:49:51.104173 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:51.104078 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv\" (UID: \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" Nov 02 17:49:51.104628 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:51.104603 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv\" (UID: \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" Nov 02 17:49:51.106373 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:51.106353 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-metrics-certs\") pod \"maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv\" (UID: \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" Nov 02 17:49:51.106497 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:51.106448 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv\" (UID: \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" Nov 02 17:49:51.114610 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:51.114588 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btv9k\" (UniqueName: \"kubernetes.io/projected/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-kube-api-access-btv9k\") pod \"maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv\" (UID: \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" Nov 02 17:49:51.123562 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:51.123537 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" Nov 02 17:49:51.246106 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:51.246038 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:51.496027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:51.476607 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv"] Nov 02 17:49:51.567400 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:51.566991 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" event={"ID":"454db7ea-3a85-4bb3-aa88-5f9dd74466ce","Type":"ContainerStarted","Data":"13aa0369c7ccd96b7154152298a8531dcc57effaad04f7c2687938d521099eb6"} Nov 02 17:49:51.779045 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:51.778697 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:52.576232 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:52.576194 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" event={"ID":"454db7ea-3a85-4bb3-aa88-5f9dd74466ce","Type":"ContainerStarted","Data":"b71c12ac3236aaf9fd5526150bdecfd3588901233ee66377a7a842cabe86df3e"} Nov 02 17:49:52.576648 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:52.576253 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" Nov 02 17:49:52.597506 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:52.597457 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" podStartSLOduration=2.138448366 podStartE2EDuration="2.597443312s" totalImagesPullingTime="458.994946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:52.593684093 +0000 UTC m=+964.382189623" watchObservedRunningTime="2026-11-02 17:49:52.597443312 +0000 UTC m=+964.385948839" Nov 02 17:49:52.739694 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:52.739641 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:52.882449 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:52.882348 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:53.050179 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:53.050146 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:53.649359 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:53.649323 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:54.130530 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:54.130483 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:54.612371 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:54.612341 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:55.060256 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:55.060221 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:55.360557 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:55.360483 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:55.512727 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:55.512696 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm" Nov 02 17:49:55.536439 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:55.536407 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:55.703076 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:55.697190 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:56.290974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:56.289870 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:57.022972 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:57.019500 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:57.812981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:57.810785 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:58.134636 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:58.134543 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:58.283869 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:58.283833 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:58.589181 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:58.589152 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" Nov 02 17:49:58.679917 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:58.679879 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:59.223185 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:59.223128 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:59.375222 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:59.375187 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:59.522164 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:49:59.522076 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:00.378286 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:00.378252 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:00.437360 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:00.437313 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff"] Nov 02 17:50:00.441877 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:00.441855 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff" Nov 02 17:50:00.456904 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:00.456876 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c8832b\"" Nov 02 17:50:00.458484 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:00.457738 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff"] Nov 02 17:50:00.522175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:00.522133 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:00.528023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:00.527996 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8916879a-b154-4fe3-aa47-08d4ce041577-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff\" (UID: \"8916879a-b154-4fe3-aa47-08d4ce041577\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff" Nov 02 17:50:00.528189 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:00.528039 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vppcf\" (UniqueName: \"kubernetes.io/projected/8916879a-b154-4fe3-aa47-08d4ce041577-kube-api-access-vppcf\") pod \"payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff\" (UID: \"8916879a-b154-4fe3-aa47-08d4ce041577\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff" Nov 02 17:50:00.677030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:00.639129 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8916879a-b154-4fe3-aa47-08d4ce041577-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff\" (UID: \"8916879a-b154-4fe3-aa47-08d4ce041577\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff" Nov 02 17:50:00.677372 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:00.677333 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vppcf\" (UniqueName: \"kubernetes.io/projected/8916879a-b154-4fe3-aa47-08d4ce041577-kube-api-access-vppcf\") pod \"payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff\" (UID: \"8916879a-b154-4fe3-aa47-08d4ce041577\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff" Nov 02 17:50:00.707613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:00.707575 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vppcf\" (UniqueName: \"kubernetes.io/projected/8916879a-b154-4fe3-aa47-08d4ce041577-kube-api-access-vppcf\") pod \"payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff\" (UID: \"8916879a-b154-4fe3-aa47-08d4ce041577\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff" Nov 02 17:50:00.760844 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:00.760611 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8916879a-b154-4fe3-aa47-08d4ce041577-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff\" (UID: \"8916879a-b154-4fe3-aa47-08d4ce041577\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff" Nov 02 17:50:01.060284 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:01.060255 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff" Nov 02 17:50:01.344729 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:01.344702 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff"] Nov 02 17:50:01.345720 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:50:01.345699 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8916879a_b154_4fe3_aa47_08d4ce041577.slice/crio-7e9428f6712a55a54d7c10ac1f5c51f67d4ea72c0dc792e8977c0a2e765b460b WatchSource:0}: Error finding container 7e9428f6712a55a54d7c10ac1f5c51f67d4ea72c0dc792e8977c0a2e765b460b: Status 404 returned error can't find the container with id 7e9428f6712a55a54d7c10ac1f5c51f67d4ea72c0dc792e8977c0a2e765b460b Nov 02 17:50:01.390566 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:01.390530 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:01.547980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:01.547937 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:01.888470 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:01.888356 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff" event={"ID":"8916879a-b154-4fe3-aa47-08d4ce041577","Type":"ContainerStarted","Data":"e7b86702b8e2149857d7819597c826ff84630f7e6dbaf9340da10339b918cb16"} Nov 02 17:50:01.888470 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:01.888433 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff" event={"ID":"8916879a-b154-4fe3-aa47-08d4ce041577","Type":"ContainerStarted","Data":"7e9428f6712a55a54d7c10ac1f5c51f67d4ea72c0dc792e8977c0a2e765b460b"} Nov 02 17:50:01.889859 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:01.889833 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff" Nov 02 17:50:01.921997 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:01.917732 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff" podStartSLOduration=1.91770921 podStartE2EDuration="1.91770921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:01.914169672 +0000 UTC m=+973.702675200" watchObservedRunningTime="2026-11-02 17:50:01.91770921 +0000 UTC m=+973.706214739" Nov 02 17:50:02.368905 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:02.368863 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:03.397647 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:03.397606 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:04.250016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:04.249978 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:05.020826 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:05.020105 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:05.776589 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:05.776549 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:06.566520 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:06.566479 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:07.015113 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:07.015078 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm"] Nov 02 17:50:07.129539 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:07.129506 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:07.293128 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:07.293045 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:07.607555 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:07.607453 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:08.006318 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:08.006270 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:08.059580 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:08.059548 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75"] Nov 02 17:50:08.064640 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:08.064612 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f187069-8tk75"] Nov 02 17:50:08.190625 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:08.190587 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:08.370695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:08.370610 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8566552-a7ce-4ed8-aab9-2d6b42309265" path="/var/lib/kubelet/pods/c8566552-a7ce-4ed8-aab9-2d6b42309265/volumes" Nov 02 17:50:08.371915 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:08.371861 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:08.907158 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:08.907127 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:09.056647 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:09.056602 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:09.235179 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:09.235145 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:09.264029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:09.264000 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:50:09.828976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:09.820251 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:10.914135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:10.914040 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:11.702971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:11.702910 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:12.015797 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:12.015711 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm" podUID="d9a8e32b-d358-433c-b026-8324c16bed35" containerName="payload-pre-processing" containerID="cri-o://37d9ab54d593266d277b6222601fe7f83efdac51fcd212dfb3accd87f71ef094" gracePeriod=25 Nov 02 17:50:12.332727 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:12.332675 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm" Nov 02 17:50:12.413674 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:12.413633 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a8e32b-d358-433c-b026-8324c16bed35-kube-api-access-zwkmz\" (UniqueName: \"kubernetes.io/projected/d9a8e32b-d358-433c-b026-8324c16bed35-kube-api-access-zwkmz\") pod \"d9a8e32b-d358-433c-b026-8324c16bed35\" (UID: \"d9a8e32b-d358-433c-b026-8324c16bed35\") " Nov 02 17:50:12.413873 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:12.413700 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9a8e32b-d358-433c-b026-8324c16bed35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a8e32b-d358-433c-b026-8324c16bed35-plugins-config-volume\") pod \"d9a8e32b-d358-433c-b026-8324c16bed35\" (UID: \"d9a8e32b-d358-433c-b026-8324c16bed35\") " Nov 02 17:50:12.415100 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:12.415067 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a8e32b-d358-433c-b026-8324c16bed35-plugins-config-volume" pod "d9a8e32b-d358-433c-b026-8324c16bed35" (UID: "d9a8e32b-d358-433c-b026-8324c16bed35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:50:12.416539 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:12.416514 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a8e32b-d358-433c-b026-8324c16bed35-kube-api-access-zwkmz" pod "d9a8e32b-d358-433c-b026-8324c16bed35" (UID: "d9a8e32b-d358-433c-b026-8324c16bed35"). InnerVolumeSpecName "kube-api-access-zwkmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:12.516713 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:12.516674 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwkmz\" (UniqueName: \"kubernetes.io/projected/d9a8e32b-d358-433c-b026-8324c16bed35-kube-api-access-zwkmz\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:12.516713 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:12.516707 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a8e32b-d358-433c-b026-8324c16bed35-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:12.740977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:12.739286 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:12.941020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:12.928010 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:12.943868 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:12.943836 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg"] Nov 02 17:50:12.961571 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:12.961464 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88"] Nov 02 17:50:13.053938 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:13.053865 5549 generic.go:356] "Generic (PLEG): container finished" podID="d9a8e32b-d358-433c-b026-8324c16bed35" containerID="37d9ab54d593266d277b6222601fe7f83efdac51fcd212dfb3accd87f71ef094" exitCode=0 Nov 02 17:50:13.054413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:13.054059 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm" Nov 02 17:50:13.054838 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:13.054077 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm" event={"ID":"d9a8e32b-d358-433c-b026-8324c16bed35","Type":"ContainerDied","Data":"37d9ab54d593266d277b6222601fe7f83efdac51fcd212dfb3accd87f71ef094"} Nov 02 17:50:13.055083 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:13.055062 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm" event={"ID":"d9a8e32b-d358-433c-b026-8324c16bed35","Type":"ContainerDied","Data":"f4dd25933e2a06f14137558cee1ec9383d09e34753dc5edbdf2fb55e565a2b1f"} Nov 02 17:50:13.055219 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:13.055204 5549 scope.go:122] "RemoveContainer" containerID="37d9ab54d593266d277b6222601fe7f83efdac51fcd212dfb3accd87f71ef094" Nov 02 17:50:13.082247 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:13.082223 5549 scope.go:122] "RemoveContainer" containerID="37d9ab54d593266d277b6222601fe7f83efdac51fcd212dfb3accd87f71ef094" Nov 02 17:50:13.117459 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:50:13.115348 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d9ab54d593266d277b6222601fe7f83efdac51fcd212dfb3accd87f71ef094\": container with ID starting with 37d9ab54d593266d277b6222601fe7f83efdac51fcd212dfb3accd87f71ef094 not found: ID does not exist" containerID="37d9ab54d593266d277b6222601fe7f83efdac51fcd212dfb3accd87f71ef094" Nov 02 17:50:13.117459 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:13.115399 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d9ab54d593266d277b6222601fe7f83efdac51fcd212dfb3accd87f71ef094"} err="failed to get container status \"37d9ab54d593266d277b6222601fe7f83efdac51fcd212dfb3accd87f71ef094\": rpc error: code = NotFound desc = could not find container \"37d9ab54d593266d277b6222601fe7f83efdac51fcd212dfb3accd87f71ef094\": container with ID starting with 37d9ab54d593266d277b6222601fe7f83efdac51fcd212dfb3accd87f71ef094 not found: ID does not exist" Nov 02 17:50:13.118306 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:13.118271 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:13.122503 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:13.122479 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm"] Nov 02 17:50:13.125135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:13.125109 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7d69c77b67-jzvcm"] Nov 02 17:50:13.305518 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:13.305432 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:13.655033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:13.653834 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:13.930561 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:13.928443 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:13.984976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:13.983120 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff" Nov 02 17:50:13.999973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:13.996262 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5"] Nov 02 17:50:14.052978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:14.048566 5549 cpu_manager.go:414] "containerMap: removing container" podUID="d9a8e32b-d358-433c-b026-8324c16bed35" containerName="payload-pre-processing" Nov 02 17:50:14.052978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:14.048594 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a8e32b-d358-433c-b026-8324c16bed35" containerName="payload-pre-processing" Nov 02 17:50:14.052978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:14.048774 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a8e32b-d358-433c-b026-8324c16bed35" containerName="payload-pre-processing" Nov 02 17:50:14.428107 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:14.427549 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5"] Nov 02 17:50:14.438155 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:14.438124 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5" Nov 02 17:50:14.445202 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:14.443855 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8f7957-dockercfg-xbk65\"" Nov 02 17:50:14.457503 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:14.457411 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8f7957\"" Nov 02 17:50:14.462227 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:14.462112 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a8e32b-d358-433c-b026-8324c16bed35" path="/var/lib/kubelet/pods/d9a8e32b-d358-433c-b026-8324c16bed35/volumes" Nov 02 17:50:14.546989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:14.544025 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca344ab-a3fb-498c-9bc9-703ea0816597-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5\" (UID: \"7ca344ab-a3fb-498c-9bc9-703ea0816597\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5" Nov 02 17:50:14.546989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:14.544457 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxnjm\" (UniqueName: \"kubernetes.io/projected/7ca344ab-a3fb-498c-9bc9-703ea0816597-kube-api-access-hxnjm\") pod \"payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5\" (UID: \"7ca344ab-a3fb-498c-9bc9-703ea0816597\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5" Nov 02 17:50:14.585575 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:14.576682 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:14.649605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:14.649559 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxnjm\" (UniqueName: \"kubernetes.io/projected/7ca344ab-a3fb-498c-9bc9-703ea0816597-kube-api-access-hxnjm\") pod \"payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5\" (UID: \"7ca344ab-a3fb-498c-9bc9-703ea0816597\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5" Nov 02 17:50:14.649782 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:14.649685 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca344ab-a3fb-498c-9bc9-703ea0816597-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5\" (UID: \"7ca344ab-a3fb-498c-9bc9-703ea0816597\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5" Nov 02 17:50:14.650662 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:14.650639 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca344ab-a3fb-498c-9bc9-703ea0816597-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5\" (UID: \"7ca344ab-a3fb-498c-9bc9-703ea0816597\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5" Nov 02 17:50:14.661350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:14.661322 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxnjm\" (UniqueName: \"kubernetes.io/projected/7ca344ab-a3fb-498c-9bc9-703ea0816597-kube-api-access-hxnjm\") pod \"payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5\" (UID: \"7ca344ab-a3fb-498c-9bc9-703ea0816597\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5" Nov 02 17:50:14.767624 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:14.767587 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5" Nov 02 17:50:14.972077 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:14.972041 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:15.078290 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:15.078260 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5"] Nov 02 17:50:15.078837 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:50:15.078812 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca344ab_a3fb_498c_9bc9_703ea0816597.slice/crio-4c296f9b4fe9fb2f7c598d6d9adc9172e2df5665dc2fa04777febec2cab3416f WatchSource:0}: Error finding container 4c296f9b4fe9fb2f7c598d6d9adc9172e2df5665dc2fa04777febec2cab3416f: Status 404 returned error can't find the container with id 4c296f9b4fe9fb2f7c598d6d9adc9172e2df5665dc2fa04777febec2cab3416f Nov 02 17:50:15.468095 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:15.468059 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5" event={"ID":"7ca344ab-a3fb-498c-9bc9-703ea0816597","Type":"ContainerStarted","Data":"ae8496ae029403bd64bcc2340a77acd8d868e7aa26db2829c24eb9e6304888f1"} Nov 02 17:50:15.468501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:15.468109 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5" event={"ID":"7ca344ab-a3fb-498c-9bc9-703ea0816597","Type":"ContainerStarted","Data":"4c296f9b4fe9fb2f7c598d6d9adc9172e2df5665dc2fa04777febec2cab3416f"} Nov 02 17:50:15.468501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:15.468175 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5" Nov 02 17:50:15.510765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:15.510692 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5" podStartSLOduration=2.510670528 podStartE2EDuration="2.510670528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:15.499360011 +0000 UTC m=+987.287865540" watchObservedRunningTime="2026-11-02 17:50:15.510670528 +0000 UTC m=+987.299176058" Nov 02 17:50:15.515234 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:15.515206 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:16.048156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.048123 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:16.117126 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.117005 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd"] Nov 02 17:50:16.123609 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.123569 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd"] Nov 02 17:50:16.123778 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.123692 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd" Nov 02 17:50:16.131091 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.131062 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-dd1cba-dockercfg-g6ddp\"" Nov 02 17:50:16.132440 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.128469 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-dd1cba\"" Nov 02 17:50:16.151876 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.151840 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b"] Nov 02 17:50:16.158007 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.157983 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b" Nov 02 17:50:16.165028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.165002 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b"] Nov 02 17:50:16.249974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.247799 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:16.280975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.279005 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n25c\" (UniqueName: \"kubernetes.io/projected/6a551174-8eda-4593-9c92-8e3a143c9a5b-kube-api-access-4n25c\") pod \"payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd\" (UID: \"6a551174-8eda-4593-9c92-8e3a143c9a5b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd" Nov 02 17:50:16.280975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.279131 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdca28da-5265-43b5-93b6-71d3d4239a6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b\" (UID: \"fdca28da-5265-43b5-93b6-71d3d4239a6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b" Nov 02 17:50:16.280975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.279285 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a551174-8eda-4593-9c92-8e3a143c9a5b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd\" (UID: \"6a551174-8eda-4593-9c92-8e3a143c9a5b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd" Nov 02 17:50:16.280975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.279384 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgxrl\" (UniqueName: \"kubernetes.io/projected/fdca28da-5265-43b5-93b6-71d3d4239a6d-kube-api-access-dgxrl\") pod \"payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b\" (UID: \"fdca28da-5265-43b5-93b6-71d3d4239a6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b" Nov 02 17:50:16.460976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.450661 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a551174-8eda-4593-9c92-8e3a143c9a5b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd\" (UID: \"6a551174-8eda-4593-9c92-8e3a143c9a5b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd" Nov 02 17:50:16.460976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.450824 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgxrl\" (UniqueName: \"kubernetes.io/projected/fdca28da-5265-43b5-93b6-71d3d4239a6d-kube-api-access-dgxrl\") pod \"payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b\" (UID: \"fdca28da-5265-43b5-93b6-71d3d4239a6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b" Nov 02 17:50:16.460976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.450986 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n25c\" (UniqueName: \"kubernetes.io/projected/6a551174-8eda-4593-9c92-8e3a143c9a5b-kube-api-access-4n25c\") pod \"payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd\" (UID: \"6a551174-8eda-4593-9c92-8e3a143c9a5b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd" Nov 02 17:50:16.460976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.451112 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdca28da-5265-43b5-93b6-71d3d4239a6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b\" (UID: \"fdca28da-5265-43b5-93b6-71d3d4239a6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b" Nov 02 17:50:16.460976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.452204 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdca28da-5265-43b5-93b6-71d3d4239a6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b\" (UID: \"fdca28da-5265-43b5-93b6-71d3d4239a6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b" Nov 02 17:50:16.509339 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.509231 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n25c\" (UniqueName: \"kubernetes.io/projected/6a551174-8eda-4593-9c92-8e3a143c9a5b-kube-api-access-4n25c\") pod \"payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd\" (UID: \"6a551174-8eda-4593-9c92-8e3a143c9a5b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd" Nov 02 17:50:16.512441 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.512336 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgxrl\" (UniqueName: \"kubernetes.io/projected/fdca28da-5265-43b5-93b6-71d3d4239a6d-kube-api-access-dgxrl\") pod \"payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b\" (UID: \"fdca28da-5265-43b5-93b6-71d3d4239a6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b" Nov 02 17:50:16.572633 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.572598 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a551174-8eda-4593-9c92-8e3a143c9a5b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd\" (UID: \"6a551174-8eda-4593-9c92-8e3a143c9a5b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd" Nov 02 17:50:16.687428 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.687385 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b" Nov 02 17:50:16.741062 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.730625 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:16.813786 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:16.813636 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd" Nov 02 17:50:17.095672 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:17.095629 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b"] Nov 02 17:50:17.096497 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:50:17.096473 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdca28da_5265_43b5_93b6_71d3d4239a6d.slice/crio-659e3277aea0fd2874046600452f312cc22031631d4297192d079c5afc0d975e WatchSource:0}: Error finding container 659e3277aea0fd2874046600452f312cc22031631d4297192d079c5afc0d975e: Status 404 returned error can't find the container with id 659e3277aea0fd2874046600452f312cc22031631d4297192d079c5afc0d975e Nov 02 17:50:17.144743 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:17.144719 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd"] Nov 02 17:50:17.146051 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:50:17.146017 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a551174_8eda_4593_9c92_8e3a143c9a5b.slice/crio-75f7f11ac1e9fd0c525cb6d6f09067b186ac6a5b2e599489498ab3cf9e310622 WatchSource:0}: Error finding container 75f7f11ac1e9fd0c525cb6d6f09067b186ac6a5b2e599489498ab3cf9e310622: Status 404 returned error can't find the container with id 75f7f11ac1e9fd0c525cb6d6f09067b186ac6a5b2e599489498ab3cf9e310622 Nov 02 17:50:17.593159 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:17.593106 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd" event={"ID":"6a551174-8eda-4593-9c92-8e3a143c9a5b","Type":"ContainerStarted","Data":"a0f3d562ba9cd670af528209d19e06103c4ad4c61e4fcc5c9de9eba2544e7bcf"} Nov 02 17:50:17.594896 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:17.594849 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd" event={"ID":"6a551174-8eda-4593-9c92-8e3a143c9a5b","Type":"ContainerStarted","Data":"75f7f11ac1e9fd0c525cb6d6f09067b186ac6a5b2e599489498ab3cf9e310622"} Nov 02 17:50:17.595129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:17.595112 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd" Nov 02 17:50:17.595464 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:50:17.595439 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:50:17.595610 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:17.595590 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b" event={"ID":"fdca28da-5265-43b5-93b6-71d3d4239a6d","Type":"ContainerStarted","Data":"93ceae6108e77527d4607e0eab6584faaba5b44612cd688d01bcf39bd213a87e"} Nov 02 17:50:17.595684 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:17.595631 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b" event={"ID":"fdca28da-5265-43b5-93b6-71d3d4239a6d","Type":"ContainerStarted","Data":"659e3277aea0fd2874046600452f312cc22031631d4297192d079c5afc0d975e"} Nov 02 17:50:17.595739 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:17.595725 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b" Nov 02 17:50:17.634492 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:17.634456 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:17.652979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:17.646381 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd" podStartSLOduration=1.6463607059999998 podStartE2EDuration="1.646360706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:17.63394823 +0000 UTC m=+989.422453760" watchObservedRunningTime="2026-11-02 17:50:17.646360706 +0000 UTC m=+989.434866236" Nov 02 17:50:17.686978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:17.670871 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b" podStartSLOduration=1.670840339 podStartE2EDuration="1.670840339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:17.655029316 +0000 UTC m=+989.443534833" watchObservedRunningTime="2026-11-02 17:50:17.670840339 +0000 UTC m=+989.459345868" Nov 02 17:50:17.945531 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:17.945496 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg" podUID="37d1d1ae-9504-4f22-9f9d-3cda90298edb" containerName="payload-processing" containerID="cri-o://0926c6fd8c357f2cb404b8f689d0f4d68dcb1ff0d2a91b43d8ede84c37d71c48" gracePeriod=25 Nov 02 17:50:17.962490 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:17.962455 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88" podUID="0fa6aa51-6ff3-4de7-b322-e3eafe126eb5" containerName="payload-pre-processing" containerID="cri-o://5482cbe0c0b9877b1ae1dfe3da8df8cea4ca1047ff6bf57684f13214eb89f9b5" gracePeriod=25 Nov 02 17:50:18.290499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.290455 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg" Nov 02 17:50:18.368434 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.368093 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88" Nov 02 17:50:18.418072 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.415964 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:18.487975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.484417 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa6aa51-6ff3-4de7-b322-e3eafe126eb5-kube-api-access-p4l5g\" (UniqueName: \"kubernetes.io/projected/0fa6aa51-6ff3-4de7-b322-e3eafe126eb5-kube-api-access-p4l5g\") pod \"0fa6aa51-6ff3-4de7-b322-e3eafe126eb5\" (UID: \"0fa6aa51-6ff3-4de7-b322-e3eafe126eb5\") " Nov 02 17:50:18.487975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.484567 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d1d1ae-9504-4f22-9f9d-3cda90298edb-kube-api-access-6q6xz\" (UniqueName: \"kubernetes.io/projected/37d1d1ae-9504-4f22-9f9d-3cda90298edb-kube-api-access-6q6xz\") pod \"37d1d1ae-9504-4f22-9f9d-3cda90298edb\" (UID: \"37d1d1ae-9504-4f22-9f9d-3cda90298edb\") " Nov 02 17:50:18.487975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.484626 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d1d1ae-9504-4f22-9f9d-3cda90298edb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d1d1ae-9504-4f22-9f9d-3cda90298edb-plugins-config-volume\") pod \"37d1d1ae-9504-4f22-9f9d-3cda90298edb\" (UID: \"37d1d1ae-9504-4f22-9f9d-3cda90298edb\") " Nov 02 17:50:18.487975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.484659 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa6aa51-6ff3-4de7-b322-e3eafe126eb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa6aa51-6ff3-4de7-b322-e3eafe126eb5-plugins-config-volume\") pod \"0fa6aa51-6ff3-4de7-b322-e3eafe126eb5\" (UID: \"0fa6aa51-6ff3-4de7-b322-e3eafe126eb5\") " Nov 02 17:50:18.487975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.486637 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d1d1ae-9504-4f22-9f9d-3cda90298edb-plugins-config-volume" pod "37d1d1ae-9504-4f22-9f9d-3cda90298edb" (UID: "37d1d1ae-9504-4f22-9f9d-3cda90298edb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:50:18.487975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.486818 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa6aa51-6ff3-4de7-b322-e3eafe126eb5-plugins-config-volume" pod "0fa6aa51-6ff3-4de7-b322-e3eafe126eb5" (UID: "0fa6aa51-6ff3-4de7-b322-e3eafe126eb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:50:18.516036 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.492321 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa6aa51-6ff3-4de7-b322-e3eafe126eb5-kube-api-access-p4l5g" pod "0fa6aa51-6ff3-4de7-b322-e3eafe126eb5" (UID: "0fa6aa51-6ff3-4de7-b322-e3eafe126eb5"). InnerVolumeSpecName "kube-api-access-p4l5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:18.516036 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.492468 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d1d1ae-9504-4f22-9f9d-3cda90298edb-kube-api-access-6q6xz" pod "37d1d1ae-9504-4f22-9f9d-3cda90298edb" (UID: "37d1d1ae-9504-4f22-9f9d-3cda90298edb"). InnerVolumeSpecName "kube-api-access-6q6xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:18.586941 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.586898 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4l5g\" (UniqueName: \"kubernetes.io/projected/0fa6aa51-6ff3-4de7-b322-e3eafe126eb5-kube-api-access-p4l5g\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:18.586941 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.586941 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q6xz\" (UniqueName: \"kubernetes.io/projected/37d1d1ae-9504-4f22-9f9d-3cda90298edb-kube-api-access-6q6xz\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:18.586941 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.587002 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d1d1ae-9504-4f22-9f9d-3cda90298edb-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:18.587317 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.587018 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa6aa51-6ff3-4de7-b322-e3eafe126eb5-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:18.608789 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.608745 5549 generic.go:356] "Generic (PLEG): container finished" podID="37d1d1ae-9504-4f22-9f9d-3cda90298edb" containerID="0926c6fd8c357f2cb404b8f689d0f4d68dcb1ff0d2a91b43d8ede84c37d71c48" exitCode=0 Nov 02 17:50:18.609302 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.608868 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg" event={"ID":"37d1d1ae-9504-4f22-9f9d-3cda90298edb","Type":"ContainerDied","Data":"0926c6fd8c357f2cb404b8f689d0f4d68dcb1ff0d2a91b43d8ede84c37d71c48"} Nov 02 17:50:18.609302 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.608939 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg" event={"ID":"37d1d1ae-9504-4f22-9f9d-3cda90298edb","Type":"ContainerDied","Data":"d24ff16af3065039070e1720eeac3721fc2337c00c5466f99752a6e33d50fa6e"} Nov 02 17:50:18.609302 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.609010 5549 scope.go:122] "RemoveContainer" containerID="0926c6fd8c357f2cb404b8f689d0f4d68dcb1ff0d2a91b43d8ede84c37d71c48" Nov 02 17:50:18.609302 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.609218 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg" Nov 02 17:50:18.613701 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.613494 5549 generic.go:356] "Generic (PLEG): container finished" podID="0fa6aa51-6ff3-4de7-b322-e3eafe126eb5" containerID="5482cbe0c0b9877b1ae1dfe3da8df8cea4ca1047ff6bf57684f13214eb89f9b5" exitCode=0 Nov 02 17:50:18.613988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.613533 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88" event={"ID":"0fa6aa51-6ff3-4de7-b322-e3eafe126eb5","Type":"ContainerDied","Data":"5482cbe0c0b9877b1ae1dfe3da8df8cea4ca1047ff6bf57684f13214eb89f9b5"} Nov 02 17:50:18.614119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.614050 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88" event={"ID":"0fa6aa51-6ff3-4de7-b322-e3eafe126eb5","Type":"ContainerDied","Data":"d9ae485748f9445d6ba883784d7486b851043a0be0ba6900cb46ae67ee1deaac"} Nov 02 17:50:18.614119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.613574 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88" Nov 02 17:50:18.636321 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.636298 5549 scope.go:122] "RemoveContainer" containerID="0926c6fd8c357f2cb404b8f689d0f4d68dcb1ff0d2a91b43d8ede84c37d71c48" Nov 02 17:50:18.636708 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:50:18.636686 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0926c6fd8c357f2cb404b8f689d0f4d68dcb1ff0d2a91b43d8ede84c37d71c48\": container with ID starting with 0926c6fd8c357f2cb404b8f689d0f4d68dcb1ff0d2a91b43d8ede84c37d71c48 not found: ID does not exist" containerID="0926c6fd8c357f2cb404b8f689d0f4d68dcb1ff0d2a91b43d8ede84c37d71c48" Nov 02 17:50:18.636777 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.636717 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0926c6fd8c357f2cb404b8f689d0f4d68dcb1ff0d2a91b43d8ede84c37d71c48"} err="failed to get container status \"0926c6fd8c357f2cb404b8f689d0f4d68dcb1ff0d2a91b43d8ede84c37d71c48\": rpc error: code = NotFound desc = could not find container \"0926c6fd8c357f2cb404b8f689d0f4d68dcb1ff0d2a91b43d8ede84c37d71c48\": container with ID starting with 0926c6fd8c357f2cb404b8f689d0f4d68dcb1ff0d2a91b43d8ede84c37d71c48 not found: ID does not exist" Nov 02 17:50:18.636777 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.636738 5549 scope.go:122] "RemoveContainer" containerID="5482cbe0c0b9877b1ae1dfe3da8df8cea4ca1047ff6bf57684f13214eb89f9b5" Nov 02 17:50:18.646637 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.646600 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg"] Nov 02 17:50:18.649632 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.649609 5549 scope.go:122] "RemoveContainer" containerID="5482cbe0c0b9877b1ae1dfe3da8df8cea4ca1047ff6bf57684f13214eb89f9b5" Nov 02 17:50:18.649991 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:50:18.649933 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5482cbe0c0b9877b1ae1dfe3da8df8cea4ca1047ff6bf57684f13214eb89f9b5\": container with ID starting with 5482cbe0c0b9877b1ae1dfe3da8df8cea4ca1047ff6bf57684f13214eb89f9b5 not found: ID does not exist" containerID="5482cbe0c0b9877b1ae1dfe3da8df8cea4ca1047ff6bf57684f13214eb89f9b5" Nov 02 17:50:18.649991 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.649977 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5482cbe0c0b9877b1ae1dfe3da8df8cea4ca1047ff6bf57684f13214eb89f9b5"} err="failed to get container status \"5482cbe0c0b9877b1ae1dfe3da8df8cea4ca1047ff6bf57684f13214eb89f9b5\": rpc error: code = NotFound desc = could not find container \"5482cbe0c0b9877b1ae1dfe3da8df8cea4ca1047ff6bf57684f13214eb89f9b5\": container with ID starting with 5482cbe0c0b9877b1ae1dfe3da8df8cea4ca1047ff6bf57684f13214eb89f9b5 not found: ID does not exist" Nov 02 17:50:18.652045 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.652022 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-da5f07e6-6f55cbb87f-wwqcg"] Nov 02 17:50:18.665015 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.663768 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88"] Nov 02 17:50:18.670590 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:18.670564 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-da5f07e6-6f5c8bcf8c-ptm88"] Nov 02 17:50:19.160407 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:19.160291 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:19.853627 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:19.853586 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:20.029136 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:20.029106 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns"] Nov 02 17:50:20.034623 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:20.034587 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c14ce4b-6xvns"] Nov 02 17:50:20.325711 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:20.325670 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fa6aa51-6ff3-4de7-b322-e3eafe126eb5" path="/var/lib/kubelet/pods/0fa6aa51-6ff3-4de7-b322-e3eafe126eb5/volumes" Nov 02 17:50:20.326280 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:20.326259 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d1d1ae-9504-4f22-9f9d-3cda90298edb" path="/var/lib/kubelet/pods/37d1d1ae-9504-4f22-9f9d-3cda90298edb/volumes" Nov 02 17:50:20.326818 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:20.326801 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bce8b771-3728-48e2-a7c5-23a7e1decd4a" path="/var/lib/kubelet/pods/bce8b771-3728-48e2-a7c5-23a7e1decd4a/volumes" Nov 02 17:50:20.675025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:20.664879 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:20.889556 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:20.889522 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:21.480019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:21.469613 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:21.637164 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:21.637072 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:21.698057 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:21.698027 5549 generic.go:356] "Generic (PLEG): container finished" podID="36f413ea-e51a-4d63-a07f-0f1a701e6357" containerID="a0fa576d446c3351778e82c3e6ad8b2fed1c7bc88ec3fe03e7a3ba89bd055e7c" exitCode=0 Nov 02 17:50:21.698236 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:21.698084 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" event={"ID":"36f413ea-e51a-4d63-a07f-0f1a701e6357","Type":"ContainerDied","Data":"a0fa576d446c3351778e82c3e6ad8b2fed1c7bc88ec3fe03e7a3ba89bd055e7c"} Nov 02 17:50:21.698539 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:21.698524 5549 scope.go:122] "RemoveContainer" containerID="a0fa576d446c3351778e82c3e6ad8b2fed1c7bc88ec3fe03e7a3ba89bd055e7c" Nov 02 17:50:21.700266 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:21.700235 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:50:21.700342 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:21.700305 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:50:21.700342 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:21.700339 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:50:22.025495 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.025459 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf"] Nov 02 17:50:22.031259 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.031233 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0162571e-6tgxf"] Nov 02 17:50:22.055499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.055459 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk"] Nov 02 17:50:22.055724 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.055697 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" podUID="177815d2-33da-445e-9084-4134446547bd" containerName="maas-api" containerID="cri-o://4e1251835da89da5a9fddeb25225b12d803c4890ffd996fb71a2fa66df963de2" gracePeriod=30 Nov 02 17:50:22.152236 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.152206 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:22.554046 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.530665 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2" path="/var/lib/kubelet/pods/4b2e5b0d-d9c9-4e0c-a04e-ef0f0e1eb8d2/volumes" Nov 02 17:50:22.603170 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.603142 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" Nov 02 17:50:22.720014 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.718877 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/177815d2-33da-445e-9084-4134446547bd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/177815d2-33da-445e-9084-4134446547bd-maas-api-tls\") pod \"177815d2-33da-445e-9084-4134446547bd\" (UID: \"177815d2-33da-445e-9084-4134446547bd\") " Nov 02 17:50:22.720014 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.719024 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/177815d2-33da-445e-9084-4134446547bd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/177815d2-33da-445e-9084-4134446547bd-openshift-service-ca\") pod \"177815d2-33da-445e-9084-4134446547bd\" (UID: \"177815d2-33da-445e-9084-4134446547bd\") " Nov 02 17:50:22.720014 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.719166 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/177815d2-33da-445e-9084-4134446547bd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/177815d2-33da-445e-9084-4134446547bd-metrics-certs\") pod \"177815d2-33da-445e-9084-4134446547bd\" (UID: \"177815d2-33da-445e-9084-4134446547bd\") " Nov 02 17:50:22.720014 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.719219 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/177815d2-33da-445e-9084-4134446547bd-kube-api-access-g789k\" (UniqueName: \"kubernetes.io/projected/177815d2-33da-445e-9084-4134446547bd-kube-api-access-g789k\") pod \"177815d2-33da-445e-9084-4134446547bd\" (UID: \"177815d2-33da-445e-9084-4134446547bd\") " Nov 02 17:50:22.723705 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.723658 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/177815d2-33da-445e-9084-4134446547bd-openshift-service-ca" pod "177815d2-33da-445e-9084-4134446547bd" (UID: "177815d2-33da-445e-9084-4134446547bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:50:22.725142 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.725115 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/177815d2-33da-445e-9084-4134446547bd-kube-api-access-g789k" pod "177815d2-33da-445e-9084-4134446547bd" (UID: "177815d2-33da-445e-9084-4134446547bd"). InnerVolumeSpecName "kube-api-access-g789k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:22.725996 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.725941 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177815d2-33da-445e-9084-4134446547bd-metrics-certs" pod "177815d2-33da-445e-9084-4134446547bd" (UID: "177815d2-33da-445e-9084-4134446547bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:50:22.726706 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.726685 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177815d2-33da-445e-9084-4134446547bd-maas-api-tls" pod "177815d2-33da-445e-9084-4134446547bd" (UID: "177815d2-33da-445e-9084-4134446547bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:50:22.729774 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.729118 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" event={"ID":"36f413ea-e51a-4d63-a07f-0f1a701e6357","Type":"ContainerStarted","Data":"b1517fb466caa9b047f2e65911c9c24762e79ca7305a99d324c2658da080d820"} Nov 02 17:50:22.731296 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.731273 5549 generic.go:356] "Generic (PLEG): container finished" podID="177815d2-33da-445e-9084-4134446547bd" containerID="4e1251835da89da5a9fddeb25225b12d803c4890ffd996fb71a2fa66df963de2" exitCode=0 Nov 02 17:50:22.731370 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.731344 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" Nov 02 17:50:22.731370 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.731359 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" event={"ID":"177815d2-33da-445e-9084-4134446547bd","Type":"ContainerDied","Data":"4e1251835da89da5a9fddeb25225b12d803c4890ffd996fb71a2fa66df963de2"} Nov 02 17:50:22.731450 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.731421 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk" event={"ID":"177815d2-33da-445e-9084-4134446547bd","Type":"ContainerDied","Data":"e9d0aa8ec5d588f97493a2d9523d3f85c114aef7517b2efb5461d6a4faa89ccb"} Nov 02 17:50:22.731450 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.731443 5549 scope.go:122] "RemoveContainer" containerID="4e1251835da89da5a9fddeb25225b12d803c4890ffd996fb71a2fa66df963de2" Nov 02 17:50:22.764141 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.764119 5549 scope.go:122] "RemoveContainer" containerID="4e1251835da89da5a9fddeb25225b12d803c4890ffd996fb71a2fa66df963de2" Nov 02 17:50:22.764482 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:50:22.764460 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1251835da89da5a9fddeb25225b12d803c4890ffd996fb71a2fa66df963de2\": container with ID starting with 4e1251835da89da5a9fddeb25225b12d803c4890ffd996fb71a2fa66df963de2 not found: ID does not exist" containerID="4e1251835da89da5a9fddeb25225b12d803c4890ffd996fb71a2fa66df963de2" Nov 02 17:50:22.764568 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.764491 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1251835da89da5a9fddeb25225b12d803c4890ffd996fb71a2fa66df963de2"} err="failed to get container status \"4e1251835da89da5a9fddeb25225b12d803c4890ffd996fb71a2fa66df963de2\": rpc error: code = NotFound desc = could not find container \"4e1251835da89da5a9fddeb25225b12d803c4890ffd996fb71a2fa66df963de2\": container with ID starting with 4e1251835da89da5a9fddeb25225b12d803c4890ffd996fb71a2fa66df963de2 not found: ID does not exist" Nov 02 17:50:22.773796 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.773768 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk"] Nov 02 17:50:22.812037 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.800300 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-da5f07e6-659579ddbb-bgpmk"] Nov 02 17:50:22.817977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.817265 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:22.838989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.822890 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/177815d2-33da-445e-9084-4134446547bd-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:22.838989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.822933 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/177815d2-33da-445e-9084-4134446547bd-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:22.838989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.822968 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/177815d2-33da-445e-9084-4134446547bd-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:22.838989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:22.822990 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g789k\" (UniqueName: \"kubernetes.io/projected/177815d2-33da-445e-9084-4134446547bd-kube-api-access-g789k\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:23.701975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:23.700025 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:50:23.704138 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:23.704103 5549 patch_prober.go:29] interesting pod/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:50:23.704399 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:23.704360 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" podUID="36f413ea-e51a-4d63-a07f-0f1a701e6357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 17:50:23.748026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:23.746126 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:24.322381 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:24.322350 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="177815d2-33da-445e-9084-4134446547bd" path="/var/lib/kubelet/pods/177815d2-33da-445e-9084-4134446547bd/volumes" Nov 02 17:50:24.507256 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:24.507217 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff"] Nov 02 17:50:24.617173 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:24.617090 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:24.703986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:24.700382 5549 patch_prober.go:29] interesting pod/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:50:24.703986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:24.700455 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" podUID="36f413ea-e51a-4d63-a07f-0f1a701e6357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 17:50:24.755129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:24.755092 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:24.905034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:24.903565 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:25.126975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:25.115411 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:25.701978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:25.700411 5549 patch_prober.go:29] interesting pod/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:50:25.701978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:25.700474 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" podUID="36f413ea-e51a-4d63-a07f-0f1a701e6357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 17:50:26.145931 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:26.139674 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:26.460145 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:26.460102 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:50:26.604185 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:26.604152 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5" Nov 02 17:50:26.704431 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:26.704393 5549 patch_prober.go:29] interesting pod/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:50:26.704625 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:26.704453 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" podUID="36f413ea-e51a-4d63-a07f-0f1a701e6357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 17:50:27.045979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:27.032444 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:27.704486 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:27.704443 5549 patch_prober.go:29] interesting pod/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:50:27.705129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:27.705089 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" podUID="36f413ea-e51a-4d63-a07f-0f1a701e6357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 17:50:27.927547 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:27.927508 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:28.622425 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:28.622394 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b" Nov 02 17:50:28.739367 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:28.739327 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:28.763534 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:28.761124 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:50:28.775877 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:28.775805 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:50:28.928520 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:28.928433 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:29.247979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:29.246806 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:29.508238 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:29.508116 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff" podUID="8916879a-b154-4fe3-aa47-08d4ce041577" containerName="payload-processing" containerID="cri-o://e7b86702b8e2149857d7819597c826ff84630f7e6dbaf9340da10339b918cb16" gracePeriod=25 Nov 02 17:50:29.578059 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:29.578024 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:29.656238 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:29.656207 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd" Nov 02 17:50:30.034659 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.030599 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:30.071262 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.071008 5549 generic.go:356] "Generic (PLEG): container finished" podID="8916879a-b154-4fe3-aa47-08d4ce041577" containerID="e7b86702b8e2149857d7819597c826ff84630f7e6dbaf9340da10339b918cb16" exitCode=0 Nov 02 17:50:30.071262 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.071181 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff" event={"ID":"8916879a-b154-4fe3-aa47-08d4ce041577","Type":"ContainerDied","Data":"e7b86702b8e2149857d7819597c826ff84630f7e6dbaf9340da10339b918cb16"} Nov 02 17:50:30.218617 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.218553 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff" Nov 02 17:50:30.348249 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.348161 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8916879a-b154-4fe3-aa47-08d4ce041577-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8916879a-b154-4fe3-aa47-08d4ce041577-plugins-config-volume\") pod \"8916879a-b154-4fe3-aa47-08d4ce041577\" (UID: \"8916879a-b154-4fe3-aa47-08d4ce041577\") " Nov 02 17:50:30.348454 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.348436 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8916879a-b154-4fe3-aa47-08d4ce041577-kube-api-access-vppcf\" (UniqueName: \"kubernetes.io/projected/8916879a-b154-4fe3-aa47-08d4ce041577-kube-api-access-vppcf\") pod \"8916879a-b154-4fe3-aa47-08d4ce041577\" (UID: \"8916879a-b154-4fe3-aa47-08d4ce041577\") " Nov 02 17:50:30.348573 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.348542 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8916879a-b154-4fe3-aa47-08d4ce041577-plugins-config-volume" pod "8916879a-b154-4fe3-aa47-08d4ce041577" (UID: "8916879a-b154-4fe3-aa47-08d4ce041577"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:50:30.348974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.348942 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8916879a-b154-4fe3-aa47-08d4ce041577-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:30.350617 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.350592 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8916879a-b154-4fe3-aa47-08d4ce041577-kube-api-access-vppcf" pod "8916879a-b154-4fe3-aa47-08d4ce041577" (UID: "8916879a-b154-4fe3-aa47-08d4ce041577"). InnerVolumeSpecName "kube-api-access-vppcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:30.451279 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.451245 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vppcf\" (UniqueName: \"kubernetes.io/projected/8916879a-b154-4fe3-aa47-08d4ce041577-kube-api-access-vppcf\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:30.461184 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.461153 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:30.780868 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.780804 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq"] Nov 02 17:50:30.782255 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.782226 5549 cpu_manager.go:414] "containerMap: removing container" podUID="37d1d1ae-9504-4f22-9f9d-3cda90298edb" containerName="payload-processing" Nov 02 17:50:30.782255 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.782256 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d1d1ae-9504-4f22-9f9d-3cda90298edb" containerName="payload-processing" Nov 02 17:50:30.782412 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.782273 5549 cpu_manager.go:414] "containerMap: removing container" podUID="0fa6aa51-6ff3-4de7-b322-e3eafe126eb5" containerName="payload-pre-processing" Nov 02 17:50:30.782412 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.782280 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa6aa51-6ff3-4de7-b322-e3eafe126eb5" containerName="payload-pre-processing" Nov 02 17:50:30.782412 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.782308 5549 cpu_manager.go:414] "containerMap: removing container" podUID="177815d2-33da-445e-9084-4134446547bd" containerName="maas-api" Nov 02 17:50:30.782412 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.782315 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="177815d2-33da-445e-9084-4134446547bd" containerName="maas-api" Nov 02 17:50:30.782412 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.782333 5549 cpu_manager.go:414] "containerMap: removing container" podUID="8916879a-b154-4fe3-aa47-08d4ce041577" containerName="payload-processing" Nov 02 17:50:30.782412 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.782341 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8916879a-b154-4fe3-aa47-08d4ce041577" containerName="payload-processing" Nov 02 17:50:30.782593 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.782551 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d1d1ae-9504-4f22-9f9d-3cda90298edb" containerName="payload-processing" Nov 02 17:50:30.782593 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.782564 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa6aa51-6ff3-4de7-b322-e3eafe126eb5" containerName="payload-pre-processing" Nov 02 17:50:30.782593 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.782577 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="177815d2-33da-445e-9084-4134446547bd" containerName="maas-api" Nov 02 17:50:30.782593 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.782587 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8916879a-b154-4fe3-aa47-08d4ce041577" containerName="payload-processing" Nov 02 17:50:30.786009 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.785989 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.791706 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.791677 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cd2738-openshift-default-dockercfg-x45j9\"" Nov 02 17:50:30.802307 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.802280 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq"] Nov 02 17:50:30.858549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.856827 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-workload-certs\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.858549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.857030 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-workload-socket\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.858549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.857117 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-envoy\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.858549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.857213 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-podinfo\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.858549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.857248 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmcwn\" (UniqueName: \"kubernetes.io/projected/b8420a84-6451-4cb6-8c46-82ac1c3562a3-kube-api-access-rmcwn\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.858549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.857343 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-credential-socket\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.858549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.857386 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-token\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.858549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.857423 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istiod-ca-cert\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.858549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.857493 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-data\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.873298 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.873261 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl"] Nov 02 17:50:30.877880 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.877849 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:30.885508 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.885249 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4c9298\"/\"openshift-service-ca.crt\"" Nov 02 17:50:30.890038 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.889985 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:30.901550 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.901498 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4c9298\"/\"kube-root-ca.crt\"" Nov 02 17:50:30.901744 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.901724 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4c9298\"/\"default-dockercfg-68lk8\"" Nov 02 17:50:30.901862 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.901846 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4c9298\"/\"ipp-route-4c9298-kserve-self-signed-certs\"" Nov 02 17:50:30.906189 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.906161 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl"] Nov 02 17:50:30.959327 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.959290 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-envoy\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.959549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.959400 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-model-cache\") pod \"ipp-route-4c9298-kserve-7789dbbfcc-tcltl\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:30.959549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.959431 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-podinfo\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.959549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.959456 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmcwn\" (UniqueName: \"kubernetes.io/projected/b8420a84-6451-4cb6-8c46-82ac1c3562a3-kube-api-access-rmcwn\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.959549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.959504 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-credential-socket\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.959549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.959528 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-token\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.959824 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.959565 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-dshm\") pod \"ipp-route-4c9298-kserve-7789dbbfcc-tcltl\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:30.959824 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.959588 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istiod-ca-cert\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.959824 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.959661 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g5zv\" (UniqueName: \"kubernetes.io/projected/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-kube-api-access-9g5zv\") pod \"ipp-route-4c9298-kserve-7789dbbfcc-tcltl\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:30.959824 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.959690 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-data\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.960068 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.960042 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-data\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.960132 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.960077 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-credential-socket\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.960194 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.960130 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-tmp-dir\") pod \"ipp-route-4c9298-kserve-7789dbbfcc-tcltl\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:30.960248 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.960195 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-workload-certs\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.960300 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.960288 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-tls-certs\") pod \"ipp-route-4c9298-kserve-7789dbbfcc-tcltl\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:30.960355 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.960321 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-workload-socket\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.961715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.960780 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istiod-ca-cert\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.961715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.960797 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-workload-socket\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.961715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.960914 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-home\") pod \"ipp-route-4c9298-kserve-7789dbbfcc-tcltl\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:30.961715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.961090 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-workload-certs\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.962363 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.962338 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-envoy\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.962996 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.962973 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-podinfo\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.974646 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.974619 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-token\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:30.975284 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:30.975262 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmcwn\" (UniqueName: \"kubernetes.io/projected/b8420a84-6451-4cb6-8c46-82ac1c3562a3-kube-api-access-rmcwn\") pod \"e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:31.062452 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.062366 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-tls-certs\") pod \"ipp-route-4c9298-kserve-7789dbbfcc-tcltl\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:31.062452 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.062410 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-home\") pod \"ipp-route-4c9298-kserve-7789dbbfcc-tcltl\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:31.063004 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.062496 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-model-cache\") pod \"ipp-route-4c9298-kserve-7789dbbfcc-tcltl\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:31.063004 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.062555 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-dshm\") pod \"ipp-route-4c9298-kserve-7789dbbfcc-tcltl\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:31.063004 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.062626 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g5zv\" (UniqueName: \"kubernetes.io/projected/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-kube-api-access-9g5zv\") pod \"ipp-route-4c9298-kserve-7789dbbfcc-tcltl\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:31.063004 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.062657 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-tmp-dir\") pod \"ipp-route-4c9298-kserve-7789dbbfcc-tcltl\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:31.063180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.063028 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-model-cache\") pod \"ipp-route-4c9298-kserve-7789dbbfcc-tcltl\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:31.063180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.063054 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-tmp-dir\") pod \"ipp-route-4c9298-kserve-7789dbbfcc-tcltl\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:31.063180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.063140 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-home\") pod \"ipp-route-4c9298-kserve-7789dbbfcc-tcltl\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:31.064767 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.064749 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-dshm\") pod \"ipp-route-4c9298-kserve-7789dbbfcc-tcltl\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:31.065071 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.065051 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-tls-certs\") pod \"ipp-route-4c9298-kserve-7789dbbfcc-tcltl\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:31.076910 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.076879 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g5zv\" (UniqueName: \"kubernetes.io/projected/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-kube-api-access-9g5zv\") pod \"ipp-route-4c9298-kserve-7789dbbfcc-tcltl\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:31.081413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.081387 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff" Nov 02 17:50:31.081636 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.081420 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff" event={"ID":"8916879a-b154-4fe3-aa47-08d4ce041577","Type":"ContainerDied","Data":"7e9428f6712a55a54d7c10ac1f5c51f67d4ea72c0dc792e8977c0a2e765b460b"} Nov 02 17:50:31.081719 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.081677 5549 scope.go:122] "RemoveContainer" containerID="e7b86702b8e2149857d7819597c826ff84630f7e6dbaf9340da10339b918cb16" Nov 02 17:50:31.081839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.081818 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:31.101770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.101738 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:31.188223 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.176942 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff"] Nov 02 17:50:31.218985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.195278 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:31.218985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.195558 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c8832b-68d9664bd7-sw6ff"] Nov 02 17:50:31.275976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.268154 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:31.669479 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.669444 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq"] Nov 02 17:50:31.670702 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:50:31.670670 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8420a84_6451_4cb6_8c46_82ac1c3562a3.slice/crio-782dc658cc4ea32960a76350047d1d3d8c49b836e7753fdd79099b259481bf0b WatchSource:0}: Error finding container 782dc658cc4ea32960a76350047d1d3d8c49b836e7753fdd79099b259481bf0b: Status 404 returned error can't find the container with id 782dc658cc4ea32960a76350047d1d3d8c49b836e7753fdd79099b259481bf0b Nov 02 17:50:31.673557 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.673518 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:50:31.673661 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.673615 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:50:31.673733 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.673665 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:50:31.701680 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.701651 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl"] Nov 02 17:50:31.708313 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:50:31.708280 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5186c8ad_6d18_4e3e_b71c_62ec5af8222a.slice/crio-e5a126c1372d34b00878cc4c1b212c71b7631be5e99bde85beeb6fbbbfca737c WatchSource:0}: Error finding container e5a126c1372d34b00878cc4c1b212c71b7631be5e99bde85beeb6fbbbfca737c: Status 404 returned error can't find the container with id e5a126c1372d34b00878cc4c1b212c71b7631be5e99bde85beeb6fbbbfca737c Nov 02 17:50:31.906974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:31.904898 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:32.097239 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:32.097164 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" event={"ID":"b8420a84-6451-4cb6-8c46-82ac1c3562a3","Type":"ContainerStarted","Data":"f7e4377f8593d2d7269992ca7c390632876106032c0990bfda53896bec8d4f0e"} Nov 02 17:50:32.097769 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:32.097280 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" event={"ID":"b8420a84-6451-4cb6-8c46-82ac1c3562a3","Type":"ContainerStarted","Data":"782dc658cc4ea32960a76350047d1d3d8c49b836e7753fdd79099b259481bf0b"} Nov 02 17:50:32.100004 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:32.099965 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" event={"ID":"5186c8ad-6d18-4e3e-b71c-62ec5af8222a","Type":"ContainerStarted","Data":"846c9430e0432c3d748d1dc27643c8ea00788800204e35f2e90ad94059321e52"} Nov 02 17:50:32.100162 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:32.100014 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" event={"ID":"5186c8ad-6d18-4e3e-b71c-62ec5af8222a","Type":"ContainerStarted","Data":"e5a126c1372d34b00878cc4c1b212c71b7631be5e99bde85beeb6fbbbfca737c"} Nov 02 17:50:32.102211 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:32.102187 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:32.106091 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:32.106060 5549 patch_prober.go:29] interesting pod/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:50:32.106216 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:32.106113 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" podUID="b8420a84-6451-4cb6-8c46-82ac1c3562a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 17:50:32.134355 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:32.134282 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" podStartSLOduration=2.134264169 podStartE2EDuration="2.134264169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:32.132071022 +0000 UTC m=+1003.920576549" watchObservedRunningTime="2026-11-02 17:50:32.134264169 +0000 UTC m=+1003.922769697" Nov 02 17:50:32.163910 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:32.163846 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" podStartSLOduration=2.163831067 podStartE2EDuration="2.163831067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:32.159626964 +0000 UTC m=+1003.948132506" watchObservedRunningTime="2026-11-02 17:50:32.163831067 +0000 UTC m=+1003.952336591" Nov 02 17:50:32.196751 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:32.196722 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:32.336980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:32.336872 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8916879a-b154-4fe3-aa47-08d4ce041577" path="/var/lib/kubelet/pods/8916879a-b154-4fe3-aa47-08d4ce041577/volumes" Nov 02 17:50:32.808988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:32.805977 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:33.107118 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:33.107049 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:33.109541 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:33.109515 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:33.110202 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:33.110183 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:50:33.717760 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:33.716090 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:34.169332 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:34.169250 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:34.237109 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:34.237075 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl"] Nov 02 17:50:34.242244 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:34.242217 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" Nov 02 17:50:34.254115 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:34.248924 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c8832b-dockercfg-vvwxg\"" Nov 02 17:50:34.254115 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:34.249038 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c8832b\"" Nov 02 17:50:34.254115 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:34.252127 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl"] Nov 02 17:50:34.306074 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:34.306037 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:34.340058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:34.340012 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c001de9-be3c-4d28-a13e-bb57b44b6667-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl\" (UID: \"7c001de9-be3c-4d28-a13e-bb57b44b6667\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" Nov 02 17:50:34.341732 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:34.340211 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smq8c\" (UniqueName: \"kubernetes.io/projected/7c001de9-be3c-4d28-a13e-bb57b44b6667-kube-api-access-smq8c\") pod \"payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl\" (UID: \"7c001de9-be3c-4d28-a13e-bb57b44b6667\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" Nov 02 17:50:34.442023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:34.441917 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smq8c\" (UniqueName: \"kubernetes.io/projected/7c001de9-be3c-4d28-a13e-bb57b44b6667-kube-api-access-smq8c\") pod \"payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl\" (UID: \"7c001de9-be3c-4d28-a13e-bb57b44b6667\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" Nov 02 17:50:34.442794 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:34.442767 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c001de9-be3c-4d28-a13e-bb57b44b6667-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl\" (UID: \"7c001de9-be3c-4d28-a13e-bb57b44b6667\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" Nov 02 17:50:34.444004 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:34.443792 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c001de9-be3c-4d28-a13e-bb57b44b6667-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl\" (UID: \"7c001de9-be3c-4d28-a13e-bb57b44b6667\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" Nov 02 17:50:34.505975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:34.493092 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smq8c\" (UniqueName: \"kubernetes.io/projected/7c001de9-be3c-4d28-a13e-bb57b44b6667-kube-api-access-smq8c\") pod \"payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl\" (UID: \"7c001de9-be3c-4d28-a13e-bb57b44b6667\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" Nov 02 17:50:34.524977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:34.510947 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:34.570353 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:34.570323 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" Nov 02 17:50:34.650985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:34.648623 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:34.917148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:34.917118 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl"] Nov 02 17:50:34.918216 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:50:34.918180 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c001de9_be3c_4d28_a13e_bb57b44b6667.slice/crio-8f4237398dfd30ab935c71855c8109aa7eca1382a1b2c4f2177a8db1e48611e7 WatchSource:0}: Error finding container 8f4237398dfd30ab935c71855c8109aa7eca1382a1b2c4f2177a8db1e48611e7: Status 404 returned error can't find the container with id 8f4237398dfd30ab935c71855c8109aa7eca1382a1b2c4f2177a8db1e48611e7 Nov 02 17:50:35.102226 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:35.066294 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:35.167650 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:35.167128 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" event={"ID":"7c001de9-be3c-4d28-a13e-bb57b44b6667","Type":"ContainerStarted","Data":"591f6abdc757c352f2f1c7a94878247cd9141a138fa32ba295e766ac1d44276d"} Nov 02 17:50:35.167650 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:35.167190 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" event={"ID":"7c001de9-be3c-4d28-a13e-bb57b44b6667","Type":"ContainerStarted","Data":"8f4237398dfd30ab935c71855c8109aa7eca1382a1b2c4f2177a8db1e48611e7"} Nov 02 17:50:35.167650 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:35.167240 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" Nov 02 17:50:35.201526 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:35.201475 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" podStartSLOduration=1.201462191 podStartE2EDuration="1.201462191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:35.200663226 +0000 UTC m=+1006.989168754" watchObservedRunningTime="2026-11-02 17:50:35.201462191 +0000 UTC m=+1006.989967756" Nov 02 17:50:35.751256 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:35.751221 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:35.885661 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:35.885626 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:36.042570 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:36.042478 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:37.918992 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:37.915373 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:38.054850 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:38.054812 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:38.805976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:38.803597 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:39.752991 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:39.751833 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:40.623829 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:40.618284 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:40.766457 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:40.766422 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:41.196106 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:41.196068 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:41.208883 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:41.208848 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:41.324587 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:41.324543 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:41.346885 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:41.346850 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:50:41.856705 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:41.856664 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:50:42.213000 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:42.204967 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:43.078301 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:43.075161 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:43.283645 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:43.283606 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:43.753277 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:43.753233 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:44.503210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:44.503163 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:44.653871 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:44.653840 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:44.789227 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:44.789149 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:45.425234 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:45.425195 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:46.223534 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:46.223495 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:46.960424 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:46.960385 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:47.201871 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:47.201841 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" Nov 02 17:50:47.587097 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:50:47.587059 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:50:47.816841 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:47.816803 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:48.750553 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:48.749256 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:49.547917 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:49.546945 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:50.261387 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.261350 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r"] Nov 02 17:50:50.265320 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.265280 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" podUID="e9154805-a9a0-4a01-b44a-1cbb6a3cd88c" containerName="maas-api" containerID="cri-o://90a03347fe25c43a52b3e5029dc06a50798bd4dabac426541f2182a1e31f2561" gracePeriod=30 Nov 02 17:50:50.366698 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.366645 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:50.471908 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.471866 5549 generic.go:356] "Generic (PLEG): container finished" podID="e9154805-a9a0-4a01-b44a-1cbb6a3cd88c" containerID="90a03347fe25c43a52b3e5029dc06a50798bd4dabac426541f2182a1e31f2561" exitCode=0 Nov 02 17:50:50.472135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.472109 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" event={"ID":"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c","Type":"ContainerDied","Data":"90a03347fe25c43a52b3e5029dc06a50798bd4dabac426541f2182a1e31f2561"} Nov 02 17:50:50.547048 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.547001 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:50.668974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.662625 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:50:50.742976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.734393 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-metrics-certs\") pod \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\" (UID: \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\") " Nov 02 17:50:50.742976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.734506 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-maas-api-tls\") pod \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\" (UID: \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\") " Nov 02 17:50:50.742976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.734592 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-kube-api-access-cnhwv\" (UniqueName: \"kubernetes.io/projected/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-kube-api-access-cnhwv\") pod \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\" (UID: \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\") " Nov 02 17:50:50.742976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.734834 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-openshift-service-ca\") pod \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\" (UID: \"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c\") " Nov 02 17:50:50.742976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.741168 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-openshift-service-ca" pod "e9154805-a9a0-4a01-b44a-1cbb6a3cd88c" (UID: "e9154805-a9a0-4a01-b44a-1cbb6a3cd88c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:50:50.755990 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.752633 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:50.759984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.758091 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-metrics-certs" pod "e9154805-a9a0-4a01-b44a-1cbb6a3cd88c" (UID: "e9154805-a9a0-4a01-b44a-1cbb6a3cd88c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:50:50.784258 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.784134 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-maas-api-tls" pod "e9154805-a9a0-4a01-b44a-1cbb6a3cd88c" (UID: "e9154805-a9a0-4a01-b44a-1cbb6a3cd88c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:50:50.784620 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.784582 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-kube-api-access-cnhwv" pod "e9154805-a9a0-4a01-b44a-1cbb6a3cd88c" (UID: "e9154805-a9a0-4a01-b44a-1cbb6a3cd88c"). InnerVolumeSpecName "kube-api-access-cnhwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:50.858168 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.851304 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:50.858168 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.851344 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:50.858168 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.851357 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnhwv\" (UniqueName: \"kubernetes.io/projected/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-kube-api-access-cnhwv\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:50.858168 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:50.851373 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:51.480780 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:51.480747 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" Nov 02 17:50:51.480780 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:51.480763 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r" event={"ID":"e9154805-a9a0-4a01-b44a-1cbb6a3cd88c","Type":"ContainerDied","Data":"52ba38c1766e7b948ba128dd2cdaf15dfd7d87a3215516aee8bc05685652d580"} Nov 02 17:50:51.481393 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:51.480845 5549 scope.go:122] "RemoveContainer" containerID="90a03347fe25c43a52b3e5029dc06a50798bd4dabac426541f2182a1e31f2561" Nov 02 17:50:51.514759 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:51.514727 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r"] Nov 02 17:50:51.517763 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:51.517739 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1fc77-65f4d596d4-mgv5r"] Nov 02 17:50:51.645974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:51.643740 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:52.329973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:52.327634 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9154805-a9a0-4a01-b44a-1cbb6a3cd88c" path="/var/lib/kubelet/pods/e9154805-a9a0-4a01-b44a-1cbb6a3cd88c/volumes" Nov 02 17:50:52.342976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:52.333375 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:52.491981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:52.491921 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:53.046979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:53.044192 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:53.834125 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:53.830359 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:54.629980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:54.629782 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:55.481586 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:55.477837 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:56.423521 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:56.423483 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:57.212505 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:57.212449 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:58.085370 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:58.085333 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:58.281721 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:58.281688 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:58.804986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:58.804938 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94"] Nov 02 17:50:58.805244 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:58.805220 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" podUID="a8ad8c31-06b1-4604-a74c-92f6b0483fa7" containerName="maas-api" containerID="cri-o://3d0530ad80beedac5ad7d6261736b1dd9408a0707eb1591764ecb6b4d22019d4" gracePeriod=30 Nov 02 17:50:58.882344 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:58.882310 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf"] Nov 02 17:50:58.882553 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:58.882527 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" podUID="89796f12-ff97-4206-af09-e8cf8e06c681" containerName="payload-processing" containerID="cri-o://d4fecb565292e842da6e3becfd748f26a9a83bf3ea97544fe945f9a46f850f03" gracePeriod=30 Nov 02 17:50:58.909729 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:58.909694 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:59.066827 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.065686 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:59.323884 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.323706 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:59.483984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.477403 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" Nov 02 17:50:59.488987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.488927 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:59.548544 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.548508 5549 generic.go:356] "Generic (PLEG): container finished" podID="89796f12-ff97-4206-af09-e8cf8e06c681" containerID="d4fecb565292e842da6e3becfd748f26a9a83bf3ea97544fe945f9a46f850f03" exitCode=0 Nov 02 17:50:59.548737 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.548705 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" event={"ID":"89796f12-ff97-4206-af09-e8cf8e06c681","Type":"ContainerDied","Data":"d4fecb565292e842da6e3becfd748f26a9a83bf3ea97544fe945f9a46f850f03"} Nov 02 17:50:59.550195 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.550171 5549 generic.go:356] "Generic (PLEG): container finished" podID="a8ad8c31-06b1-4604-a74c-92f6b0483fa7" containerID="3d0530ad80beedac5ad7d6261736b1dd9408a0707eb1591764ecb6b4d22019d4" exitCode=0 Nov 02 17:50:59.550310 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.550199 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" event={"ID":"a8ad8c31-06b1-4604-a74c-92f6b0483fa7","Type":"ContainerDied","Data":"3d0530ad80beedac5ad7d6261736b1dd9408a0707eb1591764ecb6b4d22019d4"} Nov 02 17:50:59.550310 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.550223 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" Nov 02 17:50:59.550310 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.550257 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94" event={"ID":"a8ad8c31-06b1-4604-a74c-92f6b0483fa7","Type":"ContainerDied","Data":"9bba22506f602b87cbc572fa29e4a04cd24633c04a47f8f2b50a061f3a1b638c"} Nov 02 17:50:59.550310 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.550288 5549 scope.go:122] "RemoveContainer" containerID="3d0530ad80beedac5ad7d6261736b1dd9408a0707eb1591764ecb6b4d22019d4" Nov 02 17:50:59.559233 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.559211 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" Nov 02 17:50:59.560979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.560941 5549 scope.go:122] "RemoveContainer" containerID="3d0530ad80beedac5ad7d6261736b1dd9408a0707eb1591764ecb6b4d22019d4" Nov 02 17:50:59.561247 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:50:59.561228 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0530ad80beedac5ad7d6261736b1dd9408a0707eb1591764ecb6b4d22019d4\": container with ID starting with 3d0530ad80beedac5ad7d6261736b1dd9408a0707eb1591764ecb6b4d22019d4 not found: ID does not exist" containerID="3d0530ad80beedac5ad7d6261736b1dd9408a0707eb1591764ecb6b4d22019d4" Nov 02 17:50:59.561324 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.561259 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0530ad80beedac5ad7d6261736b1dd9408a0707eb1591764ecb6b4d22019d4"} err="failed to get container status \"3d0530ad80beedac5ad7d6261736b1dd9408a0707eb1591764ecb6b4d22019d4\": rpc error: code = NotFound desc = could not find container \"3d0530ad80beedac5ad7d6261736b1dd9408a0707eb1591764ecb6b4d22019d4\": container with ID starting with 3d0530ad80beedac5ad7d6261736b1dd9408a0707eb1591764ecb6b4d22019d4 not found: ID does not exist" Nov 02 17:50:59.638023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.621721 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-openshift-service-ca\") pod \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\" (UID: \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\") " Nov 02 17:50:59.638023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.621781 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-kube-api-access-gpvn9\" (UniqueName: \"kubernetes.io/projected/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-kube-api-access-gpvn9\") pod \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\" (UID: \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\") " Nov 02 17:50:59.638023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.621831 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89796f12-ff97-4206-af09-e8cf8e06c681-kube-api-access-4w4dz\" (UniqueName: \"kubernetes.io/projected/89796f12-ff97-4206-af09-e8cf8e06c681-kube-api-access-4w4dz\") pod \"89796f12-ff97-4206-af09-e8cf8e06c681\" (UID: \"89796f12-ff97-4206-af09-e8cf8e06c681\") " Nov 02 17:50:59.638023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.622054 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-metrics-certs\") pod \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\" (UID: \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\") " Nov 02 17:50:59.638023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.622248 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-maas-api-tls\") pod \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\" (UID: \"a8ad8c31-06b1-4604-a74c-92f6b0483fa7\") " Nov 02 17:50:59.638023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.622325 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89796f12-ff97-4206-af09-e8cf8e06c681-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89796f12-ff97-4206-af09-e8cf8e06c681-plugins-config-volume\") pod \"89796f12-ff97-4206-af09-e8cf8e06c681\" (UID: \"89796f12-ff97-4206-af09-e8cf8e06c681\") " Nov 02 17:50:59.638023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.623666 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89796f12-ff97-4206-af09-e8cf8e06c681-plugins-config-volume" pod "89796f12-ff97-4206-af09-e8cf8e06c681" (UID: "89796f12-ff97-4206-af09-e8cf8e06c681"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:50:59.638023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.624266 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-openshift-service-ca" pod "a8ad8c31-06b1-4604-a74c-92f6b0483fa7" (UID: "a8ad8c31-06b1-4604-a74c-92f6b0483fa7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:50:59.640387 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.640311 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-metrics-certs" pod "a8ad8c31-06b1-4604-a74c-92f6b0483fa7" (UID: "a8ad8c31-06b1-4604-a74c-92f6b0483fa7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:50:59.640538 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.640379 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-kube-api-access-gpvn9" pod "a8ad8c31-06b1-4604-a74c-92f6b0483fa7" (UID: "a8ad8c31-06b1-4604-a74c-92f6b0483fa7"). InnerVolumeSpecName "kube-api-access-gpvn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:59.640796 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.640768 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89796f12-ff97-4206-af09-e8cf8e06c681-kube-api-access-4w4dz" pod "89796f12-ff97-4206-af09-e8cf8e06c681" (UID: "89796f12-ff97-4206-af09-e8cf8e06c681"). InnerVolumeSpecName "kube-api-access-4w4dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:59.641616 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.641587 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-maas-api-tls" pod "a8ad8c31-06b1-4604-a74c-92f6b0483fa7" (UID: "a8ad8c31-06b1-4604-a74c-92f6b0483fa7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:50:59.724511 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.724481 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:59.724511 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.724507 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:59.724511 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.724518 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89796f12-ff97-4206-af09-e8cf8e06c681-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:59.724759 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.724530 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:59.724759 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.724539 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpvn9\" (UniqueName: \"kubernetes.io/projected/a8ad8c31-06b1-4604-a74c-92f6b0483fa7-kube-api-access-gpvn9\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:59.724759 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.724548 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w4dz\" (UniqueName: \"kubernetes.io/projected/89796f12-ff97-4206-af09-e8cf8e06c681-kube-api-access-4w4dz\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:59.787528 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.787490 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:59.906040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.900988 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94"] Nov 02 17:50:59.911584 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.908465 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-82b305-5cdbf68d7b-8nb94"] Nov 02 17:50:59.939383 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:50:59.937112 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:00.305047 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:00.305008 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:00.386612 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:00.383762 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ad8c31-06b1-4604-a74c-92f6b0483fa7" path="/var/lib/kubelet/pods/a8ad8c31-06b1-4604-a74c-92f6b0483fa7/volumes" Nov 02 17:51:00.455427 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:00.455062 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" podUID="89796f12-ff97-4206-af09-e8cf8e06c681" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.118:9004: i/o timeout" Nov 02 17:51:00.485523 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:00.485494 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:00.559681 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:00.559594 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" Nov 02 17:51:00.559681 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:00.559605 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf" event={"ID":"89796f12-ff97-4206-af09-e8cf8e06c681","Type":"ContainerDied","Data":"7bf52d9e8024a7c68b9bb2ab833eb2d12fa6b811f45be1ba42d6bb91664c955d"} Nov 02 17:51:00.559681 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:00.559659 5549 scope.go:122] "RemoveContainer" containerID="d4fecb565292e842da6e3becfd748f26a9a83bf3ea97544fe945f9a46f850f03" Nov 02 17:51:00.585768 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:00.585739 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf"] Nov 02 17:51:00.591088 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:00.591059 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-82b305-6664b9c9c8-5tzqf"] Nov 02 17:51:00.697141 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:00.697108 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx"] Nov 02 17:51:00.697363 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:00.697335 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" podUID="57b8639c-8c58-4d9a-81c9-f5c262a11884" containerName="istio-proxy" containerID="cri-o://01c0f7c604eb311a0a9a65be09805f73baf58f862bda570b43cb812149d2d95e" gracePeriod=30 Nov 02 17:51:00.779110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:00.779072 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:00.977995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:00.969153 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:01.438015 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:01.437926 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:01.600335 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:01.600294 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:01.763137 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:01.763107 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:02.221649 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:02.221610 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:02.320397 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:02.320363 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89796f12-ff97-4206-af09-e8cf8e06c681" path="/var/lib/kubelet/pods/89796f12-ff97-4206-af09-e8cf8e06c681/volumes" Nov 02 17:51:02.915723 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:02.911921 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:03.703996 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:03.703860 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:04.762975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:04.757477 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:04.965697 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:04.965664 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:05.152783 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:05.152529 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:06.145212 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.121712 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:06.160237 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.160208 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:51:06.255830 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.255791 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-workload-certs\") pod \"57b8639c-8c58-4d9a-81c9-f5c262a11884\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " Nov 02 17:51:06.256001 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.255980 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-podinfo\") pod \"57b8639c-8c58-4d9a-81c9-f5c262a11884\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " Nov 02 17:51:06.256088 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.256072 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-credential-socket\") pod \"57b8639c-8c58-4d9a-81c9-f5c262a11884\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " Nov 02 17:51:06.256125 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.256116 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-envoy\") pod \"57b8639c-8c58-4d9a-81c9-f5c262a11884\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " Nov 02 17:51:06.256165 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.256142 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-workload-socket\") pod \"57b8639c-8c58-4d9a-81c9-f5c262a11884\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " Nov 02 17:51:06.256213 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.256188 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57b8639c-8c58-4d9a-81c9-f5c262a11884-kube-api-access-2qcnc\" (UniqueName: \"kubernetes.io/projected/57b8639c-8c58-4d9a-81c9-f5c262a11884-kube-api-access-2qcnc\") pod \"57b8639c-8c58-4d9a-81c9-f5c262a11884\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " Nov 02 17:51:06.256300 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.256283 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-data\") pod \"57b8639c-8c58-4d9a-81c9-f5c262a11884\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " Nov 02 17:51:06.256361 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.256351 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57b8639c-8c58-4d9a-81c9-f5c262a11884-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b8639c-8c58-4d9a-81c9-f5c262a11884-istiod-ca-cert\") pod \"57b8639c-8c58-4d9a-81c9-f5c262a11884\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " Nov 02 17:51:06.256432 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.256418 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-token\" (UniqueName: \"kubernetes.io/projected/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-token\") pod \"57b8639c-8c58-4d9a-81c9-f5c262a11884\" (UID: \"57b8639c-8c58-4d9a-81c9-f5c262a11884\") " Nov 02 17:51:06.256493 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.256448 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-workload-certs" pod "57b8639c-8c58-4d9a-81c9-f5c262a11884" (UID: "57b8639c-8c58-4d9a-81c9-f5c262a11884"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:06.256655 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.256636 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-workload-socket" pod "57b8639c-8c58-4d9a-81c9-f5c262a11884" (UID: "57b8639c-8c58-4d9a-81c9-f5c262a11884"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:06.257063 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.257041 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:06.257163 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.257067 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:06.257286 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.257268 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-credential-socket" pod "57b8639c-8c58-4d9a-81c9-f5c262a11884" (UID: "57b8639c-8c58-4d9a-81c9-f5c262a11884"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:06.257825 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.257789 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-data" pod "57b8639c-8c58-4d9a-81c9-f5c262a11884" (UID: "57b8639c-8c58-4d9a-81c9-f5c262a11884"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:06.258190 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.258157 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b8639c-8c58-4d9a-81c9-f5c262a11884-istiod-ca-cert" pod "57b8639c-8c58-4d9a-81c9-f5c262a11884" (UID: "57b8639c-8c58-4d9a-81c9-f5c262a11884"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:06.258988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.258934 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-podinfo" pod "57b8639c-8c58-4d9a-81c9-f5c262a11884" (UID: "57b8639c-8c58-4d9a-81c9-f5c262a11884"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:51:06.259531 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.259507 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-envoy" pod "57b8639c-8c58-4d9a-81c9-f5c262a11884" (UID: "57b8639c-8c58-4d9a-81c9-f5c262a11884"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:06.259948 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.259924 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-token" pod "57b8639c-8c58-4d9a-81c9-f5c262a11884" (UID: "57b8639c-8c58-4d9a-81c9-f5c262a11884"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:06.261894 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.261872 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b8639c-8c58-4d9a-81c9-f5c262a11884-kube-api-access-2qcnc" pod "57b8639c-8c58-4d9a-81c9-f5c262a11884" (UID: "57b8639c-8c58-4d9a-81c9-f5c262a11884"). InnerVolumeSpecName "kube-api-access-2qcnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:06.343199 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.343123 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:06.358700 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.358665 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:06.358700 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.358698 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:06.358866 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.358715 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qcnc\" (UniqueName: \"kubernetes.io/projected/57b8639c-8c58-4d9a-81c9-f5c262a11884-kube-api-access-2qcnc\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:06.358866 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.358729 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:06.358866 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.358739 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b8639c-8c58-4d9a-81c9-f5c262a11884-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:06.358866 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.358747 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:06.358866 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.358755 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b8639c-8c58-4d9a-81c9-f5c262a11884-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:06.366095 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.366060 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q"] Nov 02 17:51:06.366766 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.366747 5549 cpu_manager.go:414] "containerMap: removing container" podUID="57b8639c-8c58-4d9a-81c9-f5c262a11884" containerName="istio-proxy" Nov 02 17:51:06.366860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.366766 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b8639c-8c58-4d9a-81c9-f5c262a11884" containerName="istio-proxy" Nov 02 17:51:06.366860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.366805 5549 cpu_manager.go:414] "containerMap: removing container" podUID="a8ad8c31-06b1-4604-a74c-92f6b0483fa7" containerName="maas-api" Nov 02 17:51:06.366860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.366813 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ad8c31-06b1-4604-a74c-92f6b0483fa7" containerName="maas-api" Nov 02 17:51:06.366860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.366833 5549 cpu_manager.go:414] "containerMap: removing container" podUID="89796f12-ff97-4206-af09-e8cf8e06c681" containerName="payload-processing" Nov 02 17:51:06.366860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.366841 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89796f12-ff97-4206-af09-e8cf8e06c681" containerName="payload-processing" Nov 02 17:51:06.367141 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.366892 5549 cpu_manager.go:414] "containerMap: removing container" podUID="e9154805-a9a0-4a01-b44a-1cbb6a3cd88c" containerName="maas-api" Nov 02 17:51:06.367141 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.366901 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9154805-a9a0-4a01-b44a-1cbb6a3cd88c" containerName="maas-api" Nov 02 17:51:06.367141 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.367097 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="57b8639c-8c58-4d9a-81c9-f5c262a11884" containerName="istio-proxy" Nov 02 17:51:06.367141 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.367112 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="57b8639c-8c58-4d9a-81c9-f5c262a11884" containerName="istio-proxy" Nov 02 17:51:06.367141 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.367131 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ad8c31-06b1-4604-a74c-92f6b0483fa7" containerName="maas-api" Nov 02 17:51:06.367141 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.367142 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="89796f12-ff97-4206-af09-e8cf8e06c681" containerName="payload-processing" Nov 02 17:51:06.367422 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.367158 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9154805-a9a0-4a01-b44a-1cbb6a3cd88c" containerName="maas-api" Nov 02 17:51:06.371111 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.371091 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q" Nov 02 17:51:06.404978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.397323 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3568c5-dockercfg-dxz77\"" Nov 02 17:51:06.404978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.398309 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3568c5\"" Nov 02 17:51:06.410913 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.410827 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q"] Nov 02 17:51:06.422048 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.422015 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf"] Nov 02 17:51:06.423377 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.423349 5549 cpu_manager.go:414] "containerMap: removing container" podUID="57b8639c-8c58-4d9a-81c9-f5c262a11884" containerName="istio-proxy" Nov 02 17:51:06.423548 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.423532 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b8639c-8c58-4d9a-81c9-f5c262a11884" containerName="istio-proxy" Nov 02 17:51:06.428125 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.428102 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf" Nov 02 17:51:06.432273 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.432242 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf"] Nov 02 17:51:06.530469 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.530432 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:06.577604 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.577556 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df101814-f22d-4abd-888c-b3288a7707a1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3568c5-5468488957-njxbf\" (UID: \"df101814-f22d-4abd-888c-b3288a7707a1\") " pod="openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf" Nov 02 17:51:06.599060 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.577890 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpll9\" (UniqueName: \"kubernetes.io/projected/536ab9a4-1941-4ca5-8a50-691e53ee1148-kube-api-access-fpll9\") pod \"payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q\" (UID: \"536ab9a4-1941-4ca5-8a50-691e53ee1148\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q" Nov 02 17:51:06.601089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.601056 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536ab9a4-1941-4ca5-8a50-691e53ee1148-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q\" (UID: \"536ab9a4-1941-4ca5-8a50-691e53ee1148\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q" Nov 02 17:51:06.601291 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.601268 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2sqc\" (UniqueName: \"kubernetes.io/projected/df101814-f22d-4abd-888c-b3288a7707a1-kube-api-access-w2sqc\") pod \"payload-processing-e2e-api-b-3568c5-5468488957-njxbf\" (UID: \"df101814-f22d-4abd-888c-b3288a7707a1\") " pod="openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf" Nov 02 17:51:06.714977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.709066 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df101814-f22d-4abd-888c-b3288a7707a1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3568c5-5468488957-njxbf\" (UID: \"df101814-f22d-4abd-888c-b3288a7707a1\") " pod="openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf" Nov 02 17:51:06.714977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.709165 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpll9\" (UniqueName: \"kubernetes.io/projected/536ab9a4-1941-4ca5-8a50-691e53ee1148-kube-api-access-fpll9\") pod \"payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q\" (UID: \"536ab9a4-1941-4ca5-8a50-691e53ee1148\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q" Nov 02 17:51:06.714977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.709249 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536ab9a4-1941-4ca5-8a50-691e53ee1148-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q\" (UID: \"536ab9a4-1941-4ca5-8a50-691e53ee1148\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q" Nov 02 17:51:06.714977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.709278 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2sqc\" (UniqueName: \"kubernetes.io/projected/df101814-f22d-4abd-888c-b3288a7707a1-kube-api-access-w2sqc\") pod \"payload-processing-e2e-api-b-3568c5-5468488957-njxbf\" (UID: \"df101814-f22d-4abd-888c-b3288a7707a1\") " pod="openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf" Nov 02 17:51:06.730454 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.730415 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df101814-f22d-4abd-888c-b3288a7707a1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3568c5-5468488957-njxbf\" (UID: \"df101814-f22d-4abd-888c-b3288a7707a1\") " pod="openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf" Nov 02 17:51:06.740655 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.740593 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2sqc\" (UniqueName: \"kubernetes.io/projected/df101814-f22d-4abd-888c-b3288a7707a1-kube-api-access-w2sqc\") pod \"payload-processing-e2e-api-b-3568c5-5468488957-njxbf\" (UID: \"df101814-f22d-4abd-888c-b3288a7707a1\") " pod="openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf" Nov 02 17:51:06.746082 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.742662 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536ab9a4-1941-4ca5-8a50-691e53ee1148-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q\" (UID: \"536ab9a4-1941-4ca5-8a50-691e53ee1148\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q" Nov 02 17:51:06.771326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.754248 5549 generic.go:356] "Generic (PLEG): container finished" podID="57b8639c-8c58-4d9a-81c9-f5c262a11884" containerID="01c0f7c604eb311a0a9a65be09805f73baf58f862bda570b43cb812149d2d95e" exitCode=0 Nov 02 17:51:06.771326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.754525 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" event={"ID":"57b8639c-8c58-4d9a-81c9-f5c262a11884","Type":"ContainerDied","Data":"01c0f7c604eb311a0a9a65be09805f73baf58f862bda570b43cb812149d2d95e"} Nov 02 17:51:06.771326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.754578 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" event={"ID":"57b8639c-8c58-4d9a-81c9-f5c262a11884","Type":"ContainerDied","Data":"53e8008bdf29215124704071efc34afce5c2fb9666035b870b2fc4b1f319d5af"} Nov 02 17:51:06.771326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.754605 5549 scope.go:122] "RemoveContainer" containerID="01c0f7c604eb311a0a9a65be09805f73baf58f862bda570b43cb812149d2d95e" Nov 02 17:51:06.771326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.754784 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx" Nov 02 17:51:06.783353 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.772764 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf" Nov 02 17:51:06.829802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.829755 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpll9\" (UniqueName: \"kubernetes.io/projected/536ab9a4-1941-4ca5-8a50-691e53ee1148-kube-api-access-fpll9\") pod \"payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q\" (UID: \"536ab9a4-1941-4ca5-8a50-691e53ee1148\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q" Nov 02 17:51:06.851251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.851223 5549 scope.go:122] "RemoveContainer" containerID="08be164b6776ced2d5d1e0c324929647ec7b401de830adb35ab57c5396edff71" Nov 02 17:51:06.852943 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.852921 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx"] Nov 02 17:51:06.860671 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.860646 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d1fc77-openshift-default-5d7d996b8c-rqwtx"] Nov 02 17:51:06.874527 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.874505 5549 scope.go:122] "RemoveContainer" containerID="01c0f7c604eb311a0a9a65be09805f73baf58f862bda570b43cb812149d2d95e" Nov 02 17:51:06.874847 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:51:06.874822 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c0f7c604eb311a0a9a65be09805f73baf58f862bda570b43cb812149d2d95e\": container with ID starting with 01c0f7c604eb311a0a9a65be09805f73baf58f862bda570b43cb812149d2d95e not found: ID does not exist" containerID="01c0f7c604eb311a0a9a65be09805f73baf58f862bda570b43cb812149d2d95e" Nov 02 17:51:06.874986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.874852 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c0f7c604eb311a0a9a65be09805f73baf58f862bda570b43cb812149d2d95e"} err="failed to get container status \"01c0f7c604eb311a0a9a65be09805f73baf58f862bda570b43cb812149d2d95e\": rpc error: code = NotFound desc = could not find container \"01c0f7c604eb311a0a9a65be09805f73baf58f862bda570b43cb812149d2d95e\": container with ID starting with 01c0f7c604eb311a0a9a65be09805f73baf58f862bda570b43cb812149d2d95e not found: ID does not exist" Nov 02 17:51:06.874986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.874871 5549 scope.go:122] "RemoveContainer" containerID="08be164b6776ced2d5d1e0c324929647ec7b401de830adb35ab57c5396edff71" Nov 02 17:51:06.875220 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:51:06.875194 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08be164b6776ced2d5d1e0c324929647ec7b401de830adb35ab57c5396edff71\": container with ID starting with 08be164b6776ced2d5d1e0c324929647ec7b401de830adb35ab57c5396edff71 not found: ID does not exist" containerID="08be164b6776ced2d5d1e0c324929647ec7b401de830adb35ab57c5396edff71" Nov 02 17:51:06.875283 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:06.875227 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08be164b6776ced2d5d1e0c324929647ec7b401de830adb35ab57c5396edff71"} err="failed to get container status \"08be164b6776ced2d5d1e0c324929647ec7b401de830adb35ab57c5396edff71\": rpc error: code = NotFound desc = could not find container \"08be164b6776ced2d5d1e0c324929647ec7b401de830adb35ab57c5396edff71\": container with ID starting with 08be164b6776ced2d5d1e0c324929647ec7b401de830adb35ab57c5396edff71 not found: ID does not exist" Nov 02 17:51:07.024402 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:07.024370 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q" Nov 02 17:51:07.140088 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:07.140055 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf"] Nov 02 17:51:07.143349 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:51:07.143321 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf101814_f22d_4abd_888c_b3288a7707a1.slice/crio-4363bbd7b88e8a3ade46d82de1a379d84c8f4f3a52638c24c6e6fc976ad6617b WatchSource:0}: Error finding container 4363bbd7b88e8a3ade46d82de1a379d84c8f4f3a52638c24c6e6fc976ad6617b: Status 404 returned error can't find the container with id 4363bbd7b88e8a3ade46d82de1a379d84c8f4f3a52638c24c6e6fc976ad6617b Nov 02 17:51:07.338978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:07.334473 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:07.462175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:07.461923 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q"] Nov 02 17:51:07.567275 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:07.567239 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:07.763605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:07.763562 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf" event={"ID":"df101814-f22d-4abd-888c-b3288a7707a1","Type":"ContainerStarted","Data":"b1463edba5fd24ab8f020abfc1740f153cdb2d828386a54f12a3a303842c9b59"} Nov 02 17:51:07.763803 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:07.763635 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf" Nov 02 17:51:07.763803 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:07.763660 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf" event={"ID":"df101814-f22d-4abd-888c-b3288a7707a1","Type":"ContainerStarted","Data":"4363bbd7b88e8a3ade46d82de1a379d84c8f4f3a52638c24c6e6fc976ad6617b"} Nov 02 17:51:07.765392 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:07.765346 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q" event={"ID":"536ab9a4-1941-4ca5-8a50-691e53ee1148","Type":"ContainerStarted","Data":"fe690e17705ad7ec3676c817ec37cc3dfe5869d81cea238c9f754dcdb324a822"} Nov 02 17:51:07.765392 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:07.765393 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q" event={"ID":"536ab9a4-1941-4ca5-8a50-691e53ee1148","Type":"ContainerStarted","Data":"1dc538fa23fc1271cfada922555ffcaaaf63c52214cc57a314d241885821279d"} Nov 02 17:51:07.765570 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:07.765418 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q" Nov 02 17:51:07.768264 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:07.768238 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:07.826739 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:07.826649 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q" podStartSLOduration=1.826631677 podStartE2EDuration="1.826631677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:07.825563054 +0000 UTC m=+1039.614068581" watchObservedRunningTime="2026-11-02 17:51:07.826631677 +0000 UTC m=+1039.615137205" Nov 02 17:51:07.829942 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:07.829898 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf" podStartSLOduration=1.8298858949999999 podStartE2EDuration="1.829885895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:07.803770614 +0000 UTC m=+1039.592276144" watchObservedRunningTime="2026-11-02 17:51:07.829885895 +0000 UTC m=+1039.618391422" Nov 02 17:51:08.133360 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:08.133234 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:08.321214 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:08.321183 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57b8639c-8c58-4d9a-81c9-f5c262a11884" path="/var/lib/kubelet/pods/57b8639c-8c58-4d9a-81c9-f5c262a11884/volumes" Nov 02 17:51:08.517038 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:08.514054 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:08.689634 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:08.689482 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:09.345274 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:09.345240 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:09.635026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:09.632399 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:10.257066 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:10.257021 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:10.412422 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:10.412337 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd"] Nov 02 17:51:10.412646 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:10.412587 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd" podUID="6a551174-8eda-4593-9c92-8e3a143c9a5b" containerName="payload-processing" containerID="cri-o://a0f3d562ba9cd670af528209d19e06103c4ad4c61e4fcc5c9de9eba2544e7bcf" gracePeriod=30 Nov 02 17:51:10.444822 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:10.444788 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b"] Nov 02 17:51:10.445223 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:10.445193 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b" podUID="fdca28da-5265-43b5-93b6-71d3d4239a6d" containerName="payload-pre-processing" containerID="cri-o://93ceae6108e77527d4607e0eab6584faaba5b44612cd688d01bcf39bd213a87e" gracePeriod=30 Nov 02 17:51:10.450419 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:10.450354 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:10.614308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:10.614246 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:11.042325 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.037357 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:11.053918 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.053871 5549 generic.go:356] "Generic (PLEG): container finished" podID="fdca28da-5265-43b5-93b6-71d3d4239a6d" containerID="93ceae6108e77527d4607e0eab6584faaba5b44612cd688d01bcf39bd213a87e" exitCode=0 Nov 02 17:51:11.054169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.054139 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b" event={"ID":"fdca28da-5265-43b5-93b6-71d3d4239a6d","Type":"ContainerDied","Data":"93ceae6108e77527d4607e0eab6584faaba5b44612cd688d01bcf39bd213a87e"} Nov 02 17:51:11.075170 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.066610 5549 generic.go:356] "Generic (PLEG): container finished" podID="6a551174-8eda-4593-9c92-8e3a143c9a5b" containerID="a0f3d562ba9cd670af528209d19e06103c4ad4c61e4fcc5c9de9eba2544e7bcf" exitCode=0 Nov 02 17:51:11.075170 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.066667 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd" event={"ID":"6a551174-8eda-4593-9c92-8e3a143c9a5b","Type":"ContainerDied","Data":"a0f3d562ba9cd670af528209d19e06103c4ad4c61e4fcc5c9de9eba2544e7bcf"} Nov 02 17:51:11.160022 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.159718 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd" Nov 02 17:51:11.261869 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.261842 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b" Nov 02 17:51:11.351101 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.351022 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a551174-8eda-4593-9c92-8e3a143c9a5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a551174-8eda-4593-9c92-8e3a143c9a5b-plugins-config-volume\") pod \"6a551174-8eda-4593-9c92-8e3a143c9a5b\" (UID: \"6a551174-8eda-4593-9c92-8e3a143c9a5b\") " Nov 02 17:51:11.351248 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.351158 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a551174-8eda-4593-9c92-8e3a143c9a5b-kube-api-access-4n25c\" (UniqueName: \"kubernetes.io/projected/6a551174-8eda-4593-9c92-8e3a143c9a5b-kube-api-access-4n25c\") pod \"6a551174-8eda-4593-9c92-8e3a143c9a5b\" (UID: \"6a551174-8eda-4593-9c92-8e3a143c9a5b\") " Nov 02 17:51:11.351471 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.351448 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a551174-8eda-4593-9c92-8e3a143c9a5b-plugins-config-volume" pod "6a551174-8eda-4593-9c92-8e3a143c9a5b" (UID: "6a551174-8eda-4593-9c92-8e3a143c9a5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:11.351880 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.351862 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a551174-8eda-4593-9c92-8e3a143c9a5b-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:11.353205 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.353183 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a551174-8eda-4593-9c92-8e3a143c9a5b-kube-api-access-4n25c" pod "6a551174-8eda-4593-9c92-8e3a143c9a5b" (UID: "6a551174-8eda-4593-9c92-8e3a143c9a5b"). InnerVolumeSpecName "kube-api-access-4n25c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:11.452830 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.452802 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdca28da-5265-43b5-93b6-71d3d4239a6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdca28da-5265-43b5-93b6-71d3d4239a6d-plugins-config-volume\") pod \"fdca28da-5265-43b5-93b6-71d3d4239a6d\" (UID: \"fdca28da-5265-43b5-93b6-71d3d4239a6d\") " Nov 02 17:51:11.453008 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.452939 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdca28da-5265-43b5-93b6-71d3d4239a6d-kube-api-access-dgxrl\" (UniqueName: \"kubernetes.io/projected/fdca28da-5265-43b5-93b6-71d3d4239a6d-kube-api-access-dgxrl\") pod \"fdca28da-5265-43b5-93b6-71d3d4239a6d\" (UID: \"fdca28da-5265-43b5-93b6-71d3d4239a6d\") " Nov 02 17:51:11.453558 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.453243 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdca28da-5265-43b5-93b6-71d3d4239a6d-plugins-config-volume" pod "fdca28da-5265-43b5-93b6-71d3d4239a6d" (UID: "fdca28da-5265-43b5-93b6-71d3d4239a6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:11.453731 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.453709 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdca28da-5265-43b5-93b6-71d3d4239a6d-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:11.453813 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.453738 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n25c\" (UniqueName: \"kubernetes.io/projected/6a551174-8eda-4593-9c92-8e3a143c9a5b-kube-api-access-4n25c\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:11.455020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.454998 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdca28da-5265-43b5-93b6-71d3d4239a6d-kube-api-access-dgxrl" pod "fdca28da-5265-43b5-93b6-71d3d4239a6d" (UID: "fdca28da-5265-43b5-93b6-71d3d4239a6d"). InnerVolumeSpecName "kube-api-access-dgxrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:11.555429 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.555395 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgxrl\" (UniqueName: \"kubernetes.io/projected/fdca28da-5265-43b5-93b6-71d3d4239a6d-kube-api-access-dgxrl\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:11.811997 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:11.807471 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:12.077342 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:12.077252 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b" Nov 02 17:51:12.077342 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:12.077275 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b" event={"ID":"fdca28da-5265-43b5-93b6-71d3d4239a6d","Type":"ContainerDied","Data":"659e3277aea0fd2874046600452f312cc22031631d4297192d079c5afc0d975e"} Nov 02 17:51:12.077342 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:12.077337 5549 scope.go:122] "RemoveContainer" containerID="93ceae6108e77527d4607e0eab6584faaba5b44612cd688d01bcf39bd213a87e" Nov 02 17:51:12.082110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:12.082087 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd" Nov 02 17:51:12.082226 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:12.082129 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd" event={"ID":"6a551174-8eda-4593-9c92-8e3a143c9a5b","Type":"ContainerDied","Data":"75f7f11ac1e9fd0c525cb6d6f09067b186ac6a5b2e599489498ab3cf9e310622"} Nov 02 17:51:12.092388 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:12.092371 5549 scope.go:122] "RemoveContainer" containerID="a0f3d562ba9cd670af528209d19e06103c4ad4c61e4fcc5c9de9eba2544e7bcf" Nov 02 17:51:12.110897 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:12.110863 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd"] Nov 02 17:51:12.115021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:12.114992 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-d6755c6c4-7rbsd"] Nov 02 17:51:12.128186 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:12.128161 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b"] Nov 02 17:51:12.131906 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:12.131883 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-766db84587-xk66b"] Nov 02 17:51:12.321001 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:12.320964 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a551174-8eda-4593-9c92-8e3a143c9a5b" path="/var/lib/kubelet/pods/6a551174-8eda-4593-9c92-8e3a143c9a5b/volumes" Nov 02 17:51:12.321477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:12.321461 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdca28da-5265-43b5-93b6-71d3d4239a6d" path="/var/lib/kubelet/pods/fdca28da-5265-43b5-93b6-71d3d4239a6d/volumes" Nov 02 17:51:12.644650 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:12.644616 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:13.546056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:13.542886 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:14.366010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:14.360646 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:14.928922 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:14.928069 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz"] Nov 02 17:51:14.929312 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:14.929062 5549 cpu_manager.go:414] "containerMap: removing container" podUID="fdca28da-5265-43b5-93b6-71d3d4239a6d" containerName="payload-pre-processing" Nov 02 17:51:14.929312 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:14.929078 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdca28da-5265-43b5-93b6-71d3d4239a6d" containerName="payload-pre-processing" Nov 02 17:51:14.929312 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:14.929130 5549 cpu_manager.go:414] "containerMap: removing container" podUID="6a551174-8eda-4593-9c92-8e3a143c9a5b" containerName="payload-processing" Nov 02 17:51:14.929312 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:14.929138 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a551174-8eda-4593-9c92-8e3a143c9a5b" containerName="payload-processing" Nov 02 17:51:14.929312 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:14.929279 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a551174-8eda-4593-9c92-8e3a143c9a5b" containerName="payload-processing" Nov 02 17:51:14.929312 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:14.929299 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdca28da-5265-43b5-93b6-71d3d4239a6d" containerName="payload-pre-processing" Nov 02 17:51:14.933611 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:14.933588 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" Nov 02 17:51:14.946109 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:14.946067 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-3568c5-serving-cert\"" Nov 02 17:51:14.948294 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:14.947532 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz"] Nov 02 17:51:15.041256 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.041224 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:15.042828 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.042800 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lt5r\" (UniqueName: \"kubernetes.io/projected/c916c795-ef55-4664-91ff-392939fa292f-kube-api-access-7lt5r\") pod \"maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz\" (UID: \"c916c795-ef55-4664-91ff-392939fa292f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" Nov 02 17:51:15.042945 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.042845 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c916c795-ef55-4664-91ff-392939fa292f-metrics-certs\") pod \"maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz\" (UID: \"c916c795-ef55-4664-91ff-392939fa292f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" Nov 02 17:51:15.043119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.043011 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c916c795-ef55-4664-91ff-392939fa292f-maas-api-tls\") pod \"maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz\" (UID: \"c916c795-ef55-4664-91ff-392939fa292f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" Nov 02 17:51:15.043119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.043054 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c916c795-ef55-4664-91ff-392939fa292f-openshift-service-ca\") pod \"maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz\" (UID: \"c916c795-ef55-4664-91ff-392939fa292f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" Nov 02 17:51:15.151986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.150498 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lt5r\" (UniqueName: \"kubernetes.io/projected/c916c795-ef55-4664-91ff-392939fa292f-kube-api-access-7lt5r\") pod \"maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz\" (UID: \"c916c795-ef55-4664-91ff-392939fa292f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" Nov 02 17:51:15.151986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.150568 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c916c795-ef55-4664-91ff-392939fa292f-metrics-certs\") pod \"maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz\" (UID: \"c916c795-ef55-4664-91ff-392939fa292f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" Nov 02 17:51:15.151986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.150653 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c916c795-ef55-4664-91ff-392939fa292f-maas-api-tls\") pod \"maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz\" (UID: \"c916c795-ef55-4664-91ff-392939fa292f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" Nov 02 17:51:15.151986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.150686 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c916c795-ef55-4664-91ff-392939fa292f-openshift-service-ca\") pod \"maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz\" (UID: \"c916c795-ef55-4664-91ff-392939fa292f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" Nov 02 17:51:15.151986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.151685 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c916c795-ef55-4664-91ff-392939fa292f-openshift-service-ca\") pod \"maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz\" (UID: \"c916c795-ef55-4664-91ff-392939fa292f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" Nov 02 17:51:15.168424 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.168165 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c916c795-ef55-4664-91ff-392939fa292f-maas-api-tls\") pod \"maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz\" (UID: \"c916c795-ef55-4664-91ff-392939fa292f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" Nov 02 17:51:15.183057 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.170512 5549 generic.go:356] "Generic (PLEG): container finished" podID="b8420a84-6451-4cb6-8c46-82ac1c3562a3" containerID="f7e4377f8593d2d7269992ca7c390632876106032c0990bfda53896bec8d4f0e" exitCode=0 Nov 02 17:51:15.183057 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.170799 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" event={"ID":"b8420a84-6451-4cb6-8c46-82ac1c3562a3","Type":"ContainerDied","Data":"f7e4377f8593d2d7269992ca7c390632876106032c0990bfda53896bec8d4f0e"} Nov 02 17:51:15.187931 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.183936 5549 scope.go:122] "RemoveContainer" containerID="f7e4377f8593d2d7269992ca7c390632876106032c0990bfda53896bec8d4f0e" Nov 02 17:51:15.190839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.190733 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:51:15.191147 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.191117 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:51:15.191327 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.191299 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:51:15.203727 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.203650 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lt5r\" (UniqueName: \"kubernetes.io/projected/c916c795-ef55-4664-91ff-392939fa292f-kube-api-access-7lt5r\") pod \"maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz\" (UID: \"c916c795-ef55-4664-91ff-392939fa292f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" Nov 02 17:51:15.210376 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.210330 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c916c795-ef55-4664-91ff-392939fa292f-metrics-certs\") pod \"maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz\" (UID: \"c916c795-ef55-4664-91ff-392939fa292f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" Nov 02 17:51:15.238924 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.238857 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:15.255465 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.255438 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" Nov 02 17:51:15.644136 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:15.635467 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz"] Nov 02 17:51:15.644136 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:51:15.636416 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc916c795_ef55_4664_91ff_392939fa292f.slice/crio-047f930899e72b4cee3c6b0ff3722dc4ea6982ff8ce85cfcc7fd37519f0cf855 WatchSource:0}: Error finding container 047f930899e72b4cee3c6b0ff3722dc4ea6982ff8ce85cfcc7fd37519f0cf855: Status 404 returned error can't find the container with id 047f930899e72b4cee3c6b0ff3722dc4ea6982ff8ce85cfcc7fd37519f0cf855 Nov 02 17:51:16.015344 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:16.015312 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:16.214974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:16.213918 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" event={"ID":"c916c795-ef55-4664-91ff-392939fa292f","Type":"ContainerStarted","Data":"047f930899e72b4cee3c6b0ff3722dc4ea6982ff8ce85cfcc7fd37519f0cf855"} Nov 02 17:51:16.235006 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:16.221930 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" event={"ID":"b8420a84-6451-4cb6-8c46-82ac1c3562a3","Type":"ContainerStarted","Data":"42e00003b323c85b42b2554c2b83e44fe2d9a1599ed709e4da39cdc9b65d3099"} Nov 02 17:51:16.630215 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:16.630132 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl"] Nov 02 17:51:16.630412 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:16.630383 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" podUID="7c001de9-be3c-4d28-a13e-bb57b44b6667" containerName="payload-processing" containerID="cri-o://591f6abdc757c352f2f1c7a94878247cd9141a138fa32ba295e766ac1d44276d" gracePeriod=30 Nov 02 17:51:16.664425 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:16.664391 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:16.897153 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:16.884059 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:17.134972 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:17.128365 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:51:17.135492 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:17.135079 5549 patch_prober.go:29] interesting pod/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:51:17.135492 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:17.135141 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" podUID="b8420a84-6451-4cb6-8c46-82ac1c3562a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 17:51:17.336039 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:17.334392 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:17.358471 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:17.358429 5549 generic.go:356] "Generic (PLEG): container finished" podID="7c001de9-be3c-4d28-a13e-bb57b44b6667" containerID="591f6abdc757c352f2f1c7a94878247cd9141a138fa32ba295e766ac1d44276d" exitCode=0 Nov 02 17:51:17.358675 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:17.358524 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" event={"ID":"7c001de9-be3c-4d28-a13e-bb57b44b6667","Type":"ContainerDied","Data":"591f6abdc757c352f2f1c7a94878247cd9141a138fa32ba295e766ac1d44276d"} Nov 02 17:51:17.359518 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:17.359485 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" event={"ID":"7c001de9-be3c-4d28-a13e-bb57b44b6667","Type":"ContainerDied","Data":"8f4237398dfd30ab935c71855c8109aa7eca1382a1b2c4f2177a8db1e48611e7"} Nov 02 17:51:17.359654 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:17.359532 5549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f4237398dfd30ab935c71855c8109aa7eca1382a1b2c4f2177a8db1e48611e7" Nov 02 17:51:17.362712 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:17.362682 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" event={"ID":"c916c795-ef55-4664-91ff-392939fa292f","Type":"ContainerStarted","Data":"79a4ff2a10177148a34129e0bee32aeaebcd1e778a077978aaaa680e875d2f55"} Nov 02 17:51:17.363435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:17.363414 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" Nov 02 17:51:17.425698 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:17.425658 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c001de9-be3c-4d28-a13e-bb57b44b6667-kube-api-access-smq8c\" (UniqueName: \"kubernetes.io/projected/7c001de9-be3c-4d28-a13e-bb57b44b6667-kube-api-access-smq8c\") pod \"7c001de9-be3c-4d28-a13e-bb57b44b6667\" (UID: \"7c001de9-be3c-4d28-a13e-bb57b44b6667\") " Nov 02 17:51:17.425875 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:17.425811 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c001de9-be3c-4d28-a13e-bb57b44b6667-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c001de9-be3c-4d28-a13e-bb57b44b6667-plugins-config-volume\") pod \"7c001de9-be3c-4d28-a13e-bb57b44b6667\" (UID: \"7c001de9-be3c-4d28-a13e-bb57b44b6667\") " Nov 02 17:51:17.426376 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:17.426322 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c001de9-be3c-4d28-a13e-bb57b44b6667-plugins-config-volume" pod "7c001de9-be3c-4d28-a13e-bb57b44b6667" (UID: "7c001de9-be3c-4d28-a13e-bb57b44b6667"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:17.427507 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:17.427481 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c001de9-be3c-4d28-a13e-bb57b44b6667-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:17.429810 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:17.429749 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" podStartSLOduration=2.80183891 podStartE2EDuration="3.42973041s" totalImagesPullingTime="627.8915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:17.395895643 +0000 UTC m=+1049.184401172" watchObservedRunningTime="2026-11-02 17:51:17.42973041 +0000 UTC m=+1049.218235941" Nov 02 17:51:17.430418 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:17.430258 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c001de9-be3c-4d28-a13e-bb57b44b6667-kube-api-access-smq8c" pod "7c001de9-be3c-4d28-a13e-bb57b44b6667" (UID: "7c001de9-be3c-4d28-a13e-bb57b44b6667"). InnerVolumeSpecName "kube-api-access-smq8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:17.505781 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:17.505670 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:17.535727 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:17.532884 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smq8c\" (UniqueName: \"kubernetes.io/projected/7c001de9-be3c-4d28-a13e-bb57b44b6667-kube-api-access-smq8c\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:17.662590 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:51:17.662553 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:51:18.079311 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:18.079272 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:18.103002 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:18.102965 5549 patch_prober.go:29] interesting pod/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:51:18.103264 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:18.103238 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" podUID="b8420a84-6451-4cb6-8c46-82ac1c3562a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 17:51:18.123975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:18.116605 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:51:18.207698 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:18.206282 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" podUID="7c001de9-be3c-4d28-a13e-bb57b44b6667" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.128:9004: i/o timeout" Nov 02 17:51:18.265644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:18.265610 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:18.369850 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:18.369776 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl" Nov 02 17:51:18.370098 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:18.370085 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" Nov 02 17:51:18.410020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:18.409930 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl"] Nov 02 17:51:18.413512 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:18.413477 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c8832b-64f867f664-zmlgl"] Nov 02 17:51:18.530795 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:18.530637 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:18.813463 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:18.813430 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q" Nov 02 17:51:18.814064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:18.814039 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf" Nov 02 17:51:18.903981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:18.902489 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:19.091886 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:19.091798 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:19.117977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:19.105737 5549 patch_prober.go:29] interesting pod/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:51:19.117977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:19.105798 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" podUID="b8420a84-6451-4cb6-8c46-82ac1c3562a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 17:51:19.974208 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:19.974172 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:51:20.058978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:20.058566 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:20.121978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:20.113736 5549 patch_prober.go:29] interesting pod/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:51:20.121978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:20.113800 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" podUID="b8420a84-6451-4cb6-8c46-82ac1c3562a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 17:51:20.322700 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:20.322622 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c001de9-be3c-4d28-a13e-bb57b44b6667" path="/var/lib/kubelet/pods/7c001de9-be3c-4d28-a13e-bb57b44b6667/volumes" Nov 02 17:51:20.500327 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:20.500290 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:20.849456 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:20.849417 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:21.015733 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:21.015696 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:21.110428 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:21.110337 5549 patch_prober.go:29] interesting pod/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:51:21.110690 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:21.110658 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" podUID="b8420a84-6451-4cb6-8c46-82ac1c3562a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 17:51:21.668992 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:21.668894 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:22.105418 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:22.105376 5549 patch_prober.go:29] interesting pod/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:51:22.105837 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:22.105447 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" podUID="b8420a84-6451-4cb6-8c46-82ac1c3562a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 17:51:22.413032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:22.412493 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:23.107672 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:23.107639 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:51:23.108713 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:23.108686 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:51:23.135645 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:23.135612 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:24.184979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:24.178790 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:24.344125 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:24.344082 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:24.411475 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:24.411446 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" Nov 02 17:51:24.591582 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:24.591552 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:24.772403 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:24.772369 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:24.942641 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:24.942553 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:25.154427 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:25.154394 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:25.335906 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:25.335865 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:25.713666 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:25.713625 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:26.226930 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.226861 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:26.242782 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.242755 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6"] Nov 02 17:51:26.243476 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.243462 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7c001de9-be3c-4d28-a13e-bb57b44b6667" containerName="payload-processing" Nov 02 17:51:26.243524 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.243478 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c001de9-be3c-4d28-a13e-bb57b44b6667" containerName="payload-processing" Nov 02 17:51:26.243611 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.243602 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c001de9-be3c-4d28-a13e-bb57b44b6667" containerName="payload-processing" Nov 02 17:51:26.248449 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.248428 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6" Nov 02 17:51:26.261940 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.261919 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-e4fdc1\"" Nov 02 17:51:26.267933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.267910 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h"] Nov 02 17:51:26.268239 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.268220 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-e4fdc1-dockercfg-78s8w\"" Nov 02 17:51:26.272718 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.272699 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6"] Nov 02 17:51:26.272837 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.272790 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h" Nov 02 17:51:26.297430 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.297400 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h"] Nov 02 17:51:26.330979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.328598 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6hxb\" (UniqueName: \"kubernetes.io/projected/6c6c7ac6-b524-465f-8195-447bbc3e5695-kube-api-access-n6hxb\") pod \"payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h\" (UID: \"6c6c7ac6-b524-465f-8195-447bbc3e5695\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h" Nov 02 17:51:26.330979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.328742 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm4nw\" (UniqueName: \"kubernetes.io/projected/655f5af4-c96f-4657-a926-3b8ef89a002b-kube-api-access-zm4nw\") pod \"payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6\" (UID: \"655f5af4-c96f-4657-a926-3b8ef89a002b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6" Nov 02 17:51:26.330979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.328839 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c6c7ac6-b524-465f-8195-447bbc3e5695-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h\" (UID: \"6c6c7ac6-b524-465f-8195-447bbc3e5695\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h" Nov 02 17:51:26.330979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.329023 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655f5af4-c96f-4657-a926-3b8ef89a002b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6\" (UID: \"655f5af4-c96f-4657-a926-3b8ef89a002b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6" Nov 02 17:51:26.455976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.455332 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655f5af4-c96f-4657-a926-3b8ef89a002b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6\" (UID: \"655f5af4-c96f-4657-a926-3b8ef89a002b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6" Nov 02 17:51:26.455976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.455550 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6hxb\" (UniqueName: \"kubernetes.io/projected/6c6c7ac6-b524-465f-8195-447bbc3e5695-kube-api-access-n6hxb\") pod \"payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h\" (UID: \"6c6c7ac6-b524-465f-8195-447bbc3e5695\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h" Nov 02 17:51:26.455976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.455697 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm4nw\" (UniqueName: \"kubernetes.io/projected/655f5af4-c96f-4657-a926-3b8ef89a002b-kube-api-access-zm4nw\") pod \"payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6\" (UID: \"655f5af4-c96f-4657-a926-3b8ef89a002b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6" Nov 02 17:51:26.455976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.455787 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c6c7ac6-b524-465f-8195-447bbc3e5695-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h\" (UID: \"6c6c7ac6-b524-465f-8195-447bbc3e5695\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h" Nov 02 17:51:26.482029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.456741 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c6c7ac6-b524-465f-8195-447bbc3e5695-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h\" (UID: \"6c6c7ac6-b524-465f-8195-447bbc3e5695\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h" Nov 02 17:51:26.482029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.464721 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655f5af4-c96f-4657-a926-3b8ef89a002b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6\" (UID: \"655f5af4-c96f-4657-a926-3b8ef89a002b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6" Nov 02 17:51:26.508943 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.508383 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm4nw\" (UniqueName: \"kubernetes.io/projected/655f5af4-c96f-4657-a926-3b8ef89a002b-kube-api-access-zm4nw\") pod \"payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6\" (UID: \"655f5af4-c96f-4657-a926-3b8ef89a002b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6" Nov 02 17:51:26.508943 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.508874 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:26.560316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.560245 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6hxb\" (UniqueName: \"kubernetes.io/projected/6c6c7ac6-b524-465f-8195-447bbc3e5695-kube-api-access-n6hxb\") pod \"payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h\" (UID: \"6c6c7ac6-b524-465f-8195-447bbc3e5695\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h" Nov 02 17:51:26.562513 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.562270 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6" Nov 02 17:51:26.634019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.632889 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h" Nov 02 17:51:26.787386 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:26.785286 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:27.035546 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.035509 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:27.141544 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.141456 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8"] Nov 02 17:51:27.147450 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.147425 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.157896 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.157874 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-7dbdb8-openshift-default-dockercfg-9hnrh\"" Nov 02 17:51:27.168873 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.168848 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8"] Nov 02 17:51:27.173452 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.173431 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h"] Nov 02 17:51:27.174084 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:51:27.174056 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6c7ac6_b524_465f_8195_447bbc3e5695.slice/crio-a462e4ccfd5ebf835912e64dfd090140e5fad11e6259c7fb013e9117b2accf58 WatchSource:0}: Error finding container a462e4ccfd5ebf835912e64dfd090140e5fad11e6259c7fb013e9117b2accf58: Status 404 returned error can't find the container with id a462e4ccfd5ebf835912e64dfd090140e5fad11e6259c7fb013e9117b2accf58 Nov 02 17:51:27.193503 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.193470 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6"] Nov 02 17:51:27.194559 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.194531 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-workload-socket\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.194657 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.194646 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-credential-socket\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.194747 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.194733 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98455203-d4a6-4bc7-af34-b901edb31b83-istio-podinfo\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.194822 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.194760 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-workload-certs\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.194822 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.194776 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98455203-d4a6-4bc7-af34-b901edb31b83-istiod-ca-cert\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.194822 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.194794 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9plxr\" (UniqueName: \"kubernetes.io/projected/98455203-d4a6-4bc7-af34-b901edb31b83-kube-api-access-9plxr\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.194822 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.194819 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-istio-data\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.195066 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.194968 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98455203-d4a6-4bc7-af34-b901edb31b83-istio-token\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.197600 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.196331 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-istio-envoy\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.198693 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:51:27.198302 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655f5af4_c96f_4657_a926_3b8ef89a002b.slice/crio-35d44cd123807dbe1c61e344f6f49709a319a033c2d89e47f48c4b94ec83d66a WatchSource:0}: Error finding container 35d44cd123807dbe1c61e344f6f49709a319a033c2d89e47f48c4b94ec83d66a: Status 404 returned error can't find the container with id 35d44cd123807dbe1c61e344f6f49709a319a033c2d89e47f48c4b94ec83d66a Nov 02 17:51:27.367977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.319741 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98455203-d4a6-4bc7-af34-b901edb31b83-istio-podinfo\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.367977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.319800 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-workload-certs\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.367977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.319831 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98455203-d4a6-4bc7-af34-b901edb31b83-istiod-ca-cert\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.367977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.319856 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9plxr\" (UniqueName: \"kubernetes.io/projected/98455203-d4a6-4bc7-af34-b901edb31b83-kube-api-access-9plxr\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.367977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.319900 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-istio-data\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.367977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.320001 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98455203-d4a6-4bc7-af34-b901edb31b83-istio-token\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.367977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.320056 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-istio-envoy\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.367977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.320196 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-workload-socket\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.367977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.320284 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-credential-socket\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.367977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.320795 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-credential-socket\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.371211 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.369683 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-istio-data\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.371211 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.369945 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-workload-certs\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.371211 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.370580 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98455203-d4a6-4bc7-af34-b901edb31b83-istiod-ca-cert\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.372239 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.372213 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-workload-socket\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.421027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.391813 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-istio-envoy\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.447871 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.447786 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9plxr\" (UniqueName: \"kubernetes.io/projected/98455203-d4a6-4bc7-af34-b901edb31b83-kube-api-access-9plxr\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.448810 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.448737 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:27.492980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.472492 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98455203-d4a6-4bc7-af34-b901edb31b83-istio-token\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.492980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.487632 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98455203-d4a6-4bc7-af34-b901edb31b83-istio-podinfo\") pod \"e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.649982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.649934 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:27.658254 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.658218 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6" event={"ID":"655f5af4-c96f-4657-a926-3b8ef89a002b","Type":"ContainerStarted","Data":"4aa1de65b8320b444c02e24a2f9b61a1ba9865cf104b02393ffde8fcd196c109"} Nov 02 17:51:27.658390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.658281 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6" event={"ID":"655f5af4-c96f-4657-a926-3b8ef89a002b","Type":"ContainerStarted","Data":"35d44cd123807dbe1c61e344f6f49709a319a033c2d89e47f48c4b94ec83d66a"} Nov 02 17:51:27.658390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.658304 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6" Nov 02 17:51:27.660037 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.660011 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h" event={"ID":"6c6c7ac6-b524-465f-8195-447bbc3e5695","Type":"ContainerStarted","Data":"9149e24ffaab27a1b2e6fa45258db33dd07ade28cf5aac949fe82027bd95aa59"} Nov 02 17:51:27.660168 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.660059 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h" event={"ID":"6c6c7ac6-b524-465f-8195-447bbc3e5695","Type":"ContainerStarted","Data":"a462e4ccfd5ebf835912e64dfd090140e5fad11e6259c7fb013e9117b2accf58"} Nov 02 17:51:27.660168 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.660108 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h" Nov 02 17:51:27.684736 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.684637 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6" podStartSLOduration=1.68462287 podStartE2EDuration="1.68462287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:27.681993894 +0000 UTC m=+1059.470499419" watchObservedRunningTime="2026-11-02 17:51:27.68462287 +0000 UTC m=+1059.473128398" Nov 02 17:51:27.709848 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.709800 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h" podStartSLOduration=1.709787229 podStartE2EDuration="1.709787229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:27.707882489 +0000 UTC m=+1059.496388016" watchObservedRunningTime="2026-11-02 17:51:27.709787229 +0000 UTC m=+1059.498292891" Nov 02 17:51:27.763575 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.763543 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:27.850986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:27.850506 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:28.052198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:28.052168 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8"] Nov 02 17:51:28.053434 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:51:28.053400 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98455203_d4a6_4bc7_af34_b901edb31b83.slice/crio-71fbb3a16e5e239205ce290fa4fd4544f0ae54a4bb704b64e115b0a90dc3d410 WatchSource:0}: Error finding container 71fbb3a16e5e239205ce290fa4fd4544f0ae54a4bb704b64e115b0a90dc3d410: Status 404 returned error can't find the container with id 71fbb3a16e5e239205ce290fa4fd4544f0ae54a4bb704b64e115b0a90dc3d410 Nov 02 17:51:28.056278 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:28.056242 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:51:28.056347 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:28.056310 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:51:28.056347 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:28.056342 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:51:28.669795 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:28.669754 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" event={"ID":"98455203-d4a6-4bc7-af34-b901edb31b83","Type":"ContainerStarted","Data":"1ca8cb082abbaf44cedadfaad543f43d18a780ac912e6b2612370f2bf3dfd38a"} Nov 02 17:51:28.670248 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:28.669830 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" event={"ID":"98455203-d4a6-4bc7-af34-b901edb31b83","Type":"ContainerStarted","Data":"71fbb3a16e5e239205ce290fa4fd4544f0ae54a4bb704b64e115b0a90dc3d410"} Nov 02 17:51:28.701965 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:28.701077 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" podStartSLOduration=1.701057016 podStartE2EDuration="1.701057016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:28.695630142 +0000 UTC m=+1060.484135675" watchObservedRunningTime="2026-11-02 17:51:28.701057016 +0000 UTC m=+1060.489562545" Nov 02 17:51:28.764301 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:28.764268 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:29.144732 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:29.144681 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:29.364978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:29.360544 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:29.770630 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:29.770591 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:30.234988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:30.234867 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:30.457595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:30.457559 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:30.706975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:30.705819 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:30.707190 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:30.707154 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:30.714974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:30.708273 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:51:31.120731 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:31.120638 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:31.335030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:31.334981 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:31.688091 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:31.688056 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:31.909891 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:31.909847 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:32.456557 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:32.456519 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:32.695975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:32.690602 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:33.071910 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:33.068316 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:33.978606 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:33.978568 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:34.739678 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:34.739631 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:35.554363 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:35.554331 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:35.769259 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:35.769227 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:36.356301 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:36.356268 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5"] Nov 02 17:51:36.356495 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:36.356473 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" podUID="1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" containerName="istio-proxy" containerID="cri-o://33a69841875be6fe17055d08dfc74eb8eaa1f1bcb40cefb8cca7b7e4b51cb221" gracePeriod=30 Nov 02 17:51:36.359618 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:36.359595 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8"] Nov 02 17:51:36.359781 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:36.359761 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" podUID="34d4c449-0024-4b94-91c0-41b4cb484bab" containerName="istio-proxy" containerID="cri-o://296a9df9680c75e30a22ab2e477775be89e3596d995e40a01858e4ceb031246c" gracePeriod=30 Nov 02 17:51:36.502975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:36.502919 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:37.382155 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:37.382115 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:38.160808 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:38.160750 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:38.498356 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:38.498322 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:38.722975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:38.715451 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h" Nov 02 17:51:38.726974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:38.724142 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6" Nov 02 17:51:38.726974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:38.726444 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:39.269739 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:39.269653 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:39.512143 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:39.512110 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:40.080997 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:40.078686 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:40.268418 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:40.268379 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:41.251976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:41.250070 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:41.891316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:41.891256 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:51:41.936904 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:41.936879 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:51:42.009680 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.009643 5549 generic.go:356] "Generic (PLEG): container finished" podID="1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" containerID="33a69841875be6fe17055d08dfc74eb8eaa1f1bcb40cefb8cca7b7e4b51cb221" exitCode=0 Nov 02 17:51:42.009991 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.009697 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" Nov 02 17:51:42.010174 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.009693 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" event={"ID":"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3","Type":"ContainerDied","Data":"33a69841875be6fe17055d08dfc74eb8eaa1f1bcb40cefb8cca7b7e4b51cb221"} Nov 02 17:51:42.010270 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.010194 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5" event={"ID":"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3","Type":"ContainerDied","Data":"d811ed00ed5dc4447e508c7a57adaf7cd088f0520391c4d5841df36e019a5a03"} Nov 02 17:51:42.010270 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.010260 5549 scope.go:122] "RemoveContainer" containerID="33a69841875be6fe17055d08dfc74eb8eaa1f1bcb40cefb8cca7b7e4b51cb221" Nov 02 17:51:42.012189 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.012166 5549 generic.go:356] "Generic (PLEG): container finished" podID="34d4c449-0024-4b94-91c0-41b4cb484bab" containerID="296a9df9680c75e30a22ab2e477775be89e3596d995e40a01858e4ceb031246c" exitCode=0 Nov 02 17:51:42.012433 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.012253 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" event={"ID":"34d4c449-0024-4b94-91c0-41b4cb484bab","Type":"ContainerDied","Data":"296a9df9680c75e30a22ab2e477775be89e3596d995e40a01858e4ceb031246c"} Nov 02 17:51:42.012433 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.012293 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" event={"ID":"34d4c449-0024-4b94-91c0-41b4cb484bab","Type":"ContainerDied","Data":"20de5eb9733bab19a5e92c6fb3878551cc9c2e46a046eb0acfd7eb33bc135591"} Nov 02 17:51:42.012433 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.012359 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8" Nov 02 17:51:42.021851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.021830 5549 scope.go:122] "RemoveContainer" containerID="33a69841875be6fe17055d08dfc74eb8eaa1f1bcb40cefb8cca7b7e4b51cb221" Nov 02 17:51:42.022196 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:51:42.022176 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a69841875be6fe17055d08dfc74eb8eaa1f1bcb40cefb8cca7b7e4b51cb221\": container with ID starting with 33a69841875be6fe17055d08dfc74eb8eaa1f1bcb40cefb8cca7b7e4b51cb221 not found: ID does not exist" containerID="33a69841875be6fe17055d08dfc74eb8eaa1f1bcb40cefb8cca7b7e4b51cb221" Nov 02 17:51:42.022242 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.022208 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a69841875be6fe17055d08dfc74eb8eaa1f1bcb40cefb8cca7b7e4b51cb221"} err="failed to get container status \"33a69841875be6fe17055d08dfc74eb8eaa1f1bcb40cefb8cca7b7e4b51cb221\": rpc error: code = NotFound desc = could not find container \"33a69841875be6fe17055d08dfc74eb8eaa1f1bcb40cefb8cca7b7e4b51cb221\": container with ID starting with 33a69841875be6fe17055d08dfc74eb8eaa1f1bcb40cefb8cca7b7e4b51cb221 not found: ID does not exist" Nov 02 17:51:42.022242 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.022225 5549 scope.go:122] "RemoveContainer" containerID="296a9df9680c75e30a22ab2e477775be89e3596d995e40a01858e4ceb031246c" Nov 02 17:51:42.032091 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.032074 5549 scope.go:122] "RemoveContainer" containerID="296a9df9680c75e30a22ab2e477775be89e3596d995e40a01858e4ceb031246c" Nov 02 17:51:42.032370 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:51:42.032352 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296a9df9680c75e30a22ab2e477775be89e3596d995e40a01858e4ceb031246c\": container with ID starting with 296a9df9680c75e30a22ab2e477775be89e3596d995e40a01858e4ceb031246c not found: ID does not exist" containerID="296a9df9680c75e30a22ab2e477775be89e3596d995e40a01858e4ceb031246c" Nov 02 17:51:42.032426 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.032376 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296a9df9680c75e30a22ab2e477775be89e3596d995e40a01858e4ceb031246c"} err="failed to get container status \"296a9df9680c75e30a22ab2e477775be89e3596d995e40a01858e4ceb031246c\": rpc error: code = NotFound desc = could not find container \"296a9df9680c75e30a22ab2e477775be89e3596d995e40a01858e4ceb031246c\": container with ID starting with 296a9df9680c75e30a22ab2e477775be89e3596d995e40a01858e4ceb031246c not found: ID does not exist" Nov 02 17:51:42.052859 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.052838 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d4c449-0024-4b94-91c0-41b4cb484bab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d4c449-0024-4b94-91c0-41b4cb484bab-istiod-ca-cert\") pod \"34d4c449-0024-4b94-91c0-41b4cb484bab\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " Nov 02 17:51:42.052977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.052876 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-credential-socket\") pod \"34d4c449-0024-4b94-91c0-41b4cb484bab\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " Nov 02 17:51:42.052977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.052908 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-workload-certs\") pod \"34d4c449-0024-4b94-91c0-41b4cb484bab\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " Nov 02 17:51:42.052977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.052943 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-envoy\") pod \"34d4c449-0024-4b94-91c0-41b4cb484bab\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " Nov 02 17:51:42.053147 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053014 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-token\" (UniqueName: \"kubernetes.io/projected/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-token\") pod \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " Nov 02 17:51:42.053147 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053056 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-workload-certs\") pod \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " Nov 02 17:51:42.053147 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053092 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-kube-api-access-8xxtr\" (UniqueName: \"kubernetes.io/projected/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-kube-api-access-8xxtr\") pod \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " Nov 02 17:51:42.053147 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053136 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-envoy\") pod \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " Nov 02 17:51:42.053336 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053159 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-credential-socket" pod "34d4c449-0024-4b94-91c0-41b4cb484bab" (UID: "34d4c449-0024-4b94-91c0-41b4cb484bab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:42.053336 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053197 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-workload-socket\") pod \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " Nov 02 17:51:42.053336 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053228 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-workload-certs" pod "34d4c449-0024-4b94-91c0-41b4cb484bab" (UID: "34d4c449-0024-4b94-91c0-41b4cb484bab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:42.053336 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053250 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-podinfo\") pod \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " Nov 02 17:51:42.053336 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053294 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-data\") pod \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " Nov 02 17:51:42.053607 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053345 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d4c449-0024-4b94-91c0-41b4cb484bab-kube-api-access-jwhm7\" (UniqueName: \"kubernetes.io/projected/34d4c449-0024-4b94-91c0-41b4cb484bab-kube-api-access-jwhm7\") pod \"34d4c449-0024-4b94-91c0-41b4cb484bab\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " Nov 02 17:51:42.053607 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053381 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-token\" (UniqueName: \"kubernetes.io/projected/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-token\") pod \"34d4c449-0024-4b94-91c0-41b4cb484bab\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " Nov 02 17:51:42.053607 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053384 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d4c449-0024-4b94-91c0-41b4cb484bab-istiod-ca-cert" pod "34d4c449-0024-4b94-91c0-41b4cb484bab" (UID: "34d4c449-0024-4b94-91c0-41b4cb484bab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:42.053607 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053425 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-workload-socket\") pod \"34d4c449-0024-4b94-91c0-41b4cb484bab\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " Nov 02 17:51:42.053607 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053536 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-workload-socket" pod "1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" (UID: "1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:42.053607 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053564 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istiod-ca-cert\") pod \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " Nov 02 17:51:42.053607 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053586 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-data" pod "1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" (UID: "1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:42.053607 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053607 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-podinfo\") pod \"34d4c449-0024-4b94-91c0-41b4cb484bab\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " Nov 02 17:51:42.054107 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053669 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-credential-socket\") pod \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\" (UID: \"1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3\") " Nov 02 17:51:42.054107 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053702 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-data\") pod \"34d4c449-0024-4b94-91c0-41b4cb484bab\" (UID: \"34d4c449-0024-4b94-91c0-41b4cb484bab\") " Nov 02 17:51:42.054107 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.053890 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-workload-certs" pod "1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" (UID: "1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:42.054524 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.054501 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:42.054603 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.054530 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:42.054603 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.054545 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:42.054603 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.054560 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d4c449-0024-4b94-91c0-41b4cb484bab-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:42.054603 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.054574 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:42.054603 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.054588 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:42.054860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.054622 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istiod-ca-cert" pod "1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" (UID: "1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:42.054860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.054792 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-data" pod "34d4c449-0024-4b94-91c0-41b4cb484bab" (UID: "34d4c449-0024-4b94-91c0-41b4cb484bab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:42.055025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.054889 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-workload-socket" pod "34d4c449-0024-4b94-91c0-41b4cb484bab" (UID: "34d4c449-0024-4b94-91c0-41b4cb484bab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:42.055191 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.055162 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-credential-socket" pod "1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" (UID: "1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:42.055770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.055744 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-token" pod "1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" (UID: "1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:42.055925 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.055902 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-envoy" pod "1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" (UID: "1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:42.056038 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.055981 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-envoy" pod "34d4c449-0024-4b94-91c0-41b4cb484bab" (UID: "34d4c449-0024-4b94-91c0-41b4cb484bab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:42.056128 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.056107 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-token" pod "34d4c449-0024-4b94-91c0-41b4cb484bab" (UID: "34d4c449-0024-4b94-91c0-41b4cb484bab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:42.056186 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.056172 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d4c449-0024-4b94-91c0-41b4cb484bab-kube-api-access-jwhm7" pod "34d4c449-0024-4b94-91c0-41b4cb484bab" (UID: "34d4c449-0024-4b94-91c0-41b4cb484bab"). InnerVolumeSpecName "kube-api-access-jwhm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:42.057162 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.057137 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-podinfo" pod "34d4c449-0024-4b94-91c0-41b4cb484bab" (UID: "34d4c449-0024-4b94-91c0-41b4cb484bab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:51:42.057245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.057228 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-kube-api-access-8xxtr" pod "1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" (UID: "1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3"). InnerVolumeSpecName "kube-api-access-8xxtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:42.057317 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.057300 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-podinfo" pod "1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" (UID: "1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.187468 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.187518 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.187533 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.187547 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.187560 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.187573 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.187586 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xxtr\" (UniqueName: \"kubernetes.io/projected/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-kube-api-access-8xxtr\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.187600 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.187613 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.187629 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwhm7\" (UniqueName: \"kubernetes.io/projected/34d4c449-0024-4b94-91c0-41b4cb484bab-kube-api-access-jwhm7\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.187643 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d4c449-0024-4b94-91c0-41b4cb484bab-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.187657 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d4c449-0024-4b94-91c0-41b4cb484bab-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.190184 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d"] Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.191319 5549 cpu_manager.go:414] "containerMap: removing container" podUID="34d4c449-0024-4b94-91c0-41b4cb484bab" containerName="istio-proxy" Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.191340 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d4c449-0024-4b94-91c0-41b4cb484bab" containerName="istio-proxy" Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.191372 5549 cpu_manager.go:414] "containerMap: removing container" podUID="1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" containerName="istio-proxy" Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.191380 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" containerName="istio-proxy" Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.191613 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d4c449-0024-4b94-91c0-41b4cb484bab" containerName="istio-proxy" Nov 02 17:51:42.202049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.191631 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" containerName="istio-proxy" Nov 02 17:51:42.204598 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.204565 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d"] Nov 02 17:51:42.204736 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.204685 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" Nov 02 17:51:42.210690 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.210656 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-7dbdb8-serving-cert\"" Nov 02 17:51:42.298010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.296631 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:42.393046 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.390427 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bd29fd-f13a-4516-913a-45ea08661487-metrics-certs\") pod \"maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d\" (UID: \"31bd29fd-f13a-4516-913a-45ea08661487\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" Nov 02 17:51:42.393046 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.390514 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31bd29fd-f13a-4516-913a-45ea08661487-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d\" (UID: \"31bd29fd-f13a-4516-913a-45ea08661487\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" Nov 02 17:51:42.393046 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.390657 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bd29fd-f13a-4516-913a-45ea08661487-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d\" (UID: \"31bd29fd-f13a-4516-913a-45ea08661487\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" Nov 02 17:51:42.393046 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.390796 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq842\" (UniqueName: \"kubernetes.io/projected/31bd29fd-f13a-4516-913a-45ea08661487-kube-api-access-tq842\") pod \"maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d\" (UID: \"31bd29fd-f13a-4516-913a-45ea08661487\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" Nov 02 17:51:42.432663 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.432631 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5"] Nov 02 17:51:42.439912 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.439888 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-4jwl5"] Nov 02 17:51:42.463348 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.463282 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8"] Nov 02 17:51:42.469871 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.469849 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-48qf8"] Nov 02 17:51:42.492109 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.492078 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq842\" (UniqueName: \"kubernetes.io/projected/31bd29fd-f13a-4516-913a-45ea08661487-kube-api-access-tq842\") pod \"maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d\" (UID: \"31bd29fd-f13a-4516-913a-45ea08661487\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" Nov 02 17:51:42.492244 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.492158 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bd29fd-f13a-4516-913a-45ea08661487-metrics-certs\") pod \"maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d\" (UID: \"31bd29fd-f13a-4516-913a-45ea08661487\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" Nov 02 17:51:42.492244 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.492232 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31bd29fd-f13a-4516-913a-45ea08661487-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d\" (UID: \"31bd29fd-f13a-4516-913a-45ea08661487\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" Nov 02 17:51:42.492406 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.492379 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bd29fd-f13a-4516-913a-45ea08661487-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d\" (UID: \"31bd29fd-f13a-4516-913a-45ea08661487\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" Nov 02 17:51:42.493082 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.493062 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bd29fd-f13a-4516-913a-45ea08661487-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d\" (UID: \"31bd29fd-f13a-4516-913a-45ea08661487\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" Nov 02 17:51:42.494579 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.494559 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bd29fd-f13a-4516-913a-45ea08661487-metrics-certs\") pod \"maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d\" (UID: \"31bd29fd-f13a-4516-913a-45ea08661487\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" Nov 02 17:51:42.494662 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.494597 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31bd29fd-f13a-4516-913a-45ea08661487-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d\" (UID: \"31bd29fd-f13a-4516-913a-45ea08661487\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" Nov 02 17:51:42.505946 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.505921 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq842\" (UniqueName: \"kubernetes.io/projected/31bd29fd-f13a-4516-913a-45ea08661487-kube-api-access-tq842\") pod \"maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d\" (UID: \"31bd29fd-f13a-4516-913a-45ea08661487\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" Nov 02 17:51:42.539870 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.538228 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" Nov 02 17:51:42.549441 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.549408 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:42.828092 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:42.828060 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d"] Nov 02 17:51:42.829196 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:51:42.829169 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31bd29fd_f13a_4516_913a_45ea08661487.slice/crio-afe9a36a8468f8bfa7b7f39a105fa837900c0c3099ba59a115296959ce2d2d08 WatchSource:0}: Error finding container afe9a36a8468f8bfa7b7f39a105fa837900c0c3099ba59a115296959ce2d2d08: Status 404 returned error can't find the container with id afe9a36a8468f8bfa7b7f39a105fa837900c0c3099ba59a115296959ce2d2d08 Nov 02 17:51:43.076025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:43.075003 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" event={"ID":"31bd29fd-f13a-4516-913a-45ea08661487","Type":"ContainerStarted","Data":"afe9a36a8468f8bfa7b7f39a105fa837900c0c3099ba59a115296959ce2d2d08"} Nov 02 17:51:43.664208 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:43.664132 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:44.092216 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:44.092181 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" event={"ID":"31bd29fd-f13a-4516-913a-45ea08661487","Type":"ContainerStarted","Data":"01590de2bc45996922b0fc0fd1e7332c44b44b30bd3aa952470cf41f41806d47"} Nov 02 17:51:44.092407 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:44.092264 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" Nov 02 17:51:44.112666 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:44.112602 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" podStartSLOduration=1.636054935 podStartE2EDuration="2.112589491s" totalImagesPullingTime="476.534556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:44.111768131 +0000 UTC m=+1075.900273661" watchObservedRunningTime="2026-11-02 17:51:44.112589491 +0000 UTC m=+1075.901095018" Nov 02 17:51:44.352865 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:44.352788 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3" path="/var/lib/kubelet/pods/1eb3c7bb-ce84-4b8b-9f70-8b1da442fdd3/volumes" Nov 02 17:51:44.353357 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:44.353341 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34d4c449-0024-4b94-91c0-41b4cb484bab" path="/var/lib/kubelet/pods/34d4c449-0024-4b94-91c0-41b4cb484bab/volumes" Nov 02 17:51:44.557032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:44.556989 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:45.466162 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:45.466124 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:46.212194 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:46.208831 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:47.098973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:47.098691 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:47.587254 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:51:47.587219 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:51:47.848027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:47.840390 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:48.018761 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:48.018722 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:48.671418 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:48.671383 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:48.844657 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:48.844626 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-5hb2w_b693ebb5-876a-426b-bf1d-6fdf00d3e96a/payload-processing/0.log" Nov 02 17:51:49.044118 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:49.044089 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-5hb2w_b693ebb5-876a-426b-bf1d-6fdf00d3e96a/payload-processing/0.log" Nov 02 17:51:49.211893 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:49.211859 5549 scope.go:122] "RemoveContainer" containerID="b08842c6275c30a290fa1fecdda09aef14d18bef2c0df484fa1713cc4bc8dbff" Nov 02 17:51:49.222534 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:49.222513 5549 scope.go:122] "RemoveContainer" containerID="d87c57dfa30342cde736c2232b13f2fa04540deddb0662332757fe74ca3461c4" Nov 02 17:51:49.232015 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:49.231992 5549 scope.go:122] "RemoveContainer" containerID="5ae610163563cc9b6bde1b33369b85fc7b868a250e86285c55849dcc29339d6b" Nov 02 17:51:49.499151 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:49.499115 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:49.875258 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:49.875220 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:50.057804 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:50.057767 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:50.107043 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:50.107012 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" Nov 02 17:51:50.584113 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:50.584078 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:50.894635 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:50.894533 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:51.184023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:51.180099 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:51.737546 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:51.737467 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:52.361349 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:52.360849 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:52.600881 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:52.600847 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:53.614187 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:53.614148 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:54.309202 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:54.309171 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:54.533579 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:54.533539 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:54.722972 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:54.721887 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:54.954756 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:54.954719 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:55.372638 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:55.372600 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl"] Nov 02 17:51:55.477369 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:55.477337 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:55.652497 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:55.652412 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:56.113008 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:56.109178 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:56.469661 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:56.469626 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:56.923026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:56.922908 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b"] Nov 02 17:51:56.934851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:56.934820 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b" Nov 02 17:51:56.943501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:56.943461 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-009a5f-dockercfg-29gfz\"" Nov 02 17:51:56.944101 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:56.944081 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-009a5f\"" Nov 02 17:51:56.945986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:56.945927 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg"] Nov 02 17:51:56.946374 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:56.946347 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:56.959371 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:56.959346 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b"] Nov 02 17:51:56.959491 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:56.959395 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg"] Nov 02 17:51:56.959491 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:56.959489 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg" Nov 02 17:51:57.029891 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.029844 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899469e3-9cf7-455e-9481-4637653d73af-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg\" (UID: \"899469e3-9cf7-455e-9481-4637653d73af\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg" Nov 02 17:51:57.045302 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.045256 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn4hw\" (UniqueName: \"kubernetes.io/projected/c724e9bd-faa6-4867-9e87-499fad60a383-kube-api-access-gn4hw\") pod \"payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b\" (UID: \"c724e9bd-faa6-4867-9e87-499fad60a383\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b" Nov 02 17:51:57.045793 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.045761 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c724e9bd-faa6-4867-9e87-499fad60a383-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b\" (UID: \"c724e9bd-faa6-4867-9e87-499fad60a383\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b" Nov 02 17:51:57.046035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.046013 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8v6q\" (UniqueName: \"kubernetes.io/projected/899469e3-9cf7-455e-9481-4637653d73af-kube-api-access-b8v6q\") pod \"payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg\" (UID: \"899469e3-9cf7-455e-9481-4637653d73af\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg" Nov 02 17:51:57.152941 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.152893 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899469e3-9cf7-455e-9481-4637653d73af-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg\" (UID: \"899469e3-9cf7-455e-9481-4637653d73af\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg" Nov 02 17:51:57.154529 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.154498 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn4hw\" (UniqueName: \"kubernetes.io/projected/c724e9bd-faa6-4867-9e87-499fad60a383-kube-api-access-gn4hw\") pod \"payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b\" (UID: \"c724e9bd-faa6-4867-9e87-499fad60a383\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b" Nov 02 17:51:57.154861 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.154841 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c724e9bd-faa6-4867-9e87-499fad60a383-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b\" (UID: \"c724e9bd-faa6-4867-9e87-499fad60a383\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b" Nov 02 17:51:57.155885 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.155862 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8v6q\" (UniqueName: \"kubernetes.io/projected/899469e3-9cf7-455e-9481-4637653d73af-kube-api-access-b8v6q\") pod \"payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg\" (UID: \"899469e3-9cf7-455e-9481-4637653d73af\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg" Nov 02 17:51:57.163230 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.155771 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c724e9bd-faa6-4867-9e87-499fad60a383-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b\" (UID: \"c724e9bd-faa6-4867-9e87-499fad60a383\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b" Nov 02 17:51:57.163416 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.154151 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899469e3-9cf7-455e-9481-4637653d73af-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg\" (UID: \"899469e3-9cf7-455e-9481-4637653d73af\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg" Nov 02 17:51:57.211975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.203784 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn4hw\" (UniqueName: \"kubernetes.io/projected/c724e9bd-faa6-4867-9e87-499fad60a383-kube-api-access-gn4hw\") pod \"payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b\" (UID: \"c724e9bd-faa6-4867-9e87-499fad60a383\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b" Nov 02 17:51:57.225974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.218787 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8v6q\" (UniqueName: \"kubernetes.io/projected/899469e3-9cf7-455e-9481-4637653d73af-kube-api-access-b8v6q\") pod \"payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg\" (UID: \"899469e3-9cf7-455e-9481-4637653d73af\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg" Nov 02 17:51:57.305477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.290405 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b" Nov 02 17:51:57.305477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.290591 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:57.305477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.291280 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg" Nov 02 17:51:57.544100 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.544046 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:57.785159 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.785086 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:57.807962 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.807927 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg"] Nov 02 17:51:57.810466 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:51:57.810440 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899469e3_9cf7_455e_9481_4637653d73af.slice/crio-21fa98de8d8fa1778d41814ed9cf5e5903b1faa09c5d1d4f288f8823d116d92c WatchSource:0}: Error finding container 21fa98de8d8fa1778d41814ed9cf5e5903b1faa09c5d1d4f288f8823d116d92c: Status 404 returned error can't find the container with id 21fa98de8d8fa1778d41814ed9cf5e5903b1faa09c5d1d4f288f8823d116d92c Nov 02 17:51:57.822815 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:57.822794 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b"] Nov 02 17:51:57.825879 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:51:57.825856 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc724e9bd_faa6_4867_9e87_499fad60a383.slice/crio-8b6e41c1968805e58452a954779a5cba8bcaf140fdc75cf958759033e05aec7b WatchSource:0}: Error finding container 8b6e41c1968805e58452a954779a5cba8bcaf140fdc75cf958759033e05aec7b: Status 404 returned error can't find the container with id 8b6e41c1968805e58452a954779a5cba8bcaf140fdc75cf958759033e05aec7b Nov 02 17:51:58.113753 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:58.113670 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:58.545975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:58.520411 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:58.694016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:58.669874 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq"] Nov 02 17:51:58.694016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:58.673695 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" podUID="b8420a84-6451-4cb6-8c46-82ac1c3562a3" containerName="istio-proxy" containerID="cri-o://42e00003b323c85b42b2554c2b83e44fe2d9a1599ed709e4da39cdc9b65d3099" gracePeriod=30 Nov 02 17:51:58.694016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:58.675008 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b" event={"ID":"c724e9bd-faa6-4867-9e87-499fad60a383","Type":"ContainerStarted","Data":"a077a2500ea5c50a08359789ec595f6ac8fee24a9bf9d33f185e28dd1ee5cca5"} Nov 02 17:51:58.694016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:58.675095 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b" event={"ID":"c724e9bd-faa6-4867-9e87-499fad60a383","Type":"ContainerStarted","Data":"8b6e41c1968805e58452a954779a5cba8bcaf140fdc75cf958759033e05aec7b"} Nov 02 17:51:58.694016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:58.675735 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b" Nov 02 17:51:58.694016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:58.678138 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg" event={"ID":"899469e3-9cf7-455e-9481-4637653d73af","Type":"ContainerStarted","Data":"9102b9f3453608648d94244fd2c2dbe45dfea621f0eb0a96e0df2b69d91be886"} Nov 02 17:51:58.694016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:58.678206 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg" event={"ID":"899469e3-9cf7-455e-9481-4637653d73af","Type":"ContainerStarted","Data":"21fa98de8d8fa1778d41814ed9cf5e5903b1faa09c5d1d4f288f8823d116d92c"} Nov 02 17:51:58.694016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:58.678563 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg" Nov 02 17:51:58.790599 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:58.771922 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg" podStartSLOduration=2.77190152 podStartE2EDuration="2.77190152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:58.770710273 +0000 UTC m=+1090.559215802" watchObservedRunningTime="2026-11-02 17:51:58.77190152 +0000 UTC m=+1090.560407048" Nov 02 17:51:58.826624 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:58.826506 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b" podStartSLOduration=2.826485071 podStartE2EDuration="2.826485071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:58.80929913 +0000 UTC m=+1090.597804650" watchObservedRunningTime="2026-11-02 17:51:58.826485071 +0000 UTC m=+1090.614990599" Nov 02 17:51:58.871008 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:58.870937 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:59.085270 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:59.085185 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:59.322009 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:59.321942 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:59.885645 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:51:59.885603 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:00.783130 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:00.782184 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:01.629976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:01.622928 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:02.338129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:02.338091 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:02.518015 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:02.517978 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:03.342100 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:03.342017 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:03.657831 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:03.657756 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5"] Nov 02 17:52:03.659520 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:03.659492 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:03.663544 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:03.663522 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5" Nov 02 17:52:03.673738 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:03.673703 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-4a5f09\"" Nov 02 17:52:03.677325 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:03.677300 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5"] Nov 02 17:52:03.677528 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:03.677504 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-4a5f09-dockercfg-4l2tn\"" Nov 02 17:52:03.779380 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:03.779346 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n466v\" (UniqueName: \"kubernetes.io/projected/c4c1e656-62c3-41d7-b29d-c120c194315e-kube-api-access-n466v\") pod \"payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5\" (UID: \"c4c1e656-62c3-41d7-b29d-c120c194315e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5" Nov 02 17:52:03.779640 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:03.779613 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c1e656-62c3-41d7-b29d-c120c194315e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5\" (UID: \"c4c1e656-62c3-41d7-b29d-c120c194315e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5" Nov 02 17:52:03.886080 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:03.885148 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c1e656-62c3-41d7-b29d-c120c194315e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5\" (UID: \"c4c1e656-62c3-41d7-b29d-c120c194315e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5" Nov 02 17:52:03.886080 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:03.885403 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n466v\" (UniqueName: \"kubernetes.io/projected/c4c1e656-62c3-41d7-b29d-c120c194315e-kube-api-access-n466v\") pod \"payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5\" (UID: \"c4c1e656-62c3-41d7-b29d-c120c194315e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5" Nov 02 17:52:03.906538 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:03.887065 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c1e656-62c3-41d7-b29d-c120c194315e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5\" (UID: \"c4c1e656-62c3-41d7-b29d-c120c194315e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5" Nov 02 17:52:04.072995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.072726 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n466v\" (UniqueName: \"kubernetes.io/projected/c4c1e656-62c3-41d7-b29d-c120c194315e-kube-api-access-n466v\") pod \"payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5\" (UID: \"c4c1e656-62c3-41d7-b29d-c120c194315e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5" Nov 02 17:52:04.218650 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.218612 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:04.258508 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.258480 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:52:04.281285 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.281257 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5" Nov 02 17:52:04.380008 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.378398 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:04.423995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.420455 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8420a84-6451-4cb6-8c46-82ac1c3562a3-kube-api-access-rmcwn\" (UniqueName: \"kubernetes.io/projected/b8420a84-6451-4cb6-8c46-82ac1c3562a3-kube-api-access-rmcwn\") pod \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " Nov 02 17:52:04.423995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.420516 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-podinfo\") pod \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " Nov 02 17:52:04.423995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.420572 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-data\") pod \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " Nov 02 17:52:04.423995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.420663 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-workload-socket\") pod \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " Nov 02 17:52:04.423995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.420807 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-token\" (UniqueName: \"kubernetes.io/projected/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-token\") pod \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " Nov 02 17:52:04.423995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.420841 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istiod-ca-cert\") pod \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " Nov 02 17:52:04.423995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.420883 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-envoy\") pod \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " Nov 02 17:52:04.423995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.420921 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-credential-socket\") pod \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " Nov 02 17:52:04.423995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.420993 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-workload-certs\") pod \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\" (UID: \"b8420a84-6451-4cb6-8c46-82ac1c3562a3\") " Nov 02 17:52:04.437012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.427099 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-workload-certs" pod "b8420a84-6451-4cb6-8c46-82ac1c3562a3" (UID: "b8420a84-6451-4cb6-8c46-82ac1c3562a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:04.437012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.434209 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-workload-socket" pod "b8420a84-6451-4cb6-8c46-82ac1c3562a3" (UID: "b8420a84-6451-4cb6-8c46-82ac1c3562a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:04.437012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.434290 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-data" pod "b8420a84-6451-4cb6-8c46-82ac1c3562a3" (UID: "b8420a84-6451-4cb6-8c46-82ac1c3562a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:04.437012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.435218 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istiod-ca-cert" pod "b8420a84-6451-4cb6-8c46-82ac1c3562a3" (UID: "b8420a84-6451-4cb6-8c46-82ac1c3562a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:04.437012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.435356 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-credential-socket" pod "b8420a84-6451-4cb6-8c46-82ac1c3562a3" (UID: "b8420a84-6451-4cb6-8c46-82ac1c3562a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:04.440404 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.440372 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-token" pod "b8420a84-6451-4cb6-8c46-82ac1c3562a3" (UID: "b8420a84-6451-4cb6-8c46-82ac1c3562a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:04.440654 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.440499 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-podinfo" pod "b8420a84-6451-4cb6-8c46-82ac1c3562a3" (UID: "b8420a84-6451-4cb6-8c46-82ac1c3562a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:52:04.440654 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.440555 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-envoy" pod "b8420a84-6451-4cb6-8c46-82ac1c3562a3" (UID: "b8420a84-6451-4cb6-8c46-82ac1c3562a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:04.443384 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.443347 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8420a84-6451-4cb6-8c46-82ac1c3562a3-kube-api-access-rmcwn" pod "b8420a84-6451-4cb6-8c46-82ac1c3562a3" (UID: "b8420a84-6451-4cb6-8c46-82ac1c3562a3"). InnerVolumeSpecName "kube-api-access-rmcwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:04.513686 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.513650 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf"] Nov 02 17:52:04.524076 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.524044 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmcwn\" (UniqueName: \"kubernetes.io/projected/b8420a84-6451-4cb6-8c46-82ac1c3562a3-kube-api-access-rmcwn\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:04.524076 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.524076 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:04.524345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.524092 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:04.524345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.524107 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:04.524345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.524121 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:04.524345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.524135 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:04.524345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.524148 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:04.524345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.524162 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:04.524345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.524177 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8420a84-6451-4cb6-8c46-82ac1c3562a3-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:04.532851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.532820 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q"] Nov 02 17:52:04.660540 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.660494 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:04.689043 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.689014 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5"] Nov 02 17:52:04.689836 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:52:04.689815 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c1e656_62c3_41d7_b29d_c120c194315e.slice/crio-547dbefa23be7df52fd80da249328683d91b4a7413512b2f5dd267588a63696e WatchSource:0}: Error finding container 547dbefa23be7df52fd80da249328683d91b4a7413512b2f5dd267588a63696e: Status 404 returned error can't find the container with id 547dbefa23be7df52fd80da249328683d91b4a7413512b2f5dd267588a63696e Nov 02 17:52:04.722748 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.722682 5549 generic.go:356] "Generic (PLEG): container finished" podID="b8420a84-6451-4cb6-8c46-82ac1c3562a3" containerID="42e00003b323c85b42b2554c2b83e44fe2d9a1599ed709e4da39cdc9b65d3099" exitCode=0 Nov 02 17:52:04.722748 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.722738 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" Nov 02 17:52:04.722916 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.722765 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" event={"ID":"b8420a84-6451-4cb6-8c46-82ac1c3562a3","Type":"ContainerDied","Data":"42e00003b323c85b42b2554c2b83e44fe2d9a1599ed709e4da39cdc9b65d3099"} Nov 02 17:52:04.722916 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.722826 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq" event={"ID":"b8420a84-6451-4cb6-8c46-82ac1c3562a3","Type":"ContainerDied","Data":"782dc658cc4ea32960a76350047d1d3d8c49b836e7753fdd79099b259481bf0b"} Nov 02 17:52:04.722916 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.722852 5549 scope.go:122] "RemoveContainer" containerID="42e00003b323c85b42b2554c2b83e44fe2d9a1599ed709e4da39cdc9b65d3099" Nov 02 17:52:04.724359 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.724337 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5" event={"ID":"c4c1e656-62c3-41d7-b29d-c120c194315e","Type":"ContainerStarted","Data":"547dbefa23be7df52fd80da249328683d91b4a7413512b2f5dd267588a63696e"} Nov 02 17:52:04.738867 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.738848 5549 scope.go:122] "RemoveContainer" containerID="f7e4377f8593d2d7269992ca7c390632876106032c0990bfda53896bec8d4f0e" Nov 02 17:52:04.753037 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.753018 5549 scope.go:122] "RemoveContainer" containerID="42e00003b323c85b42b2554c2b83e44fe2d9a1599ed709e4da39cdc9b65d3099" Nov 02 17:52:04.753413 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:52:04.753387 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e00003b323c85b42b2554c2b83e44fe2d9a1599ed709e4da39cdc9b65d3099\": container with ID starting with 42e00003b323c85b42b2554c2b83e44fe2d9a1599ed709e4da39cdc9b65d3099 not found: ID does not exist" containerID="42e00003b323c85b42b2554c2b83e44fe2d9a1599ed709e4da39cdc9b65d3099" Nov 02 17:52:04.753516 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.753423 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e00003b323c85b42b2554c2b83e44fe2d9a1599ed709e4da39cdc9b65d3099"} err="failed to get container status \"42e00003b323c85b42b2554c2b83e44fe2d9a1599ed709e4da39cdc9b65d3099\": rpc error: code = NotFound desc = could not find container \"42e00003b323c85b42b2554c2b83e44fe2d9a1599ed709e4da39cdc9b65d3099\": container with ID starting with 42e00003b323c85b42b2554c2b83e44fe2d9a1599ed709e4da39cdc9b65d3099 not found: ID does not exist" Nov 02 17:52:04.753516 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.753446 5549 scope.go:122] "RemoveContainer" containerID="f7e4377f8593d2d7269992ca7c390632876106032c0990bfda53896bec8d4f0e" Nov 02 17:52:04.753723 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:52:04.753703 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e4377f8593d2d7269992ca7c390632876106032c0990bfda53896bec8d4f0e\": container with ID starting with f7e4377f8593d2d7269992ca7c390632876106032c0990bfda53896bec8d4f0e not found: ID does not exist" containerID="f7e4377f8593d2d7269992ca7c390632876106032c0990bfda53896bec8d4f0e" Nov 02 17:52:04.753764 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.753732 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e4377f8593d2d7269992ca7c390632876106032c0990bfda53896bec8d4f0e"} err="failed to get container status \"f7e4377f8593d2d7269992ca7c390632876106032c0990bfda53896bec8d4f0e\": rpc error: code = NotFound desc = could not find container \"f7e4377f8593d2d7269992ca7c390632876106032c0990bfda53896bec8d4f0e\": container with ID starting with f7e4377f8593d2d7269992ca7c390632876106032c0990bfda53896bec8d4f0e not found: ID does not exist" Nov 02 17:52:04.766583 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.766557 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq"] Nov 02 17:52:04.777944 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.777922 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cd2738-openshift-default-6fcb5885b7-7qclq"] Nov 02 17:52:04.839892 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:04.839859 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:05.009321 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.009240 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:05.068854 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.068824 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq"] Nov 02 17:52:05.069510 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.069497 5549 cpu_manager.go:414] "containerMap: removing container" podUID="b8420a84-6451-4cb6-8c46-82ac1c3562a3" containerName="istio-proxy" Nov 02 17:52:05.069561 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.069512 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8420a84-6451-4cb6-8c46-82ac1c3562a3" containerName="istio-proxy" Nov 02 17:52:05.069561 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.069541 5549 cpu_manager.go:414] "containerMap: removing container" podUID="b8420a84-6451-4cb6-8c46-82ac1c3562a3" containerName="istio-proxy" Nov 02 17:52:05.069561 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.069546 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8420a84-6451-4cb6-8c46-82ac1c3562a3" containerName="istio-proxy" Nov 02 17:52:05.069763 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.069754 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8420a84-6451-4cb6-8c46-82ac1c3562a3" containerName="istio-proxy" Nov 02 17:52:05.073520 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.073500 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq" Nov 02 17:52:05.085783 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.085759 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6098d0\"" Nov 02 17:52:05.086058 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.086040 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6098d0-dockercfg-gcvfn\"" Nov 02 17:52:05.087731 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.087709 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq"] Nov 02 17:52:05.276023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.191461 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:05.276023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.195466 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8q9d\" (UniqueName: \"kubernetes.io/projected/b63ebb7d-159f-4648-8d9d-a96c608d3799-kube-api-access-b8q9d\") pod \"payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq\" (UID: \"b63ebb7d-159f-4648-8d9d-a96c608d3799\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq" Nov 02 17:52:05.276023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.211519 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b63ebb7d-159f-4648-8d9d-a96c608d3799-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq\" (UID: \"b63ebb7d-159f-4648-8d9d-a96c608d3799\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq" Nov 02 17:52:05.276023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.225105 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx"] Nov 02 17:52:05.276023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.226453 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8420a84-6451-4cb6-8c46-82ac1c3562a3" containerName="istio-proxy" Nov 02 17:52:05.276023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.249582 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx" Nov 02 17:52:05.337976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.337035 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b63ebb7d-159f-4648-8d9d-a96c608d3799-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq\" (UID: \"b63ebb7d-159f-4648-8d9d-a96c608d3799\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq" Nov 02 17:52:05.337976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.337213 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8q9d\" (UniqueName: \"kubernetes.io/projected/b63ebb7d-159f-4648-8d9d-a96c608d3799-kube-api-access-b8q9d\") pod \"payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq\" (UID: \"b63ebb7d-159f-4648-8d9d-a96c608d3799\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq" Nov 02 17:52:05.360656 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.338526 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b63ebb7d-159f-4648-8d9d-a96c608d3799-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq\" (UID: \"b63ebb7d-159f-4648-8d9d-a96c608d3799\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq" Nov 02 17:52:05.385977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.372257 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx"] Nov 02 17:52:05.426893 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.426848 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8q9d\" (UniqueName: \"kubernetes.io/projected/b63ebb7d-159f-4648-8d9d-a96c608d3799-kube-api-access-b8q9d\") pod \"payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq\" (UID: \"b63ebb7d-159f-4648-8d9d-a96c608d3799\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq" Nov 02 17:52:05.439277 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.439242 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa505463-b683-461b-a1d3-2a0d4cb1e693-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx\" (UID: \"fa505463-b683-461b-a1d3-2a0d4cb1e693\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx" Nov 02 17:52:05.439435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.439344 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tztq\" (UniqueName: \"kubernetes.io/projected/fa505463-b683-461b-a1d3-2a0d4cb1e693-kube-api-access-7tztq\") pod \"payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx\" (UID: \"fa505463-b683-461b-a1d3-2a0d4cb1e693\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx" Nov 02 17:52:05.540983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.540880 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tztq\" (UniqueName: \"kubernetes.io/projected/fa505463-b683-461b-a1d3-2a0d4cb1e693-kube-api-access-7tztq\") pod \"payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx\" (UID: \"fa505463-b683-461b-a1d3-2a0d4cb1e693\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx" Nov 02 17:52:05.541252 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.541231 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa505463-b683-461b-a1d3-2a0d4cb1e693-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx\" (UID: \"fa505463-b683-461b-a1d3-2a0d4cb1e693\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx" Nov 02 17:52:05.541941 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.541921 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa505463-b683-461b-a1d3-2a0d4cb1e693-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx\" (UID: \"fa505463-b683-461b-a1d3-2a0d4cb1e693\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx" Nov 02 17:52:05.550447 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.550417 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tztq\" (UniqueName: \"kubernetes.io/projected/fa505463-b683-461b-a1d3-2a0d4cb1e693-kube-api-access-7tztq\") pod \"payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx\" (UID: \"fa505463-b683-461b-a1d3-2a0d4cb1e693\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx" Nov 02 17:52:05.689434 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.689397 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq" Nov 02 17:52:05.734454 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.734420 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5" event={"ID":"c4c1e656-62c3-41d7-b29d-c120c194315e","Type":"ContainerStarted","Data":"e0e2e009bf01d8995d01a42ad47691055f95eb0a9800828222cd142d9e20406d"} Nov 02 17:52:05.734628 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.734500 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5" Nov 02 17:52:05.748312 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.748288 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx" Nov 02 17:52:05.760912 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.760837 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5" podStartSLOduration=2.760819048 podStartE2EDuration="2.760819048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:05.759094807 +0000 UTC m=+1097.547600373" watchObservedRunningTime="2026-11-02 17:52:05.760819048 +0000 UTC m=+1097.549324579" Nov 02 17:52:05.803483 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:05.803301 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:06.013411 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:06.013383 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq"] Nov 02 17:52:06.054399 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:06.054336 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx"] Nov 02 17:52:06.218277 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:06.218241 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg"] Nov 02 17:52:06.218531 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:06.218461 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" podUID="78d911a8-7c46-4ea5-b75e-919fad9e1d22" containerName="istio-proxy" containerID="cri-o://0518a8bc7b9a48726ea2108a3cce5263574462fd176a311f28ce8abec003c792" gracePeriod=30 Nov 02 17:52:06.221574 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:06.221545 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh"] Nov 02 17:52:06.221749 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:06.221728 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" podUID="57692113-df46-4797-9fd6-49b5aad6f5f0" containerName="istio-proxy" containerID="cri-o://e859cf0e94e063033c940f8123c0d743d1479552a1d487f516994329efb799bb" gracePeriod=30 Nov 02 17:52:06.321248 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:06.321179 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8420a84-6451-4cb6-8c46-82ac1c3562a3" path="/var/lib/kubelet/pods/b8420a84-6451-4cb6-8c46-82ac1c3562a3/volumes" Nov 02 17:52:06.744116 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:06.744081 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq" event={"ID":"b63ebb7d-159f-4648-8d9d-a96c608d3799","Type":"ContainerStarted","Data":"3cab779930a20d895e4464537bfe2528955873a6c10c96660bf81518c42455b6"} Nov 02 17:52:06.744537 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:06.744143 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq" event={"ID":"b63ebb7d-159f-4648-8d9d-a96c608d3799","Type":"ContainerStarted","Data":"95704fc595743149b303afca7b5c8dcebf856c7ae801e3d7a6bdfd6eedaae02e"} Nov 02 17:52:06.744537 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:06.744176 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq" Nov 02 17:52:06.745926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:06.745893 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx" event={"ID":"fa505463-b683-461b-a1d3-2a0d4cb1e693","Type":"ContainerStarted","Data":"ab8d70086dd5a807c03e1683965178f427c25ade413b7446bf7dbd8861b87745"} Nov 02 17:52:06.746083 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:06.745946 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx" event={"ID":"fa505463-b683-461b-a1d3-2a0d4cb1e693","Type":"ContainerStarted","Data":"f53204edfda31932e2105baa2573b3d9313cc3ee7d51ecdfc48060bd59926033"} Nov 02 17:52:06.746083 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:06.746072 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx" Nov 02 17:52:06.781391 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:06.781340 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq" podStartSLOduration=1.781322337 podStartE2EDuration="1.781322337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:06.77891026 +0000 UTC m=+1098.567415789" watchObservedRunningTime="2026-11-02 17:52:06.781322337 +0000 UTC m=+1098.569827866" Nov 02 17:52:06.873600 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:06.873552 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:07.598550 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:07.598493 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx" podStartSLOduration=2.598477536 podStartE2EDuration="2.598477536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:06.834254126 +0000 UTC m=+1098.622759698" watchObservedRunningTime="2026-11-02 17:52:07.598477536 +0000 UTC m=+1099.386983078" Nov 02 17:52:07.600815 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:07.600793 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:52:08.023639 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.023599 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p"] Nov 02 17:52:08.035040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.035008 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" Nov 02 17:52:08.051858 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.051279 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p"] Nov 02 17:52:08.067170 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.067135 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932f3be9-81c1-4473-8b41-46870d0fc7fa-metrics-certs\") pod \"maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p\" (UID: \"932f3be9-81c1-4473-8b41-46870d0fc7fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" Nov 02 17:52:08.067326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.067200 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/932f3be9-81c1-4473-8b41-46870d0fc7fa-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p\" (UID: \"932f3be9-81c1-4473-8b41-46870d0fc7fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" Nov 02 17:52:08.067326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.067239 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-948ql\" (UniqueName: \"kubernetes.io/projected/932f3be9-81c1-4473-8b41-46870d0fc7fa-kube-api-access-948ql\") pod \"maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p\" (UID: \"932f3be9-81c1-4473-8b41-46870d0fc7fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" Nov 02 17:52:08.067598 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.067558 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/932f3be9-81c1-4473-8b41-46870d0fc7fa-maas-api-tls\") pod \"maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p\" (UID: \"932f3be9-81c1-4473-8b41-46870d0fc7fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" Nov 02 17:52:08.082562 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.082533 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-009a5f-serving-cert\"" Nov 02 17:52:08.158334 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.158298 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:08.168859 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.168827 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/932f3be9-81c1-4473-8b41-46870d0fc7fa-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p\" (UID: \"932f3be9-81c1-4473-8b41-46870d0fc7fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" Nov 02 17:52:08.168995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.168863 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-948ql\" (UniqueName: \"kubernetes.io/projected/932f3be9-81c1-4473-8b41-46870d0fc7fa-kube-api-access-948ql\") pod \"maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p\" (UID: \"932f3be9-81c1-4473-8b41-46870d0fc7fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" Nov 02 17:52:08.169166 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.169143 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/932f3be9-81c1-4473-8b41-46870d0fc7fa-maas-api-tls\") pod \"maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p\" (UID: \"932f3be9-81c1-4473-8b41-46870d0fc7fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" Nov 02 17:52:08.169265 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.169246 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932f3be9-81c1-4473-8b41-46870d0fc7fa-metrics-certs\") pod \"maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p\" (UID: \"932f3be9-81c1-4473-8b41-46870d0fc7fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" Nov 02 17:52:08.169635 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.169607 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/932f3be9-81c1-4473-8b41-46870d0fc7fa-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p\" (UID: \"932f3be9-81c1-4473-8b41-46870d0fc7fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" Nov 02 17:52:08.171775 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.171751 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/932f3be9-81c1-4473-8b41-46870d0fc7fa-maas-api-tls\") pod \"maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p\" (UID: \"932f3be9-81c1-4473-8b41-46870d0fc7fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" Nov 02 17:52:08.171873 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.171793 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932f3be9-81c1-4473-8b41-46870d0fc7fa-metrics-certs\") pod \"maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p\" (UID: \"932f3be9-81c1-4473-8b41-46870d0fc7fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" Nov 02 17:52:08.180588 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.180560 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-948ql\" (UniqueName: \"kubernetes.io/projected/932f3be9-81c1-4473-8b41-46870d0fc7fa-kube-api-access-948ql\") pod \"maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p\" (UID: \"932f3be9-81c1-4473-8b41-46870d0fc7fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" Nov 02 17:52:08.355564 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.355483 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" Nov 02 17:52:08.531158 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:08.517266 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:09.254426 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:09.237902 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p"] Nov 02 17:52:09.268900 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:52:09.268774 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932f3be9_81c1_4473_8b41_46870d0fc7fa.slice/crio-0f78348fa70a5379742761daf8204e48a1f101e70859361e715afef89a371d6e WatchSource:0}: Error finding container 0f78348fa70a5379742761daf8204e48a1f101e70859361e715afef89a371d6e: Status 404 returned error can't find the container with id 0f78348fa70a5379742761daf8204e48a1f101e70859361e715afef89a371d6e Nov 02 17:52:09.270476 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:09.270444 5549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:52:09.543032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:09.541138 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q" podUID="536ab9a4-1941-4ca5-8a50-691e53ee1148" containerName="payload-pre-processing" containerID="cri-o://fe690e17705ad7ec3676c817ec37cc3dfe5869d81cea238c9f754dcdb324a822" gracePeriod=25 Nov 02 17:52:09.543032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:09.541357 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf" podUID="df101814-f22d-4abd-888c-b3288a7707a1" containerName="payload-processing" containerID="cri-o://b1463edba5fd24ab8f020abfc1740f153cdb2d828386a54f12a3a303842c9b59" gracePeriod=25 Nov 02 17:52:09.558222 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:09.549873 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:09.709292 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:09.709266 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg" Nov 02 17:52:09.808985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:09.805627 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:09.935522 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:09.935488 5549 generic.go:356] "Generic (PLEG): container finished" podID="df101814-f22d-4abd-888c-b3288a7707a1" containerID="b1463edba5fd24ab8f020abfc1740f153cdb2d828386a54f12a3a303842c9b59" exitCode=0 Nov 02 17:52:09.935883 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:09.935682 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf" event={"ID":"df101814-f22d-4abd-888c-b3288a7707a1","Type":"ContainerDied","Data":"b1463edba5fd24ab8f020abfc1740f153cdb2d828386a54f12a3a303842c9b59"} Nov 02 17:52:09.952846 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:09.952698 5549 generic.go:356] "Generic (PLEG): container finished" podID="536ab9a4-1941-4ca5-8a50-691e53ee1148" containerID="fe690e17705ad7ec3676c817ec37cc3dfe5869d81cea238c9f754dcdb324a822" exitCode=0 Nov 02 17:52:09.952846 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:09.952739 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q" event={"ID":"536ab9a4-1941-4ca5-8a50-691e53ee1148","Type":"ContainerDied","Data":"fe690e17705ad7ec3676c817ec37cc3dfe5869d81cea238c9f754dcdb324a822"} Nov 02 17:52:09.958155 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:09.955869 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" event={"ID":"932f3be9-81c1-4473-8b41-46870d0fc7fa","Type":"ContainerStarted","Data":"c50e625665d23d90366261dc7016c64c6a4f9ac0a748ee82965ab2b58cb9fdb6"} Nov 02 17:52:09.958155 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:09.955907 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" event={"ID":"932f3be9-81c1-4473-8b41-46870d0fc7fa","Type":"ContainerStarted","Data":"0f78348fa70a5379742761daf8204e48a1f101e70859361e715afef89a371d6e"} Nov 02 17:52:09.958155 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:09.957823 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" Nov 02 17:52:09.972885 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:09.972855 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf" Nov 02 17:52:10.024385 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.024356 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q" Nov 02 17:52:10.051178 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.051082 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" podStartSLOduration=2.588653628 podStartE2EDuration="3.051065863s" totalImagesPullingTime="462.412235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:10.009490957 +0000 UTC m=+1101.797996534" watchObservedRunningTime="2026-11-02 17:52:10.051065863 +0000 UTC m=+1101.839571392" Nov 02 17:52:10.168984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.146077 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df101814-f22d-4abd-888c-b3288a7707a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df101814-f22d-4abd-888c-b3288a7707a1-plugins-config-volume\") pod \"df101814-f22d-4abd-888c-b3288a7707a1\" (UID: \"df101814-f22d-4abd-888c-b3288a7707a1\") " Nov 02 17:52:10.190303 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.190268 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df101814-f22d-4abd-888c-b3288a7707a1-kube-api-access-w2sqc\" (UniqueName: \"kubernetes.io/projected/df101814-f22d-4abd-888c-b3288a7707a1-kube-api-access-w2sqc\") pod \"df101814-f22d-4abd-888c-b3288a7707a1\" (UID: \"df101814-f22d-4abd-888c-b3288a7707a1\") " Nov 02 17:52:10.210267 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.210226 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df101814-f22d-4abd-888c-b3288a7707a1-plugins-config-volume" pod "df101814-f22d-4abd-888c-b3288a7707a1" (UID: "df101814-f22d-4abd-888c-b3288a7707a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:10.255234 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.255197 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df101814-f22d-4abd-888c-b3288a7707a1-kube-api-access-w2sqc" pod "df101814-f22d-4abd-888c-b3288a7707a1" (UID: "df101814-f22d-4abd-888c-b3288a7707a1"). InnerVolumeSpecName "kube-api-access-w2sqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:10.292479 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.292431 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536ab9a4-1941-4ca5-8a50-691e53ee1148-kube-api-access-fpll9\" (UniqueName: \"kubernetes.io/projected/536ab9a4-1941-4ca5-8a50-691e53ee1148-kube-api-access-fpll9\") pod \"536ab9a4-1941-4ca5-8a50-691e53ee1148\" (UID: \"536ab9a4-1941-4ca5-8a50-691e53ee1148\") " Nov 02 17:52:10.293268 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.293239 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/536ab9a4-1941-4ca5-8a50-691e53ee1148-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536ab9a4-1941-4ca5-8a50-691e53ee1148-plugins-config-volume\") pod \"536ab9a4-1941-4ca5-8a50-691e53ee1148\" (UID: \"536ab9a4-1941-4ca5-8a50-691e53ee1148\") " Nov 02 17:52:10.294359 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.294338 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df101814-f22d-4abd-888c-b3288a7707a1-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:10.294498 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.294484 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2sqc\" (UniqueName: \"kubernetes.io/projected/df101814-f22d-4abd-888c-b3288a7707a1-kube-api-access-w2sqc\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:10.294930 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.294908 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536ab9a4-1941-4ca5-8a50-691e53ee1148-plugins-config-volume" pod "536ab9a4-1941-4ca5-8a50-691e53ee1148" (UID: "536ab9a4-1941-4ca5-8a50-691e53ee1148"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:10.298511 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.298474 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536ab9a4-1941-4ca5-8a50-691e53ee1148-kube-api-access-fpll9" pod "536ab9a4-1941-4ca5-8a50-691e53ee1148" (UID: "536ab9a4-1941-4ca5-8a50-691e53ee1148"). InnerVolumeSpecName "kube-api-access-fpll9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:10.404240 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.404199 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpll9\" (UniqueName: \"kubernetes.io/projected/536ab9a4-1941-4ca5-8a50-691e53ee1148-kube-api-access-fpll9\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:10.404491 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.404470 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536ab9a4-1941-4ca5-8a50-691e53ee1148-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:10.407390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.407347 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" podUID="5186c8ad-6d18-4e3e-b71c-62ec5af8222a" containerName="main" containerID="cri-o://846c9430e0432c3d748d1dc27643c8ea00788800204e35f2e90ad94059321e52" gracePeriod=45 Nov 02 17:52:10.736928 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.734634 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b" Nov 02 17:52:10.736928 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.736845 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:10.818859 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.818815 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:52:10.936337 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.936246 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-dshm\") pod \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " Nov 02 17:52:10.936508 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.936402 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-model-cache\") pod \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " Nov 02 17:52:10.936508 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.936442 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-home\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-home\") pod \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " Nov 02 17:52:10.936508 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.936494 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-tls-certs\" (UniqueName: \"kubernetes.io/secret/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-tls-certs\") pod \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " Nov 02 17:52:10.936667 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.936531 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-kube-api-access-9g5zv\" (UniqueName: \"kubernetes.io/projected/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-kube-api-access-9g5zv\") pod \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " Nov 02 17:52:10.936717 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.936684 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-model-cache" pod "5186c8ad-6d18-4e3e-b71c-62ec5af8222a" (UID: "5186c8ad-6d18-4e3e-b71c-62ec5af8222a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:10.936768 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.936686 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-tmp-dir\") pod \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\" (UID: \"5186c8ad-6d18-4e3e-b71c-62ec5af8222a\") " Nov 02 17:52:10.936822 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.936771 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-home" pod "5186c8ad-6d18-4e3e-b71c-62ec5af8222a" (UID: "5186c8ad-6d18-4e3e-b71c-62ec5af8222a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:10.936926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.936899 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-tmp-dir" pod "5186c8ad-6d18-4e3e-b71c-62ec5af8222a" (UID: "5186c8ad-6d18-4e3e-b71c-62ec5af8222a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:10.937663 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.937638 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:10.937767 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.937672 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:10.937767 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.937689 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:10.938939 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.938905 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-kube-api-access-9g5zv" pod "5186c8ad-6d18-4e3e-b71c-62ec5af8222a" (UID: "5186c8ad-6d18-4e3e-b71c-62ec5af8222a"). InnerVolumeSpecName "kube-api-access-9g5zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:10.938939 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.938913 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-tls-certs" pod "5186c8ad-6d18-4e3e-b71c-62ec5af8222a" (UID: "5186c8ad-6d18-4e3e-b71c-62ec5af8222a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:52:10.939113 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.938981 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-dshm" pod "5186c8ad-6d18-4e3e-b71c-62ec5af8222a" (UID: "5186c8ad-6d18-4e3e-b71c-62ec5af8222a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:10.975150 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.975100 5549 generic.go:356] "Generic (PLEG): container finished" podID="5186c8ad-6d18-4e3e-b71c-62ec5af8222a" containerID="846c9430e0432c3d748d1dc27643c8ea00788800204e35f2e90ad94059321e52" exitCode=0 Nov 02 17:52:10.975357 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.975327 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" event={"ID":"5186c8ad-6d18-4e3e-b71c-62ec5af8222a","Type":"ContainerDied","Data":"846c9430e0432c3d748d1dc27643c8ea00788800204e35f2e90ad94059321e52"} Nov 02 17:52:10.975482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.975401 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" event={"ID":"5186c8ad-6d18-4e3e-b71c-62ec5af8222a","Type":"ContainerDied","Data":"e5a126c1372d34b00878cc4c1b212c71b7631be5e99bde85beeb6fbbbfca737c"} Nov 02 17:52:10.975482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.975427 5549 scope.go:122] "RemoveContainer" containerID="846c9430e0432c3d748d1dc27643c8ea00788800204e35f2e90ad94059321e52" Nov 02 17:52:10.975722 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:10.975267 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl" Nov 02 17:52:11.008149 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.008116 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf" event={"ID":"df101814-f22d-4abd-888c-b3288a7707a1","Type":"ContainerDied","Data":"4363bbd7b88e8a3ade46d82de1a379d84c8f4f3a52638c24c6e6fc976ad6617b"} Nov 02 17:52:11.008759 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.008726 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf" Nov 02 17:52:11.011933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.011906 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q" Nov 02 17:52:11.012553 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.012530 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q" event={"ID":"536ab9a4-1941-4ca5-8a50-691e53ee1148","Type":"ContainerDied","Data":"1dc538fa23fc1271cfada922555ffcaaaf63c52214cc57a314d241885821279d"} Nov 02 17:52:11.025105 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.025081 5549 scope.go:122] "RemoveContainer" containerID="846c9430e0432c3d748d1dc27643c8ea00788800204e35f2e90ad94059321e52" Nov 02 17:52:11.025368 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:52:11.025349 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"846c9430e0432c3d748d1dc27643c8ea00788800204e35f2e90ad94059321e52\": container with ID starting with 846c9430e0432c3d748d1dc27643c8ea00788800204e35f2e90ad94059321e52 not found: ID does not exist" containerID="846c9430e0432c3d748d1dc27643c8ea00788800204e35f2e90ad94059321e52" Nov 02 17:52:11.025425 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.025375 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846c9430e0432c3d748d1dc27643c8ea00788800204e35f2e90ad94059321e52"} err="failed to get container status \"846c9430e0432c3d748d1dc27643c8ea00788800204e35f2e90ad94059321e52\": rpc error: code = NotFound desc = could not find container \"846c9430e0432c3d748d1dc27643c8ea00788800204e35f2e90ad94059321e52\": container with ID starting with 846c9430e0432c3d748d1dc27643c8ea00788800204e35f2e90ad94059321e52 not found: ID does not exist" Nov 02 17:52:11.025425 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.025391 5549 scope.go:122] "RemoveContainer" containerID="b1463edba5fd24ab8f020abfc1740f153cdb2d828386a54f12a3a303842c9b59" Nov 02 17:52:11.034886 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.034859 5549 scope.go:122] "RemoveContainer" containerID="fe690e17705ad7ec3676c817ec37cc3dfe5869d81cea238c9f754dcdb324a822" Nov 02 17:52:11.038702 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.038682 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.038702 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.038704 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.038830 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.038714 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g5zv\" (UniqueName: \"kubernetes.io/projected/5186c8ad-6d18-4e3e-b71c-62ec5af8222a-kube-api-access-9g5zv\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.043819 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.043797 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl"] Nov 02 17:52:11.051673 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.051652 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-4c9298/ipp-route-4c9298-kserve-7789dbbfcc-tcltl"] Nov 02 17:52:11.072240 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.072214 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q"] Nov 02 17:52:11.076188 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.076162 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3568c5-799fc4fc99-bzq6q"] Nov 02 17:52:11.125477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.125446 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf"] Nov 02 17:52:11.131029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.131004 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3568c5-5468488957-njxbf"] Nov 02 17:52:11.620979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.616158 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:52:11.651973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.651433 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:11.670971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.669757 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-podinfo\") pod \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " Nov 02 17:52:11.670971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.669848 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-credential-socket\") pod \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " Nov 02 17:52:11.670971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.669907 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-workload-certs\") pod \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " Nov 02 17:52:11.670971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.670050 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-workload-socket\") pod \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " Nov 02 17:52:11.670971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.670250 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-envoy\") pod \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " Nov 02 17:52:11.670971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.670280 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istiod-ca-cert\") pod \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " Nov 02 17:52:11.670971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.670331 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d911a8-7c46-4ea5-b75e-919fad9e1d22-kube-api-access-xj4pv\" (UniqueName: \"kubernetes.io/projected/78d911a8-7c46-4ea5-b75e-919fad9e1d22-kube-api-access-xj4pv\") pod \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " Nov 02 17:52:11.670971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.670405 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-data\") pod \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " Nov 02 17:52:11.670971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.670674 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-token\" (UniqueName: \"kubernetes.io/projected/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-token\") pod \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\" (UID: \"78d911a8-7c46-4ea5-b75e-919fad9e1d22\") " Nov 02 17:52:11.700007 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.693092 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-workload-certs" pod "78d911a8-7c46-4ea5-b75e-919fad9e1d22" (UID: "78d911a8-7c46-4ea5-b75e-919fad9e1d22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:11.700007 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.699220 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istiod-ca-cert" pod "78d911a8-7c46-4ea5-b75e-919fad9e1d22" (UID: "78d911a8-7c46-4ea5-b75e-919fad9e1d22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:11.700007 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.699539 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-workload-socket" pod "78d911a8-7c46-4ea5-b75e-919fad9e1d22" (UID: "78d911a8-7c46-4ea5-b75e-919fad9e1d22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:11.718003 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.717766 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-credential-socket" pod "78d911a8-7c46-4ea5-b75e-919fad9e1d22" (UID: "78d911a8-7c46-4ea5-b75e-919fad9e1d22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:11.720449 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.720268 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-podinfo" pod "78d911a8-7c46-4ea5-b75e-919fad9e1d22" (UID: "78d911a8-7c46-4ea5-b75e-919fad9e1d22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:52:11.720840 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.720649 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d911a8-7c46-4ea5-b75e-919fad9e1d22-kube-api-access-xj4pv" pod "78d911a8-7c46-4ea5-b75e-919fad9e1d22" (UID: "78d911a8-7c46-4ea5-b75e-919fad9e1d22"). InnerVolumeSpecName "kube-api-access-xj4pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:11.720840 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.720823 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-data" pod "78d911a8-7c46-4ea5-b75e-919fad9e1d22" (UID: "78d911a8-7c46-4ea5-b75e-919fad9e1d22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:11.721040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.720986 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.721040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.720999 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.721040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.721009 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.721040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.721019 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.721040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.721027 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.721040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.721036 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj4pv\" (UniqueName: \"kubernetes.io/projected/78d911a8-7c46-4ea5-b75e-919fad9e1d22-kube-api-access-xj4pv\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.721354 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.721045 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.722679 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.722644 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-token" pod "78d911a8-7c46-4ea5-b75e-919fad9e1d22" (UID: "78d911a8-7c46-4ea5-b75e-919fad9e1d22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:11.723114 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.723085 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-envoy" pod "78d911a8-7c46-4ea5-b75e-919fad9e1d22" (UID: "78d911a8-7c46-4ea5-b75e-919fad9e1d22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:11.725256 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.725234 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:52:11.822468 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.822372 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-workload-certs\") pod \"57692113-df46-4797-9fd6-49b5aad6f5f0\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " Nov 02 17:52:11.822468 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.822417 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57692113-df46-4797-9fd6-49b5aad6f5f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57692113-df46-4797-9fd6-49b5aad6f5f0-istiod-ca-cert\") pod \"57692113-df46-4797-9fd6-49b5aad6f5f0\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " Nov 02 17:52:11.822701 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.822477 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-podinfo\") pod \"57692113-df46-4797-9fd6-49b5aad6f5f0\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " Nov 02 17:52:11.822701 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.822574 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-data\") pod \"57692113-df46-4797-9fd6-49b5aad6f5f0\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " Nov 02 17:52:11.822701 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.822632 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-workload-socket\") pod \"57692113-df46-4797-9fd6-49b5aad6f5f0\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " Nov 02 17:52:11.822701 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.822652 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-workload-certs" pod "57692113-df46-4797-9fd6-49b5aad6f5f0" (UID: "57692113-df46-4797-9fd6-49b5aad6f5f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:11.822921 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.822717 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-credential-socket\") pod \"57692113-df46-4797-9fd6-49b5aad6f5f0\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " Nov 02 17:52:11.822921 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.822838 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57692113-df46-4797-9fd6-49b5aad6f5f0-kube-api-access-vmrd4\" (UniqueName: \"kubernetes.io/projected/57692113-df46-4797-9fd6-49b5aad6f5f0-kube-api-access-vmrd4\") pod \"57692113-df46-4797-9fd6-49b5aad6f5f0\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " Nov 02 17:52:11.822921 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.822863 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-data" pod "57692113-df46-4797-9fd6-49b5aad6f5f0" (UID: "57692113-df46-4797-9fd6-49b5aad6f5f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:11.823101 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.822907 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57692113-df46-4797-9fd6-49b5aad6f5f0-istiod-ca-cert" pod "57692113-df46-4797-9fd6-49b5aad6f5f0" (UID: "57692113-df46-4797-9fd6-49b5aad6f5f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:11.823101 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.822917 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-token\" (UniqueName: \"kubernetes.io/projected/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-token\") pod \"57692113-df46-4797-9fd6-49b5aad6f5f0\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " Nov 02 17:52:11.823206 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.823104 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-credential-socket" pod "57692113-df46-4797-9fd6-49b5aad6f5f0" (UID: "57692113-df46-4797-9fd6-49b5aad6f5f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:11.823206 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.823118 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-workload-socket" pod "57692113-df46-4797-9fd6-49b5aad6f5f0" (UID: "57692113-df46-4797-9fd6-49b5aad6f5f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:11.823206 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.823137 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-envoy\") pod \"57692113-df46-4797-9fd6-49b5aad6f5f0\" (UID: \"57692113-df46-4797-9fd6-49b5aad6f5f0\") " Nov 02 17:52:11.823909 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.823881 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.824112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.823915 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.824112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.823933 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57692113-df46-4797-9fd6-49b5aad6f5f0-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.824112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.823948 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.824112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.823978 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.824112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.823993 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d911a8-7c46-4ea5-b75e-919fad9e1d22-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.824112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.824007 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.825350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.825317 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-podinfo" pod "57692113-df46-4797-9fd6-49b5aad6f5f0" (UID: "57692113-df46-4797-9fd6-49b5aad6f5f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:52:11.825350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.825327 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57692113-df46-4797-9fd6-49b5aad6f5f0-kube-api-access-vmrd4" pod "57692113-df46-4797-9fd6-49b5aad6f5f0" (UID: "57692113-df46-4797-9fd6-49b5aad6f5f0"). InnerVolumeSpecName "kube-api-access-vmrd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:11.825620 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.825597 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-token" pod "57692113-df46-4797-9fd6-49b5aad6f5f0" (UID: "57692113-df46-4797-9fd6-49b5aad6f5f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:11.825939 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.825925 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-envoy" pod "57692113-df46-4797-9fd6-49b5aad6f5f0" (UID: "57692113-df46-4797-9fd6-49b5aad6f5f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:11.925202 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.925170 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.925202 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.925198 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmrd4\" (UniqueName: \"kubernetes.io/projected/57692113-df46-4797-9fd6-49b5aad6f5f0-kube-api-access-vmrd4\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.925202 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.925210 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:11.925433 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:11.925220 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57692113-df46-4797-9fd6-49b5aad6f5f0-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:12.022664 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.022630 5549 generic.go:356] "Generic (PLEG): container finished" podID="57692113-df46-4797-9fd6-49b5aad6f5f0" containerID="e859cf0e94e063033c940f8123c0d743d1479552a1d487f516994329efb799bb" exitCode=0 Nov 02 17:52:12.022860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.022680 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" Nov 02 17:52:12.022860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.022712 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" event={"ID":"57692113-df46-4797-9fd6-49b5aad6f5f0","Type":"ContainerDied","Data":"e859cf0e94e063033c940f8123c0d743d1479552a1d487f516994329efb799bb"} Nov 02 17:52:12.022860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.022763 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh" event={"ID":"57692113-df46-4797-9fd6-49b5aad6f5f0","Type":"ContainerDied","Data":"d9aced77b8cdbb9d8b65c7d5508335cdee451195abf1e67cf34de20058683d22"} Nov 02 17:52:12.022860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.022786 5549 scope.go:122] "RemoveContainer" containerID="e859cf0e94e063033c940f8123c0d743d1479552a1d487f516994329efb799bb" Nov 02 17:52:12.024167 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.024147 5549 generic.go:356] "Generic (PLEG): container finished" podID="78d911a8-7c46-4ea5-b75e-919fad9e1d22" containerID="0518a8bc7b9a48726ea2108a3cce5263574462fd176a311f28ce8abec003c792" exitCode=0 Nov 02 17:52:12.024265 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.024228 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" event={"ID":"78d911a8-7c46-4ea5-b75e-919fad9e1d22","Type":"ContainerDied","Data":"0518a8bc7b9a48726ea2108a3cce5263574462fd176a311f28ce8abec003c792"} Nov 02 17:52:12.024265 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.024242 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" Nov 02 17:52:12.024374 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.024273 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg" event={"ID":"78d911a8-7c46-4ea5-b75e-919fad9e1d22","Type":"ContainerDied","Data":"2260df6246d329a7d078ff2f1e3655a1e2c7446376a545aef510f35536001bca"} Nov 02 17:52:12.033903 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.033888 5549 scope.go:122] "RemoveContainer" containerID="e859cf0e94e063033c940f8123c0d743d1479552a1d487f516994329efb799bb" Nov 02 17:52:12.034233 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:52:12.034211 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e859cf0e94e063033c940f8123c0d743d1479552a1d487f516994329efb799bb\": container with ID starting with e859cf0e94e063033c940f8123c0d743d1479552a1d487f516994329efb799bb not found: ID does not exist" containerID="e859cf0e94e063033c940f8123c0d743d1479552a1d487f516994329efb799bb" Nov 02 17:52:12.034298 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.034246 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e859cf0e94e063033c940f8123c0d743d1479552a1d487f516994329efb799bb"} err="failed to get container status \"e859cf0e94e063033c940f8123c0d743d1479552a1d487f516994329efb799bb\": rpc error: code = NotFound desc = could not find container \"e859cf0e94e063033c940f8123c0d743d1479552a1d487f516994329efb799bb\": container with ID starting with e859cf0e94e063033c940f8123c0d743d1479552a1d487f516994329efb799bb not found: ID does not exist" Nov 02 17:52:12.034298 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.034272 5549 scope.go:122] "RemoveContainer" containerID="0518a8bc7b9a48726ea2108a3cce5263574462fd176a311f28ce8abec003c792" Nov 02 17:52:12.043628 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.043613 5549 scope.go:122] "RemoveContainer" containerID="0518a8bc7b9a48726ea2108a3cce5263574462fd176a311f28ce8abec003c792" Nov 02 17:52:12.043893 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:52:12.043872 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0518a8bc7b9a48726ea2108a3cce5263574462fd176a311f28ce8abec003c792\": container with ID starting with 0518a8bc7b9a48726ea2108a3cce5263574462fd176a311f28ce8abec003c792 not found: ID does not exist" containerID="0518a8bc7b9a48726ea2108a3cce5263574462fd176a311f28ce8abec003c792" Nov 02 17:52:12.044043 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.043895 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0518a8bc7b9a48726ea2108a3cce5263574462fd176a311f28ce8abec003c792"} err="failed to get container status \"0518a8bc7b9a48726ea2108a3cce5263574462fd176a311f28ce8abec003c792\": rpc error: code = NotFound desc = could not find container \"0518a8bc7b9a48726ea2108a3cce5263574462fd176a311f28ce8abec003c792\": container with ID starting with 0518a8bc7b9a48726ea2108a3cce5263574462fd176a311f28ce8abec003c792 not found: ID does not exist" Nov 02 17:52:12.049308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.049286 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh"] Nov 02 17:52:12.056048 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.056015 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-x2lvh"] Nov 02 17:52:12.068392 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.068371 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg"] Nov 02 17:52:12.074895 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.074827 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-szstg"] Nov 02 17:52:12.322738 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.322699 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5186c8ad-6d18-4e3e-b71c-62ec5af8222a" path="/var/lib/kubelet/pods/5186c8ad-6d18-4e3e-b71c-62ec5af8222a/volumes" Nov 02 17:52:12.323220 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.323205 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="536ab9a4-1941-4ca5-8a50-691e53ee1148" path="/var/lib/kubelet/pods/536ab9a4-1941-4ca5-8a50-691e53ee1148/volumes" Nov 02 17:52:12.323671 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.323658 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57692113-df46-4797-9fd6-49b5aad6f5f0" path="/var/lib/kubelet/pods/57692113-df46-4797-9fd6-49b5aad6f5f0/volumes" Nov 02 17:52:12.324424 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.324409 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d911a8-7c46-4ea5-b75e-919fad9e1d22" path="/var/lib/kubelet/pods/78d911a8-7c46-4ea5-b75e-919fad9e1d22/volumes" Nov 02 17:52:12.324799 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.324786 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df101814-f22d-4abd-888c-b3288a7707a1" path="/var/lib/kubelet/pods/df101814-f22d-4abd-888c-b3288a7707a1/volumes" Nov 02 17:52:12.558265 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:12.556636 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:13.413300 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:13.413265 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:14.190976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.190383 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:14.357278 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.357241 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:14.816006 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.815964 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j"] Nov 02 17:52:14.817110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.817087 5549 cpu_manager.go:414] "containerMap: removing container" podUID="78d911a8-7c46-4ea5-b75e-919fad9e1d22" containerName="istio-proxy" Nov 02 17:52:14.817647 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.817625 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d911a8-7c46-4ea5-b75e-919fad9e1d22" containerName="istio-proxy" Nov 02 17:52:14.817889 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.817864 5549 cpu_manager.go:414] "containerMap: removing container" podUID="57692113-df46-4797-9fd6-49b5aad6f5f0" containerName="istio-proxy" Nov 02 17:52:14.817889 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.817886 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57692113-df46-4797-9fd6-49b5aad6f5f0" containerName="istio-proxy" Nov 02 17:52:14.818089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.817900 5549 cpu_manager.go:414] "containerMap: removing container" podUID="df101814-f22d-4abd-888c-b3288a7707a1" containerName="payload-processing" Nov 02 17:52:14.818089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.817908 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df101814-f22d-4abd-888c-b3288a7707a1" containerName="payload-processing" Nov 02 17:52:14.818089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.817971 5549 cpu_manager.go:414] "containerMap: removing container" podUID="5186c8ad-6d18-4e3e-b71c-62ec5af8222a" containerName="main" Nov 02 17:52:14.818089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.817980 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5186c8ad-6d18-4e3e-b71c-62ec5af8222a" containerName="main" Nov 02 17:52:14.818089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.817991 5549 cpu_manager.go:414] "containerMap: removing container" podUID="536ab9a4-1941-4ca5-8a50-691e53ee1148" containerName="payload-pre-processing" Nov 02 17:52:14.818089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.817997 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536ab9a4-1941-4ca5-8a50-691e53ee1148" containerName="payload-pre-processing" Nov 02 17:52:14.818422 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.818149 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="57692113-df46-4797-9fd6-49b5aad6f5f0" containerName="istio-proxy" Nov 02 17:52:14.818422 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.818161 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="df101814-f22d-4abd-888c-b3288a7707a1" containerName="payload-processing" Nov 02 17:52:14.818422 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.818179 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5186c8ad-6d18-4e3e-b71c-62ec5af8222a" containerName="main" Nov 02 17:52:14.818422 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.818189 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="536ab9a4-1941-4ca5-8a50-691e53ee1148" containerName="payload-pre-processing" Nov 02 17:52:14.818422 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.818197 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d911a8-7c46-4ea5-b75e-919fad9e1d22" containerName="istio-proxy" Nov 02 17:52:14.829893 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.829865 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j"] Nov 02 17:52:14.830056 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.830020 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" Nov 02 17:52:14.835205 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.835182 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-4a5f09-serving-cert\"" Nov 02 17:52:14.927648 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:14.927598 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:15.072288 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:15.072176 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcklb\" (UniqueName: \"kubernetes.io/projected/fad6f82c-b39c-4b16-92b2-64964826ad15-kube-api-access-qcklb\") pod \"maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j\" (UID: \"fad6f82c-b39c-4b16-92b2-64964826ad15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" Nov 02 17:52:15.072840 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:15.072795 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad6f82c-b39c-4b16-92b2-64964826ad15-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j\" (UID: \"fad6f82c-b39c-4b16-92b2-64964826ad15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" Nov 02 17:52:15.073088 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:15.073047 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad6f82c-b39c-4b16-92b2-64964826ad15-metrics-certs\") pod \"maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j\" (UID: \"fad6f82c-b39c-4b16-92b2-64964826ad15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" Nov 02 17:52:15.073426 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:15.073237 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fad6f82c-b39c-4b16-92b2-64964826ad15-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j\" (UID: \"fad6f82c-b39c-4b16-92b2-64964826ad15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" Nov 02 17:52:15.116079 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:15.100911 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:15.176129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:15.176095 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fad6f82c-b39c-4b16-92b2-64964826ad15-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j\" (UID: \"fad6f82c-b39c-4b16-92b2-64964826ad15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" Nov 02 17:52:15.176353 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:15.176333 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcklb\" (UniqueName: \"kubernetes.io/projected/fad6f82c-b39c-4b16-92b2-64964826ad15-kube-api-access-qcklb\") pod \"maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j\" (UID: \"fad6f82c-b39c-4b16-92b2-64964826ad15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" Nov 02 17:52:15.176456 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:15.176441 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad6f82c-b39c-4b16-92b2-64964826ad15-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j\" (UID: \"fad6f82c-b39c-4b16-92b2-64964826ad15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" Nov 02 17:52:15.176513 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:15.176471 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad6f82c-b39c-4b16-92b2-64964826ad15-metrics-certs\") pod \"maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j\" (UID: \"fad6f82c-b39c-4b16-92b2-64964826ad15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" Nov 02 17:52:15.177174 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:15.177148 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad6f82c-b39c-4b16-92b2-64964826ad15-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j\" (UID: \"fad6f82c-b39c-4b16-92b2-64964826ad15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" Nov 02 17:52:15.178741 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:15.178721 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fad6f82c-b39c-4b16-92b2-64964826ad15-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j\" (UID: \"fad6f82c-b39c-4b16-92b2-64964826ad15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" Nov 02 17:52:15.179000 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:15.178940 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad6f82c-b39c-4b16-92b2-64964826ad15-metrics-certs\") pod \"maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j\" (UID: \"fad6f82c-b39c-4b16-92b2-64964826ad15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" Nov 02 17:52:15.189147 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:15.189115 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcklb\" (UniqueName: \"kubernetes.io/projected/fad6f82c-b39c-4b16-92b2-64964826ad15-kube-api-access-qcklb\") pod \"maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j\" (UID: \"fad6f82c-b39c-4b16-92b2-64964826ad15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" Nov 02 17:52:15.464031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:15.451242 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" Nov 02 17:52:15.841884 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:15.826845 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:15.887906 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:15.887869 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j"] Nov 02 17:52:15.895640 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:52:15.894326 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad6f82c_b39c_4b16_92b2_64964826ad15.slice/crio-0f1bfe6deb4d9a31f71a43f147a430db356d49b595e1ada690c109ef0a159184 WatchSource:0}: Error finding container 0f1bfe6deb4d9a31f71a43f147a430db356d49b595e1ada690c109ef0a159184: Status 404 returned error can't find the container with id 0f1bfe6deb4d9a31f71a43f147a430db356d49b595e1ada690c109ef0a159184 Nov 02 17:52:16.113400 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.113323 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" event={"ID":"fad6f82c-b39c-4b16-92b2-64964826ad15","Type":"ContainerStarted","Data":"0f1bfe6deb4d9a31f71a43f147a430db356d49b595e1ada690c109ef0a159184"} Nov 02 17:52:16.251883 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.251837 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:16.359923 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.359538 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz"] Nov 02 17:52:16.359923 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.359759 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" podUID="c916c795-ef55-4664-91ff-392939fa292f" containerName="maas-api" containerID="cri-o://79a4ff2a10177148a34129e0bee32aeaebcd1e778a077978aaaa680e875d2f55" gracePeriod=30 Nov 02 17:52:16.463449 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.463415 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:16.753456 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.753073 5549 patch_prober.go:29] interesting pod/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:52:16.753456 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.753138 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" podUID="98455203-d4a6-4bc7-af34-b901edb31b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:16.753456 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.753233 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" Nov 02 17:52:16.807757 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.807721 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c916c795-ef55-4664-91ff-392939fa292f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c916c795-ef55-4664-91ff-392939fa292f-metrics-certs\") pod \"c916c795-ef55-4664-91ff-392939fa292f\" (UID: \"c916c795-ef55-4664-91ff-392939fa292f\") " Nov 02 17:52:16.809224 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.808011 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c916c795-ef55-4664-91ff-392939fa292f-kube-api-access-7lt5r\" (UniqueName: \"kubernetes.io/projected/c916c795-ef55-4664-91ff-392939fa292f-kube-api-access-7lt5r\") pod \"c916c795-ef55-4664-91ff-392939fa292f\" (UID: \"c916c795-ef55-4664-91ff-392939fa292f\") " Nov 02 17:52:16.809224 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.808112 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c916c795-ef55-4664-91ff-392939fa292f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c916c795-ef55-4664-91ff-392939fa292f-maas-api-tls\") pod \"c916c795-ef55-4664-91ff-392939fa292f\" (UID: \"c916c795-ef55-4664-91ff-392939fa292f\") " Nov 02 17:52:16.809224 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.808243 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c916c795-ef55-4664-91ff-392939fa292f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c916c795-ef55-4664-91ff-392939fa292f-openshift-service-ca\") pod \"c916c795-ef55-4664-91ff-392939fa292f\" (UID: \"c916c795-ef55-4664-91ff-392939fa292f\") " Nov 02 17:52:16.809628 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.809604 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c916c795-ef55-4664-91ff-392939fa292f-openshift-service-ca" pod "c916c795-ef55-4664-91ff-392939fa292f" (UID: "c916c795-ef55-4664-91ff-392939fa292f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:16.820246 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.818529 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5" Nov 02 17:52:16.839470 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.839378 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:16.839920 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.839866 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c916c795-ef55-4664-91ff-392939fa292f-maas-api-tls" pod "c916c795-ef55-4664-91ff-392939fa292f" (UID: "c916c795-ef55-4664-91ff-392939fa292f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:52:16.846273 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.846230 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c916c795-ef55-4664-91ff-392939fa292f-metrics-certs" pod "c916c795-ef55-4664-91ff-392939fa292f" (UID: "c916c795-ef55-4664-91ff-392939fa292f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:52:16.857980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.856573 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c916c795-ef55-4664-91ff-392939fa292f-kube-api-access-7lt5r" pod "c916c795-ef55-4664-91ff-392939fa292f" (UID: "c916c795-ef55-4664-91ff-392939fa292f"). InnerVolumeSpecName "kube-api-access-7lt5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:16.924175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.923373 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c916c795-ef55-4664-91ff-392939fa292f-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:16.924175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.923414 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lt5r\" (UniqueName: \"kubernetes.io/projected/c916c795-ef55-4664-91ff-392939fa292f-kube-api-access-7lt5r\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:16.924175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.923430 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c916c795-ef55-4664-91ff-392939fa292f-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:16.924175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:16.923445 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c916c795-ef55-4664-91ff-392939fa292f-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:17.063118 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:17.063085 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" Nov 02 17:52:17.128377 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:17.127141 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" event={"ID":"fad6f82c-b39c-4b16-92b2-64964826ad15","Type":"ContainerStarted","Data":"032d08caefd2ddac1885c6e1cdd31c00f0b405e4a6ac48427e150949e9e0a594"} Nov 02 17:52:17.128377 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:17.128310 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" Nov 02 17:52:17.132000 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:17.130454 5549 generic.go:356] "Generic (PLEG): container finished" podID="c916c795-ef55-4664-91ff-392939fa292f" containerID="79a4ff2a10177148a34129e0bee32aeaebcd1e778a077978aaaa680e875d2f55" exitCode=0 Nov 02 17:52:17.132000 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:17.130565 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" event={"ID":"c916c795-ef55-4664-91ff-392939fa292f","Type":"ContainerDied","Data":"79a4ff2a10177148a34129e0bee32aeaebcd1e778a077978aaaa680e875d2f55"} Nov 02 17:52:17.132000 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:17.130607 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" event={"ID":"c916c795-ef55-4664-91ff-392939fa292f","Type":"ContainerDied","Data":"047f930899e72b4cee3c6b0ff3722dc4ea6982ff8ce85cfcc7fd37519f0cf855"} Nov 02 17:52:17.132000 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:17.130638 5549 scope.go:122] "RemoveContainer" containerID="79a4ff2a10177148a34129e0bee32aeaebcd1e778a077978aaaa680e875d2f55" Nov 02 17:52:17.132000 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:17.130775 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz" Nov 02 17:52:17.155677 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:17.155653 5549 scope.go:122] "RemoveContainer" containerID="79a4ff2a10177148a34129e0bee32aeaebcd1e778a077978aaaa680e875d2f55" Nov 02 17:52:17.156048 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:52:17.156021 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a4ff2a10177148a34129e0bee32aeaebcd1e778a077978aaaa680e875d2f55\": container with ID starting with 79a4ff2a10177148a34129e0bee32aeaebcd1e778a077978aaaa680e875d2f55 not found: ID does not exist" containerID="79a4ff2a10177148a34129e0bee32aeaebcd1e778a077978aaaa680e875d2f55" Nov 02 17:52:17.156148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:17.156056 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a4ff2a10177148a34129e0bee32aeaebcd1e778a077978aaaa680e875d2f55"} err="failed to get container status \"79a4ff2a10177148a34129e0bee32aeaebcd1e778a077978aaaa680e875d2f55\": rpc error: code = NotFound desc = could not find container \"79a4ff2a10177148a34129e0bee32aeaebcd1e778a077978aaaa680e875d2f55\": container with ID starting with 79a4ff2a10177148a34129e0bee32aeaebcd1e778a077978aaaa680e875d2f55 not found: ID does not exist" Nov 02 17:52:17.161764 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:17.161698 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" podStartSLOduration=2.743767531 podStartE2EDuration="3.161680298s" totalImagesPullingTime="417.912767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:17.15850295 +0000 UTC m=+1108.947008478" watchObservedRunningTime="2026-11-02 17:52:17.161680298 +0000 UTC m=+1108.950185826" Nov 02 17:52:17.178005 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:17.177926 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz"] Nov 02 17:52:17.180970 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:17.180925 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3568c5-7b4694fff4-j5mfz"] Nov 02 17:52:17.588070 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:52:17.588035 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:52:17.658807 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:17.658656 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:17.822401 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:17.822372 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx" Nov 02 17:52:17.835809 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:17.835784 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq" Nov 02 17:52:18.452529 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:18.451813 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c916c795-ef55-4664-91ff-392939fa292f" path="/var/lib/kubelet/pods/c916c795-ef55-4664-91ff-392939fa292f/volumes" Nov 02 17:52:18.682980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:18.675443 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:19.598863 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:19.598610 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:20.155220 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:20.155188 5549 generic.go:356] "Generic (PLEG): container finished" podID="98455203-d4a6-4bc7-af34-b901edb31b83" containerID="1ca8cb082abbaf44cedadfaad543f43d18a780ac912e6b2612370f2bf3dfd38a" exitCode=0 Nov 02 17:52:20.155412 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:20.155264 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" event={"ID":"98455203-d4a6-4bc7-af34-b901edb31b83","Type":"ContainerDied","Data":"1ca8cb082abbaf44cedadfaad543f43d18a780ac912e6b2612370f2bf3dfd38a"} Nov 02 17:52:20.155738 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:20.155709 5549 scope.go:122] "RemoveContainer" containerID="1ca8cb082abbaf44cedadfaad543f43d18a780ac912e6b2612370f2bf3dfd38a" Nov 02 17:52:20.157305 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:20.157272 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:52:20.157434 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:20.157373 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:52:20.157500 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:20.157429 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:52:20.519024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:20.517389 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:21.192016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.187942 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" event={"ID":"98455203-d4a6-4bc7-af34-b901edb31b83","Type":"ContainerStarted","Data":"84073160c33beb89c4c9e242994bd75e9ad5b332294db11d9ffe5313f0474c28"} Nov 02 17:52:21.250979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.249809 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:21.413311 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.413276 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:21.611199 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.611161 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd"] Nov 02 17:52:21.611874 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.611860 5549 cpu_manager.go:414] "containerMap: removing container" podUID="c916c795-ef55-4664-91ff-392939fa292f" containerName="maas-api" Nov 02 17:52:21.611919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.611876 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c916c795-ef55-4664-91ff-392939fa292f" containerName="maas-api" Nov 02 17:52:21.612044 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.612024 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c916c795-ef55-4664-91ff-392939fa292f" containerName="maas-api" Nov 02 17:52:21.614792 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.614770 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.631123 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.631099 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-eda8e2-openshift-default-dockercfg-2k984\"" Nov 02 17:52:21.631598 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.631574 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd"] Nov 02 17:52:21.762280 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.754310 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:21.762280 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.755253 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-credential-socket\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.762280 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.755297 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-workload-socket\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.762280 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.755337 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-workload-certs\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.762280 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.755483 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-token\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.762280 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.755535 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-podinfo\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.762280 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.755635 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-data\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.762280 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.755674 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f78f64-e109-490b-a5ae-4fcec9644276-istiod-ca-cert\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.762280 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.755707 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-envoy\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.762280 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.755731 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx85z\" (UniqueName: \"kubernetes.io/projected/c9f78f64-e109-490b-a5ae-4fcec9644276-kube-api-access-vx85z\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.779093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.778069 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:52:21.779093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.778618 5549 patch_prober.go:29] interesting pod/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 17:52:21.779093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.778670 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" podUID="98455203-d4a6-4bc7-af34-b901edb31b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 17:52:21.863026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.862882 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-data\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.863026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.862966 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f78f64-e109-490b-a5ae-4fcec9644276-istiod-ca-cert\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.863026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.863010 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-envoy\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.863326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.863036 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx85z\" (UniqueName: \"kubernetes.io/projected/c9f78f64-e109-490b-a5ae-4fcec9644276-kube-api-access-vx85z\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.863326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.863096 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-credential-socket\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.863326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.863127 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-workload-socket\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.863326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.863161 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-workload-certs\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.863531 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.863344 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-token\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.863531 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.863412 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-podinfo\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.865229 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.864182 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-data\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.865229 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.864883 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f78f64-e109-490b-a5ae-4fcec9644276-istiod-ca-cert\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.865763 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.865736 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-workload-socket\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.881603 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.866149 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-workload-certs\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.881789 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.866660 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-credential-socket\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.913946 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.913870 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx85z\" (UniqueName: \"kubernetes.io/projected/c9f78f64-e109-490b-a5ae-4fcec9644276-kube-api-access-vx85z\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.927486 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.927379 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-envoy\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.928245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.928219 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-podinfo\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.929157 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.929134 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-token\") pod \"e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:21.950555 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:21.950269 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:22.265965 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:22.265920 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:22.399987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:22.397771 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:52:22.429015 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:22.397940 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:52:22.432988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:22.429353 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:52:22.452978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:22.447243 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd"] Nov 02 17:52:22.622027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:22.619099 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:22.764345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:22.764313 5549 patch_prober.go:29] interesting pod/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 17:52:22.764487 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:22.764369 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" podUID="98455203-d4a6-4bc7-af34-b901edb31b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 17:52:23.275511 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:23.275467 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" event={"ID":"c9f78f64-e109-490b-a5ae-4fcec9644276","Type":"ContainerStarted","Data":"84931c92ab04e142b5e666ece6dac8e8ff0cfb0155929d5c4f286fdcf0cda1cb"} Nov 02 17:52:23.275930 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:23.275539 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" event={"ID":"c9f78f64-e109-490b-a5ae-4fcec9644276","Type":"ContainerStarted","Data":"edd8c45727174994ee67d35c8ecc34be4ef0bd2ab36165594d3bfbb963aee10a"} Nov 02 17:52:23.346990 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:23.346905 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" podStartSLOduration=2.346871569 podStartE2EDuration="2.346871569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:23.325545231 +0000 UTC m=+1115.114050761" watchObservedRunningTime="2026-11-02 17:52:23.346871569 +0000 UTC m=+1115.135377098" Nov 02 17:52:23.523530 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:23.523496 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:23.765367 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:23.765328 5549 patch_prober.go:29] interesting pod/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 17:52:23.765786 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:23.765751 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" podUID="98455203-d4a6-4bc7-af34-b901edb31b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 17:52:23.961288 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:23.961249 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:23.990108 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:23.989809 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:24.152559 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:24.152480 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" Nov 02 17:52:24.326371 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:24.326344 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:24.326795 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:24.326408 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:52:24.606991 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:24.606935 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:24.764864 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:24.764822 5549 patch_prober.go:29] interesting pod/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 17:52:24.765074 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:24.764889 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" podUID="98455203-d4a6-4bc7-af34-b901edb31b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 17:52:25.165513 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:25.165477 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:25.342993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:25.342932 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:25.557047 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:25.557014 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:25.763968 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:25.763926 5549 patch_prober.go:29] interesting pod/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 17:52:25.764166 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:25.764008 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" podUID="98455203-d4a6-4bc7-af34-b901edb31b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 17:52:25.764223 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:25.764202 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:25.936662 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:25.936576 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:26.137480 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:26.137443 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:26.883755 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:26.880456 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:52:26.900349 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:26.899699 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:52:26.913265 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:26.911310 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:52:26.928445 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:26.928391 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:27.239872 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.239835 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb"] Nov 02 17:52:27.243455 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.243436 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb" Nov 02 17:52:27.246461 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.246433 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:27.247586 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.247558 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eda8e2\"" Nov 02 17:52:27.247697 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.247626 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eda8e2-dockercfg-xk8fj\"" Nov 02 17:52:27.259498 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.259469 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb"] Nov 02 17:52:27.281595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.281557 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69"] Nov 02 17:52:27.286120 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.286098 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69" Nov 02 17:52:27.297099 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.297073 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69"] Nov 02 17:52:27.368974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.361878 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf97k\" (UniqueName: \"kubernetes.io/projected/769444b9-fcd1-46d4-8753-27a9a8a7d3f2-kube-api-access-cf97k\") pod \"payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb\" (UID: \"769444b9-fcd1-46d4-8753-27a9a8a7d3f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb" Nov 02 17:52:27.368974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.362079 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/769444b9-fcd1-46d4-8753-27a9a8a7d3f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb\" (UID: \"769444b9-fcd1-46d4-8753-27a9a8a7d3f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb" Nov 02 17:52:27.497031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.464332 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf97k\" (UniqueName: \"kubernetes.io/projected/769444b9-fcd1-46d4-8753-27a9a8a7d3f2-kube-api-access-cf97k\") pod \"payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb\" (UID: \"769444b9-fcd1-46d4-8753-27a9a8a7d3f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb" Nov 02 17:52:27.497381 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.497349 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007b76f2-5968-404f-a034-09bc84e9db73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69\" (UID: \"007b76f2-5968-404f-a034-09bc84e9db73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69" Nov 02 17:52:27.520973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.504129 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/769444b9-fcd1-46d4-8753-27a9a8a7d3f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb\" (UID: \"769444b9-fcd1-46d4-8753-27a9a8a7d3f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb" Nov 02 17:52:27.520973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.504255 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwggg\" (UniqueName: \"kubernetes.io/projected/007b76f2-5968-404f-a034-09bc84e9db73-kube-api-access-mwggg\") pod \"payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69\" (UID: \"007b76f2-5968-404f-a034-09bc84e9db73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69" Nov 02 17:52:27.520973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.505274 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/769444b9-fcd1-46d4-8753-27a9a8a7d3f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb\" (UID: \"769444b9-fcd1-46d4-8753-27a9a8a7d3f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb" Nov 02 17:52:27.568513 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.547458 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:27.610025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.608248 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwggg\" (UniqueName: \"kubernetes.io/projected/007b76f2-5968-404f-a034-09bc84e9db73-kube-api-access-mwggg\") pod \"payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69\" (UID: \"007b76f2-5968-404f-a034-09bc84e9db73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69" Nov 02 17:52:27.610025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.608520 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007b76f2-5968-404f-a034-09bc84e9db73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69\" (UID: \"007b76f2-5968-404f-a034-09bc84e9db73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69" Nov 02 17:52:27.610025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.609546 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007b76f2-5968-404f-a034-09bc84e9db73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69\" (UID: \"007b76f2-5968-404f-a034-09bc84e9db73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69" Nov 02 17:52:27.611152 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.611117 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf97k\" (UniqueName: \"kubernetes.io/projected/769444b9-fcd1-46d4-8753-27a9a8a7d3f2-kube-api-access-cf97k\") pod \"payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb\" (UID: \"769444b9-fcd1-46d4-8753-27a9a8a7d3f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb" Nov 02 17:52:27.618698 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.618672 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwggg\" (UniqueName: \"kubernetes.io/projected/007b76f2-5968-404f-a034-09bc84e9db73-kube-api-access-mwggg\") pod \"payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69\" (UID: \"007b76f2-5968-404f-a034-09bc84e9db73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69" Nov 02 17:52:27.735851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.735817 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:27.857805 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.857722 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb" Nov 02 17:52:27.905208 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.905174 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69" Nov 02 17:52:27.971252 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:27.971206 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:28.212982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:28.206454 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:28.304322 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:28.304291 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb"] Nov 02 17:52:28.437786 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:28.437757 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69"] Nov 02 17:52:28.438993 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:52:28.438969 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007b76f2_5968_404f_a034_09bc84e9db73.slice/crio-80cc23f396436c70175602ffc8aaeba727ac8079492cd1bc496c71e8a248b72f WatchSource:0}: Error finding container 80cc23f396436c70175602ffc8aaeba727ac8079492cd1bc496c71e8a248b72f: Status 404 returned error can't find the container with id 80cc23f396436c70175602ffc8aaeba727ac8079492cd1bc496c71e8a248b72f Nov 02 17:52:28.616085 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:28.615922 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:28.638748 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:28.638702 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69" event={"ID":"007b76f2-5968-404f-a034-09bc84e9db73","Type":"ContainerStarted","Data":"a669dc534e1866c9a43e42bbaca3d8b956128fbaceed0fbe3bc97b3045b2f2d7"} Nov 02 17:52:28.638930 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:28.638766 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69" event={"ID":"007b76f2-5968-404f-a034-09bc84e9db73","Type":"ContainerStarted","Data":"80cc23f396436c70175602ffc8aaeba727ac8079492cd1bc496c71e8a248b72f"} Nov 02 17:52:28.638930 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:28.638797 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69" Nov 02 17:52:28.640777 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:28.640751 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb" event={"ID":"769444b9-fcd1-46d4-8753-27a9a8a7d3f2","Type":"ContainerStarted","Data":"efb049584f721aa17c804dbf2e33487f5cf3723f781b9bfb6a741d178162b89b"} Nov 02 17:52:28.640910 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:28.640804 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb" event={"ID":"769444b9-fcd1-46d4-8753-27a9a8a7d3f2","Type":"ContainerStarted","Data":"d001ab103ae59e5c1a7606fd987dc2c2615dfcae1f52f2b785a56ded6cdc474b"} Nov 02 17:52:28.641029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:28.640981 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb" Nov 02 17:52:28.660293 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:28.660237 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69" podStartSLOduration=1.660221458 podStartE2EDuration="1.660221458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:28.658433072 +0000 UTC m=+1120.446938601" watchObservedRunningTime="2026-11-02 17:52:28.660221458 +0000 UTC m=+1120.448726986" Nov 02 17:52:28.680335 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:28.680277 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb" podStartSLOduration=1.680259914 podStartE2EDuration="1.680259914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:28.67740265 +0000 UTC m=+1120.465908172" watchObservedRunningTime="2026-11-02 17:52:28.680259914 +0000 UTC m=+1120.468765443" Nov 02 17:52:28.999791 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:28.999760 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:29.769976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:29.769919 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:30.012586 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:30.012550 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:30.679031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:30.678972 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:31.685864 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:31.685827 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:32.616071 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:32.616040 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:32.770499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:32.770453 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:33.117027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:33.114555 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:33.893159 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:33.893130 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:33.948632 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:33.948586 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c"] Nov 02 17:52:33.954143 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:33.954117 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" Nov 02 17:52:33.958537 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:33.958510 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c"] Nov 02 17:52:34.016104 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:34.016072 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d8e217\"" Nov 02 17:52:34.016938 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:34.016910 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d8e217-dockercfg-ctkkn\"" Nov 02 17:52:34.148031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:34.123962 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:34.148031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:34.124853 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbrwp\" (UniqueName: \"kubernetes.io/projected/c3689510-357c-4d9c-93d5-8971c2bddea9-kube-api-access-hbrwp\") pod \"payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c\" (UID: \"c3689510-357c-4d9c-93d5-8971c2bddea9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" Nov 02 17:52:34.148031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:34.137349 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3689510-357c-4d9c-93d5-8971c2bddea9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c\" (UID: \"c3689510-357c-4d9c-93d5-8971c2bddea9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" Nov 02 17:52:34.263990 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:34.261233 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3689510-357c-4d9c-93d5-8971c2bddea9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c\" (UID: \"c3689510-357c-4d9c-93d5-8971c2bddea9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" Nov 02 17:52:34.263990 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:34.261513 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbrwp\" (UniqueName: \"kubernetes.io/projected/c3689510-357c-4d9c-93d5-8971c2bddea9-kube-api-access-hbrwp\") pod \"payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c\" (UID: \"c3689510-357c-4d9c-93d5-8971c2bddea9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" Nov 02 17:52:34.267883 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:34.266263 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3689510-357c-4d9c-93d5-8971c2bddea9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c\" (UID: \"c3689510-357c-4d9c-93d5-8971c2bddea9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" Nov 02 17:52:34.302755 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:34.302716 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:34.306867 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:34.306790 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbrwp\" (UniqueName: \"kubernetes.io/projected/c3689510-357c-4d9c-93d5-8971c2bddea9-kube-api-access-hbrwp\") pod \"payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c\" (UID: \"c3689510-357c-4d9c-93d5-8971c2bddea9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" Nov 02 17:52:34.573774 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:34.573741 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" Nov 02 17:52:34.745773 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:34.745735 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:34.873105 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:34.873048 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c"] Nov 02 17:52:34.874379 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:52:34.874349 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3689510_357c_4d9c_93d5_8971c2bddea9.slice/crio-7602139d500aa7946f082972dea79ddb59150faa8a872f2e5c91afa3047391b4 WatchSource:0}: Error finding container 7602139d500aa7946f082972dea79ddb59150faa8a872f2e5c91afa3047391b4: Status 404 returned error can't find the container with id 7602139d500aa7946f082972dea79ddb59150faa8a872f2e5c91afa3047391b4 Nov 02 17:52:34.953015 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:34.952942 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:35.129634 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:35.129440 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:35.382345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:35.382239 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:35.741971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:35.741921 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" event={"ID":"c3689510-357c-4d9c-93d5-8971c2bddea9","Type":"ContainerStarted","Data":"e85125f3b9ae5e88653cb38356feece6727a994f42a9423e47a00b3bd7011151"} Nov 02 17:52:35.742277 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:35.742245 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" event={"ID":"c3689510-357c-4d9c-93d5-8971c2bddea9","Type":"ContainerStarted","Data":"7602139d500aa7946f082972dea79ddb59150faa8a872f2e5c91afa3047391b4"} Nov 02 17:52:35.742384 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:35.742290 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" Nov 02 17:52:35.763521 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:35.763474 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" podStartSLOduration=2.763459375 podStartE2EDuration="2.763459375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:35.760667992 +0000 UTC m=+1127.549173520" watchObservedRunningTime="2026-11-02 17:52:35.763459375 +0000 UTC m=+1127.551964942" Nov 02 17:52:36.089888 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:36.089801 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6"] Nov 02 17:52:36.105740 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:36.105705 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h"] Nov 02 17:52:36.214459 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:36.214425 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:36.412307 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:36.410754 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:36.582789 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:36.582747 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:37.157839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:37.157177 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:37.932099 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:37.932032 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:38.126762 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:38.126725 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:38.754042 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:38.753903 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:38.950215 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:38.950183 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:39.177695 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:39.177627 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:39.653719 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:39.653686 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb" Nov 02 17:52:39.654550 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:39.654523 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69" Nov 02 17:52:39.783070 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:39.780044 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:40.545358 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:40.545327 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:41.090733 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.090686 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6" podUID="655f5af4-c96f-4657-a926-3b8ef89a002b" containerName="payload-processing" containerID="cri-o://4aa1de65b8320b444c02e24a2f9b61a1ba9865cf104b02393ffde8fcd196c109" gracePeriod=25 Nov 02 17:52:41.106130 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.106089 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h" podUID="6c6c7ac6-b524-465f-8195-447bbc3e5695" containerName="payload-pre-processing" containerID="cri-o://9149e24ffaab27a1b2e6fa45258db33dd07ade28cf5aac949fe82027bd95aa59" gracePeriod=25 Nov 02 17:52:41.361367 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.361072 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:41.469783 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.466077 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6" Nov 02 17:52:41.511998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.508075 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/655f5af4-c96f-4657-a926-3b8ef89a002b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655f5af4-c96f-4657-a926-3b8ef89a002b-plugins-config-volume\") pod \"655f5af4-c96f-4657-a926-3b8ef89a002b\" (UID: \"655f5af4-c96f-4657-a926-3b8ef89a002b\") " Nov 02 17:52:41.511998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.508320 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655f5af4-c96f-4657-a926-3b8ef89a002b-kube-api-access-zm4nw\" (UniqueName: \"kubernetes.io/projected/655f5af4-c96f-4657-a926-3b8ef89a002b-kube-api-access-zm4nw\") pod \"655f5af4-c96f-4657-a926-3b8ef89a002b\" (UID: \"655f5af4-c96f-4657-a926-3b8ef89a002b\") " Nov 02 17:52:41.511998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.509418 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655f5af4-c96f-4657-a926-3b8ef89a002b-plugins-config-volume" pod "655f5af4-c96f-4657-a926-3b8ef89a002b" (UID: "655f5af4-c96f-4657-a926-3b8ef89a002b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:41.511998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.509975 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h" Nov 02 17:52:41.511998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.511609 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655f5af4-c96f-4657-a926-3b8ef89a002b-kube-api-access-zm4nw" pod "655f5af4-c96f-4657-a926-3b8ef89a002b" (UID: "655f5af4-c96f-4657-a926-3b8ef89a002b"). InnerVolumeSpecName "kube-api-access-zm4nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:41.610135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.610094 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c6c7ac6-b524-465f-8195-447bbc3e5695-kube-api-access-n6hxb\" (UniqueName: \"kubernetes.io/projected/6c6c7ac6-b524-465f-8195-447bbc3e5695-kube-api-access-n6hxb\") pod \"6c6c7ac6-b524-465f-8195-447bbc3e5695\" (UID: \"6c6c7ac6-b524-465f-8195-447bbc3e5695\") " Nov 02 17:52:41.610308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.610286 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c6c7ac6-b524-465f-8195-447bbc3e5695-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c6c7ac6-b524-465f-8195-447bbc3e5695-plugins-config-volume\") pod \"6c6c7ac6-b524-465f-8195-447bbc3e5695\" (UID: \"6c6c7ac6-b524-465f-8195-447bbc3e5695\") " Nov 02 17:52:41.610657 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.610636 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm4nw\" (UniqueName: \"kubernetes.io/projected/655f5af4-c96f-4657-a926-3b8ef89a002b-kube-api-access-zm4nw\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:41.610657 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.610643 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6c7ac6-b524-465f-8195-447bbc3e5695-plugins-config-volume" pod "6c6c7ac6-b524-465f-8195-447bbc3e5695" (UID: "6c6c7ac6-b524-465f-8195-447bbc3e5695"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:41.610816 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.610652 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655f5af4-c96f-4657-a926-3b8ef89a002b-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:41.612296 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.612248 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6c7ac6-b524-465f-8195-447bbc3e5695-kube-api-access-n6hxb" pod "6c6c7ac6-b524-465f-8195-447bbc3e5695" (UID: "6c6c7ac6-b524-465f-8195-447bbc3e5695"). InnerVolumeSpecName "kube-api-access-n6hxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:41.712591 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.712551 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c6c7ac6-b524-465f-8195-447bbc3e5695-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:41.712591 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.712584 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6hxb\" (UniqueName: \"kubernetes.io/projected/6c6c7ac6-b524-465f-8195-447bbc3e5695-kube-api-access-n6hxb\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:41.817378 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.817337 5549 generic.go:356] "Generic (PLEG): container finished" podID="655f5af4-c96f-4657-a926-3b8ef89a002b" containerID="4aa1de65b8320b444c02e24a2f9b61a1ba9865cf104b02393ffde8fcd196c109" exitCode=0 Nov 02 17:52:41.817378 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.817376 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6" Nov 02 17:52:41.817645 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.817418 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6" event={"ID":"655f5af4-c96f-4657-a926-3b8ef89a002b","Type":"ContainerDied","Data":"4aa1de65b8320b444c02e24a2f9b61a1ba9865cf104b02393ffde8fcd196c109"} Nov 02 17:52:41.817645 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.817470 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6" event={"ID":"655f5af4-c96f-4657-a926-3b8ef89a002b","Type":"ContainerDied","Data":"35d44cd123807dbe1c61e344f6f49709a319a033c2d89e47f48c4b94ec83d66a"} Nov 02 17:52:41.817645 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.817499 5549 scope.go:122] "RemoveContainer" containerID="4aa1de65b8320b444c02e24a2f9b61a1ba9865cf104b02393ffde8fcd196c109" Nov 02 17:52:41.821359 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.821329 5549 generic.go:356] "Generic (PLEG): container finished" podID="6c6c7ac6-b524-465f-8195-447bbc3e5695" containerID="9149e24ffaab27a1b2e6fa45258db33dd07ade28cf5aac949fe82027bd95aa59" exitCode=0 Nov 02 17:52:41.821542 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.821518 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h" event={"ID":"6c6c7ac6-b524-465f-8195-447bbc3e5695","Type":"ContainerDied","Data":"9149e24ffaab27a1b2e6fa45258db33dd07ade28cf5aac949fe82027bd95aa59"} Nov 02 17:52:41.821665 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.821560 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h" event={"ID":"6c6c7ac6-b524-465f-8195-447bbc3e5695","Type":"ContainerDied","Data":"a462e4ccfd5ebf835912e64dfd090140e5fad11e6259c7fb013e9117b2accf58"} Nov 02 17:52:41.821665 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.821634 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h" Nov 02 17:52:41.835035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.835011 5549 scope.go:122] "RemoveContainer" containerID="4aa1de65b8320b444c02e24a2f9b61a1ba9865cf104b02393ffde8fcd196c109" Nov 02 17:52:41.835443 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:52:41.835400 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa1de65b8320b444c02e24a2f9b61a1ba9865cf104b02393ffde8fcd196c109\": container with ID starting with 4aa1de65b8320b444c02e24a2f9b61a1ba9865cf104b02393ffde8fcd196c109 not found: ID does not exist" containerID="4aa1de65b8320b444c02e24a2f9b61a1ba9865cf104b02393ffde8fcd196c109" Nov 02 17:52:41.835566 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.835444 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa1de65b8320b444c02e24a2f9b61a1ba9865cf104b02393ffde8fcd196c109"} err="failed to get container status \"4aa1de65b8320b444c02e24a2f9b61a1ba9865cf104b02393ffde8fcd196c109\": rpc error: code = NotFound desc = could not find container \"4aa1de65b8320b444c02e24a2f9b61a1ba9865cf104b02393ffde8fcd196c109\": container with ID starting with 4aa1de65b8320b444c02e24a2f9b61a1ba9865cf104b02393ffde8fcd196c109 not found: ID does not exist" Nov 02 17:52:41.835566 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.835469 5549 scope.go:122] "RemoveContainer" containerID="9149e24ffaab27a1b2e6fa45258db33dd07ade28cf5aac949fe82027bd95aa59" Nov 02 17:52:41.852745 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.852724 5549 scope.go:122] "RemoveContainer" containerID="9149e24ffaab27a1b2e6fa45258db33dd07ade28cf5aac949fe82027bd95aa59" Nov 02 17:52:41.853110 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:52:41.853086 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9149e24ffaab27a1b2e6fa45258db33dd07ade28cf5aac949fe82027bd95aa59\": container with ID starting with 9149e24ffaab27a1b2e6fa45258db33dd07ade28cf5aac949fe82027bd95aa59 not found: ID does not exist" containerID="9149e24ffaab27a1b2e6fa45258db33dd07ade28cf5aac949fe82027bd95aa59" Nov 02 17:52:41.853201 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.853120 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9149e24ffaab27a1b2e6fa45258db33dd07ade28cf5aac949fe82027bd95aa59"} err="failed to get container status \"9149e24ffaab27a1b2e6fa45258db33dd07ade28cf5aac949fe82027bd95aa59\": rpc error: code = NotFound desc = could not find container \"9149e24ffaab27a1b2e6fa45258db33dd07ade28cf5aac949fe82027bd95aa59\": container with ID starting with 9149e24ffaab27a1b2e6fa45258db33dd07ade28cf5aac949fe82027bd95aa59 not found: ID does not exist" Nov 02 17:52:41.860283 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.860254 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6"] Nov 02 17:52:41.872863 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.872807 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e4fdc1-67ccbbd9c7-gqzb6"] Nov 02 17:52:41.889939 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.889916 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h"] Nov 02 17:52:41.896302 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.896277 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e4fdc1-5f4588b47f-8fd6h"] Nov 02 17:52:41.982969 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:41.982928 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:42.179230 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:42.179152 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:42.320631 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:42.320599 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="655f5af4-c96f-4657-a926-3b8ef89a002b" path="/var/lib/kubelet/pods/655f5af4-c96f-4657-a926-3b8ef89a002b/volumes" Nov 02 17:52:42.321060 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:42.321044 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c6c7ac6-b524-465f-8195-447bbc3e5695" path="/var/lib/kubelet/pods/6c6c7ac6-b524-465f-8195-447bbc3e5695/volumes" Nov 02 17:52:42.390284 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:42.390248 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:42.563992 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:42.563632 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:42.967409 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:42.967366 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:43.184884 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:43.184854 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:52:43.393583 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:43.393496 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:44.217283 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:44.217222 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:44.418878 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:44.418835 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:44.643076 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:44.640338 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:44.934089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:44.928196 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:45.267676 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.267636 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl"] Nov 02 17:52:45.268308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.267941 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" podUID="316538fa-77cd-4d89-8844-0be9cb16cc5c" containerName="maas-api" containerID="cri-o://77a3ab5605a7de74d0b4c5336057b88baaf2f5eec17b61d593a394d5e999741f" gracePeriod=30 Nov 02 17:52:45.418406 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.418371 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:45.672978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.662813 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:45.701002 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.700635 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" Nov 02 17:52:45.839254 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.839159 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/316538fa-77cd-4d89-8844-0be9cb16cc5c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/316538fa-77cd-4d89-8844-0be9cb16cc5c-metrics-certs\") pod \"316538fa-77cd-4d89-8844-0be9cb16cc5c\" (UID: \"316538fa-77cd-4d89-8844-0be9cb16cc5c\") " Nov 02 17:52:45.839254 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.839234 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/316538fa-77cd-4d89-8844-0be9cb16cc5c-kube-api-access-nf2kb\" (UniqueName: \"kubernetes.io/projected/316538fa-77cd-4d89-8844-0be9cb16cc5c-kube-api-access-nf2kb\") pod \"316538fa-77cd-4d89-8844-0be9cb16cc5c\" (UID: \"316538fa-77cd-4d89-8844-0be9cb16cc5c\") " Nov 02 17:52:45.839491 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.839349 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/316538fa-77cd-4d89-8844-0be9cb16cc5c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316538fa-77cd-4d89-8844-0be9cb16cc5c-openshift-service-ca\") pod \"316538fa-77cd-4d89-8844-0be9cb16cc5c\" (UID: \"316538fa-77cd-4d89-8844-0be9cb16cc5c\") " Nov 02 17:52:45.839491 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.839398 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/316538fa-77cd-4d89-8844-0be9cb16cc5c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316538fa-77cd-4d89-8844-0be9cb16cc5c-maas-api-tls\") pod \"316538fa-77cd-4d89-8844-0be9cb16cc5c\" (UID: \"316538fa-77cd-4d89-8844-0be9cb16cc5c\") " Nov 02 17:52:45.839793 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.839763 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/316538fa-77cd-4d89-8844-0be9cb16cc5c-openshift-service-ca" pod "316538fa-77cd-4d89-8844-0be9cb16cc5c" (UID: "316538fa-77cd-4d89-8844-0be9cb16cc5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:45.840179 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.840153 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316538fa-77cd-4d89-8844-0be9cb16cc5c-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:45.841525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.841493 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/316538fa-77cd-4d89-8844-0be9cb16cc5c-maas-api-tls" pod "316538fa-77cd-4d89-8844-0be9cb16cc5c" (UID: "316538fa-77cd-4d89-8844-0be9cb16cc5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:52:45.842022 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.841988 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/316538fa-77cd-4d89-8844-0be9cb16cc5c-metrics-certs" pod "316538fa-77cd-4d89-8844-0be9cb16cc5c" (UID: "316538fa-77cd-4d89-8844-0be9cb16cc5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:52:45.842124 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.842018 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316538fa-77cd-4d89-8844-0be9cb16cc5c-kube-api-access-nf2kb" pod "316538fa-77cd-4d89-8844-0be9cb16cc5c" (UID: "316538fa-77cd-4d89-8844-0be9cb16cc5c"). InnerVolumeSpecName "kube-api-access-nf2kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:45.860089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.860062 5549 generic.go:356] "Generic (PLEG): container finished" podID="316538fa-77cd-4d89-8844-0be9cb16cc5c" containerID="77a3ab5605a7de74d0b4c5336057b88baaf2f5eec17b61d593a394d5e999741f" exitCode=0 Nov 02 17:52:45.860198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.860111 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" Nov 02 17:52:45.860198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.860151 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" event={"ID":"316538fa-77cd-4d89-8844-0be9cb16cc5c","Type":"ContainerDied","Data":"77a3ab5605a7de74d0b4c5336057b88baaf2f5eec17b61d593a394d5e999741f"} Nov 02 17:52:45.860278 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.860212 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" event={"ID":"316538fa-77cd-4d89-8844-0be9cb16cc5c","Type":"ContainerDied","Data":"fd0a2ba40ea06783b8c7fdbacf1f1c9956997eeebf38d502b975f8c4ddb8c601"} Nov 02 17:52:45.860278 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.860244 5549 scope.go:122] "RemoveContainer" containerID="77a3ab5605a7de74d0b4c5336057b88baaf2f5eec17b61d593a394d5e999741f" Nov 02 17:52:45.882995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.882961 5549 scope.go:122] "RemoveContainer" containerID="77a3ab5605a7de74d0b4c5336057b88baaf2f5eec17b61d593a394d5e999741f" Nov 02 17:52:45.889291 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:52:45.888702 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a3ab5605a7de74d0b4c5336057b88baaf2f5eec17b61d593a394d5e999741f\": container with ID starting with 77a3ab5605a7de74d0b4c5336057b88baaf2f5eec17b61d593a394d5e999741f not found: ID does not exist" containerID="77a3ab5605a7de74d0b4c5336057b88baaf2f5eec17b61d593a394d5e999741f" Nov 02 17:52:45.889291 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.888808 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a3ab5605a7de74d0b4c5336057b88baaf2f5eec17b61d593a394d5e999741f"} err="failed to get container status \"77a3ab5605a7de74d0b4c5336057b88baaf2f5eec17b61d593a394d5e999741f\": rpc error: code = NotFound desc = could not find container \"77a3ab5605a7de74d0b4c5336057b88baaf2f5eec17b61d593a394d5e999741f\": container with ID starting with 77a3ab5605a7de74d0b4c5336057b88baaf2f5eec17b61d593a394d5e999741f not found: ID does not exist" Nov 02 17:52:45.911853 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.911749 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl"] Nov 02 17:52:45.945141 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.945062 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/316538fa-77cd-4d89-8844-0be9cb16cc5c-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:45.945141 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.945099 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf2kb\" (UniqueName: \"kubernetes.io/projected/316538fa-77cd-4d89-8844-0be9cb16cc5c-kube-api-access-nf2kb\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:45.945141 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.945115 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316538fa-77cd-4d89-8844-0be9cb16cc5c-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:45.946835 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.946784 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl"] Nov 02 17:52:45.967143 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:45.967113 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:46.321100 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:46.321070 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="316538fa-77cd-4d89-8844-0be9cb16cc5c" path="/var/lib/kubelet/pods/316538fa-77cd-4d89-8844-0be9cb16cc5c/volumes" Nov 02 17:52:46.752438 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:46.752409 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" Nov 02 17:52:46.865962 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:46.865919 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:47.591859 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:52:47.591818 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:52:47.654323 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:47.654284 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d"] Nov 02 17:52:47.654557 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:47.654530 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" podUID="31bd29fd-f13a-4516-913a-45ea08661487" containerName="maas-api" containerID="cri-o://01590de2bc45996922b0fc0fd1e7332c44b44b30bd3aa952470cf41f41806d47" gracePeriod=30 Nov 02 17:52:47.881345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:47.881249 5549 generic.go:356] "Generic (PLEG): container finished" podID="31bd29fd-f13a-4516-913a-45ea08661487" containerID="01590de2bc45996922b0fc0fd1e7332c44b44b30bd3aa952470cf41f41806d47" exitCode=0 Nov 02 17:52:47.881345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:47.881326 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" event={"ID":"31bd29fd-f13a-4516-913a-45ea08661487","Type":"ContainerDied","Data":"01590de2bc45996922b0fc0fd1e7332c44b44b30bd3aa952470cf41f41806d47"} Nov 02 17:52:48.050350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.042782 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" Nov 02 17:52:48.099772 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.099738 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31bd29fd-f13a-4516-913a-45ea08661487-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31bd29fd-f13a-4516-913a-45ea08661487-maas-api-tls\") pod \"31bd29fd-f13a-4516-913a-45ea08661487\" (UID: \"31bd29fd-f13a-4516-913a-45ea08661487\") " Nov 02 17:52:48.100198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.100172 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31bd29fd-f13a-4516-913a-45ea08661487-metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bd29fd-f13a-4516-913a-45ea08661487-metrics-certs\") pod \"31bd29fd-f13a-4516-913a-45ea08661487\" (UID: \"31bd29fd-f13a-4516-913a-45ea08661487\") " Nov 02 17:52:48.100434 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.100415 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31bd29fd-f13a-4516-913a-45ea08661487-kube-api-access-tq842\" (UniqueName: \"kubernetes.io/projected/31bd29fd-f13a-4516-913a-45ea08661487-kube-api-access-tq842\") pod \"31bd29fd-f13a-4516-913a-45ea08661487\" (UID: \"31bd29fd-f13a-4516-913a-45ea08661487\") " Nov 02 17:52:48.103096 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.103071 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31bd29fd-f13a-4516-913a-45ea08661487-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bd29fd-f13a-4516-913a-45ea08661487-openshift-service-ca\") pod \"31bd29fd-f13a-4516-913a-45ea08661487\" (UID: \"31bd29fd-f13a-4516-913a-45ea08661487\") " Nov 02 17:52:48.103830 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.103778 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31bd29fd-f13a-4516-913a-45ea08661487-openshift-service-ca" pod "31bd29fd-f13a-4516-913a-45ea08661487" (UID: "31bd29fd-f13a-4516-913a-45ea08661487"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:48.104391 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.104368 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bd29fd-f13a-4516-913a-45ea08661487-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:48.114159 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.114133 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31bd29fd-f13a-4516-913a-45ea08661487-metrics-certs" pod "31bd29fd-f13a-4516-913a-45ea08661487" (UID: "31bd29fd-f13a-4516-913a-45ea08661487"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:52:48.117446 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.117421 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31bd29fd-f13a-4516-913a-45ea08661487-kube-api-access-tq842" pod "31bd29fd-f13a-4516-913a-45ea08661487" (UID: "31bd29fd-f13a-4516-913a-45ea08661487"). InnerVolumeSpecName "kube-api-access-tq842". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:48.120976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.120924 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31bd29fd-f13a-4516-913a-45ea08661487-maas-api-tls" pod "31bd29fd-f13a-4516-913a-45ea08661487" (UID: "31bd29fd-f13a-4516-913a-45ea08661487"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:52:48.228386 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.210452 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31bd29fd-f13a-4516-913a-45ea08661487-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:48.228386 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.210487 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bd29fd-f13a-4516-913a-45ea08661487-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:48.228386 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.210500 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq842\" (UniqueName: \"kubernetes.io/projected/31bd29fd-f13a-4516-913a-45ea08661487-kube-api-access-tq842\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:48.239384 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.239310 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:48.528587 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.528539 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c9298-54f4ccb9-db6sl" podUID="316538fa-77cd-4d89-8844-0be9cb16cc5c" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.110:8443/health\": context deadline exceeded" Nov 02 17:52:48.559158 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.556194 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:48.631308 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:52:48.631272 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31bd29fd_f13a_4516_913a_45ea08661487.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31bd29fd_f13a_4516_913a_45ea08661487.slice/crio-afe9a36a8468f8bfa7b7f39a105fa837900c0c3099ba59a115296959ce2d2d08\": RecentStats: unable to find data in memory cache]" Nov 02 17:52:48.889902 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.889823 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" event={"ID":"31bd29fd-f13a-4516-913a-45ea08661487","Type":"ContainerDied","Data":"afe9a36a8468f8bfa7b7f39a105fa837900c0c3099ba59a115296959ce2d2d08"} Nov 02 17:52:48.889902 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.889863 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d" Nov 02 17:52:48.889902 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.889882 5549 scope.go:122] "RemoveContainer" containerID="01590de2bc45996922b0fc0fd1e7332c44b44b30bd3aa952470cf41f41806d47" Nov 02 17:52:48.920160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.920131 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d"] Nov 02 17:52:48.929844 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:48.929820 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7dbdb8-679f86b9b7-4998d"] Nov 02 17:52:49.495975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:49.495781 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:50.359976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:50.351760 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:50.382985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:50.382938 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31bd29fd-f13a-4516-913a-45ea08661487" path="/var/lib/kubelet/pods/31bd29fd-f13a-4516-913a-45ea08661487/volumes" Nov 02 17:52:50.547542 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:50.547508 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:50.961976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:50.959494 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:51.236035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:51.235943 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8"] Nov 02 17:52:51.236301 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:51.236273 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" podUID="98455203-d4a6-4bc7-af34-b901edb31b83" containerName="istio-proxy" containerID="cri-o://84073160c33beb89c4c9e242994bd75e9ad5b332294db11d9ffe5313f0474c28" gracePeriod=30 Nov 02 17:52:51.365976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:51.361196 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:51.565455 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:51.565371 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:52.195974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:52.191599 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:52.376709 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:52.376674 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:52.747125 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:52.747074 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:53.021132 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:53.021043 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k"] Nov 02 17:52:53.021306 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:53.021272 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" podUID="a56a6b8e-233b-4256-b47f-48a323ce8f55" containerName="istio-proxy" containerID="cri-o://6b435e4d67a67a5d850dbda594960f74658c4d0e0e02c997672db7d7f641662f" gracePeriod=30 Nov 02 17:52:53.325043 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:53.322438 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:53.836426 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:53.836385 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:54.154048 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:54.153966 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:54.555839 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:54.552574 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:55.554287 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:55.554248 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:56.039585 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.039531 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb"] Nov 02 17:52:56.043381 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.043351 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69"] Nov 02 17:52:56.174304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.174261 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:56.382651 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.382621 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:56.656074 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.654460 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:56.713580 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.713557 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:52:56.765213 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.765181 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-workload-certs\") pod \"98455203-d4a6-4bc7-af34-b901edb31b83\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " Nov 02 17:52:56.765402 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.765226 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98455203-d4a6-4bc7-af34-b901edb31b83-kube-api-access-9plxr\" (UniqueName: \"kubernetes.io/projected/98455203-d4a6-4bc7-af34-b901edb31b83-kube-api-access-9plxr\") pod \"98455203-d4a6-4bc7-af34-b901edb31b83\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " Nov 02 17:52:56.765402 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.765264 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-istio-data\") pod \"98455203-d4a6-4bc7-af34-b901edb31b83\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " Nov 02 17:52:56.765402 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.765297 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-workload-socket\") pod \"98455203-d4a6-4bc7-af34-b901edb31b83\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " Nov 02 17:52:56.765402 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.765362 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98455203-d4a6-4bc7-af34-b901edb31b83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98455203-d4a6-4bc7-af34-b901edb31b83-istio-podinfo\") pod \"98455203-d4a6-4bc7-af34-b901edb31b83\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " Nov 02 17:52:56.765641 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.765419 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-credential-socket\") pod \"98455203-d4a6-4bc7-af34-b901edb31b83\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " Nov 02 17:52:56.765641 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.765514 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-workload-certs" pod "98455203-d4a6-4bc7-af34-b901edb31b83" (UID: "98455203-d4a6-4bc7-af34-b901edb31b83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:56.765749 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.765671 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98455203-d4a6-4bc7-af34-b901edb31b83-istio-token\" (UniqueName: \"kubernetes.io/projected/98455203-d4a6-4bc7-af34-b901edb31b83-istio-token\") pod \"98455203-d4a6-4bc7-af34-b901edb31b83\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " Nov 02 17:52:56.765749 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.765704 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-workload-socket" pod "98455203-d4a6-4bc7-af34-b901edb31b83" (UID: "98455203-d4a6-4bc7-af34-b901edb31b83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:56.765749 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.765706 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-istio-envoy\") pod \"98455203-d4a6-4bc7-af34-b901edb31b83\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " Nov 02 17:52:56.765749 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.765727 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-credential-socket" pod "98455203-d4a6-4bc7-af34-b901edb31b83" (UID: "98455203-d4a6-4bc7-af34-b901edb31b83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:56.765749 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.765521 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-istio-data" pod "98455203-d4a6-4bc7-af34-b901edb31b83" (UID: "98455203-d4a6-4bc7-af34-b901edb31b83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:56.766040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.765819 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98455203-d4a6-4bc7-af34-b901edb31b83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98455203-d4a6-4bc7-af34-b901edb31b83-istiod-ca-cert\") pod \"98455203-d4a6-4bc7-af34-b901edb31b83\" (UID: \"98455203-d4a6-4bc7-af34-b901edb31b83\") " Nov 02 17:52:56.766397 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.766367 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98455203-d4a6-4bc7-af34-b901edb31b83-istiod-ca-cert" pod "98455203-d4a6-4bc7-af34-b901edb31b83" (UID: "98455203-d4a6-4bc7-af34-b901edb31b83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:56.766619 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.766598 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:56.766690 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.766625 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:56.766690 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.766643 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:56.766690 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.766659 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:56.766690 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.766675 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98455203-d4a6-4bc7-af34-b901edb31b83-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:56.768360 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.768326 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98455203-d4a6-4bc7-af34-b901edb31b83-istio-token" pod "98455203-d4a6-4bc7-af34-b901edb31b83" (UID: "98455203-d4a6-4bc7-af34-b901edb31b83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:56.768759 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.768736 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-istio-envoy" pod "98455203-d4a6-4bc7-af34-b901edb31b83" (UID: "98455203-d4a6-4bc7-af34-b901edb31b83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:56.768917 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.768854 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98455203-d4a6-4bc7-af34-b901edb31b83-istio-podinfo" pod "98455203-d4a6-4bc7-af34-b901edb31b83" (UID: "98455203-d4a6-4bc7-af34-b901edb31b83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:52:56.769012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.768935 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98455203-d4a6-4bc7-af34-b901edb31b83-kube-api-access-9plxr" pod "98455203-d4a6-4bc7-af34-b901edb31b83" (UID: "98455203-d4a6-4bc7-af34-b901edb31b83"). InnerVolumeSpecName "kube-api-access-9plxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:56.868655 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.868620 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98455203-d4a6-4bc7-af34-b901edb31b83-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:56.868655 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.868651 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98455203-d4a6-4bc7-af34-b901edb31b83-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:56.868655 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.868662 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9plxr\" (UniqueName: \"kubernetes.io/projected/98455203-d4a6-4bc7-af34-b901edb31b83-kube-api-access-9plxr\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:56.868896 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.868671 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98455203-d4a6-4bc7-af34-b901edb31b83-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:56.952851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.952819 5549 generic.go:356] "Generic (PLEG): container finished" podID="98455203-d4a6-4bc7-af34-b901edb31b83" containerID="84073160c33beb89c4c9e242994bd75e9ad5b332294db11d9ffe5313f0474c28" exitCode=0 Nov 02 17:52:56.953038 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.952873 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" Nov 02 17:52:56.953038 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.952905 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" event={"ID":"98455203-d4a6-4bc7-af34-b901edb31b83","Type":"ContainerDied","Data":"84073160c33beb89c4c9e242994bd75e9ad5b332294db11d9ffe5313f0474c28"} Nov 02 17:52:56.953038 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.952971 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8" event={"ID":"98455203-d4a6-4bc7-af34-b901edb31b83","Type":"ContainerDied","Data":"71fbb3a16e5e239205ce290fa4fd4544f0ae54a4bb704b64e115b0a90dc3d410"} Nov 02 17:52:56.953038 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.953007 5549 scope.go:122] "RemoveContainer" containerID="84073160c33beb89c4c9e242994bd75e9ad5b332294db11d9ffe5313f0474c28" Nov 02 17:52:56.964725 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.964706 5549 scope.go:122] "RemoveContainer" containerID="1ca8cb082abbaf44cedadfaad543f43d18a780ac912e6b2612370f2bf3dfd38a" Nov 02 17:52:56.974576 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.974474 5549 scope.go:122] "RemoveContainer" containerID="84073160c33beb89c4c9e242994bd75e9ad5b332294db11d9ffe5313f0474c28" Nov 02 17:52:56.974850 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:52:56.974822 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84073160c33beb89c4c9e242994bd75e9ad5b332294db11d9ffe5313f0474c28\": container with ID starting with 84073160c33beb89c4c9e242994bd75e9ad5b332294db11d9ffe5313f0474c28 not found: ID does not exist" containerID="84073160c33beb89c4c9e242994bd75e9ad5b332294db11d9ffe5313f0474c28" Nov 02 17:52:56.975026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.974859 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84073160c33beb89c4c9e242994bd75e9ad5b332294db11d9ffe5313f0474c28"} err="failed to get container status \"84073160c33beb89c4c9e242994bd75e9ad5b332294db11d9ffe5313f0474c28\": rpc error: code = NotFound desc = could not find container \"84073160c33beb89c4c9e242994bd75e9ad5b332294db11d9ffe5313f0474c28\": container with ID starting with 84073160c33beb89c4c9e242994bd75e9ad5b332294db11d9ffe5313f0474c28 not found: ID does not exist" Nov 02 17:52:56.975026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.974889 5549 scope.go:122] "RemoveContainer" containerID="1ca8cb082abbaf44cedadfaad543f43d18a780ac912e6b2612370f2bf3dfd38a" Nov 02 17:52:56.975205 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:52:56.975179 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca8cb082abbaf44cedadfaad543f43d18a780ac912e6b2612370f2bf3dfd38a\": container with ID starting with 1ca8cb082abbaf44cedadfaad543f43d18a780ac912e6b2612370f2bf3dfd38a not found: ID does not exist" containerID="1ca8cb082abbaf44cedadfaad543f43d18a780ac912e6b2612370f2bf3dfd38a" Nov 02 17:52:56.975271 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.975214 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca8cb082abbaf44cedadfaad543f43d18a780ac912e6b2612370f2bf3dfd38a"} err="failed to get container status \"1ca8cb082abbaf44cedadfaad543f43d18a780ac912e6b2612370f2bf3dfd38a\": rpc error: code = NotFound desc = could not find container \"1ca8cb082abbaf44cedadfaad543f43d18a780ac912e6b2612370f2bf3dfd38a\": container with ID starting with 1ca8cb082abbaf44cedadfaad543f43d18a780ac912e6b2612370f2bf3dfd38a not found: ID does not exist" Nov 02 17:52:56.978116 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.978092 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8"] Nov 02 17:52:56.985052 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:56.985028 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-7dbdb8-openshift-default-7bb994464f-mhrv8"] Nov 02 17:52:57.433028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:57.423385 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:57.614929 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:57.614895 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5"] Nov 02 17:52:57.736737 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:57.736703 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:57.894877 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:57.894827 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:58.443829 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.443791 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98455203-d4a6-4bc7-af34-b901edb31b83" path="/var/lib/kubelet/pods/98455203-d4a6-4bc7-af34-b901edb31b83/volumes" Nov 02 17:52:58.466768 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.466730 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:52:58.514716 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.514679 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-podinfo\") pod \"a56a6b8e-233b-4256-b47f-48a323ce8f55\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " Nov 02 17:52:58.514914 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.514763 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-workload-socket\") pod \"a56a6b8e-233b-4256-b47f-48a323ce8f55\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " Nov 02 17:52:58.514914 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.514830 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a56a6b8e-233b-4256-b47f-48a323ce8f55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a56a6b8e-233b-4256-b47f-48a323ce8f55-istiod-ca-cert\") pod \"a56a6b8e-233b-4256-b47f-48a323ce8f55\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " Nov 02 17:52:58.515106 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.514922 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-token\" (UniqueName: \"kubernetes.io/projected/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-token\") pod \"a56a6b8e-233b-4256-b47f-48a323ce8f55\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " Nov 02 17:52:58.515106 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.515010 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-credential-socket\") pod \"a56a6b8e-233b-4256-b47f-48a323ce8f55\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " Nov 02 17:52:58.515106 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.515049 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-workload-certs\") pod \"a56a6b8e-233b-4256-b47f-48a323ce8f55\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " Nov 02 17:52:58.515106 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.515079 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-envoy\") pod \"a56a6b8e-233b-4256-b47f-48a323ce8f55\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " Nov 02 17:52:58.515345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.515105 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-workload-socket" pod "a56a6b8e-233b-4256-b47f-48a323ce8f55" (UID: "a56a6b8e-233b-4256-b47f-48a323ce8f55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:58.515345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.515164 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a56a6b8e-233b-4256-b47f-48a323ce8f55-kube-api-access-5tbdr\" (UniqueName: \"kubernetes.io/projected/a56a6b8e-233b-4256-b47f-48a323ce8f55-kube-api-access-5tbdr\") pod \"a56a6b8e-233b-4256-b47f-48a323ce8f55\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " Nov 02 17:52:58.515345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.515336 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-credential-socket" pod "a56a6b8e-233b-4256-b47f-48a323ce8f55" (UID: "a56a6b8e-233b-4256-b47f-48a323ce8f55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:58.515613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.515593 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a56a6b8e-233b-4256-b47f-48a323ce8f55-istiod-ca-cert" pod "a56a6b8e-233b-4256-b47f-48a323ce8f55" (UID: "a56a6b8e-233b-4256-b47f-48a323ce8f55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:58.515925 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.515895 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:58.516070 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.515930 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a56a6b8e-233b-4256-b47f-48a323ce8f55-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:58.516070 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.515965 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:58.518180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.518151 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-workload-certs" pod "a56a6b8e-233b-4256-b47f-48a323ce8f55" (UID: "a56a6b8e-233b-4256-b47f-48a323ce8f55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:58.519233 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.519209 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-envoy" pod "a56a6b8e-233b-4256-b47f-48a323ce8f55" (UID: "a56a6b8e-233b-4256-b47f-48a323ce8f55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:58.520097 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.520048 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-podinfo" pod "a56a6b8e-233b-4256-b47f-48a323ce8f55" (UID: "a56a6b8e-233b-4256-b47f-48a323ce8f55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:52:58.520429 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.520403 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-token" pod "a56a6b8e-233b-4256-b47f-48a323ce8f55" (UID: "a56a6b8e-233b-4256-b47f-48a323ce8f55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:58.521136 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.521117 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a56a6b8e-233b-4256-b47f-48a323ce8f55-kube-api-access-5tbdr" pod "a56a6b8e-233b-4256-b47f-48a323ce8f55" (UID: "a56a6b8e-233b-4256-b47f-48a323ce8f55"). InnerVolumeSpecName "kube-api-access-5tbdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:58.639046 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.630219 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:58.657605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.650535 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-data\") pod \"a56a6b8e-233b-4256-b47f-48a323ce8f55\" (UID: \"a56a6b8e-233b-4256-b47f-48a323ce8f55\") " Nov 02 17:52:58.657605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.651498 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:58.657605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.651532 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:58.657605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.651546 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:58.657605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.651560 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:58.657605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.651574 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tbdr\" (UniqueName: \"kubernetes.io/projected/a56a6b8e-233b-4256-b47f-48a323ce8f55-kube-api-access-5tbdr\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:58.661222 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.661138 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-data" pod "a56a6b8e-233b-4256-b47f-48a323ce8f55" (UID: "a56a6b8e-233b-4256-b47f-48a323ce8f55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:58.753332 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.753299 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a56a6b8e-233b-4256-b47f-48a323ce8f55-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:58.846517 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:58.846426 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:59.015219 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:59.015130 5549 generic.go:356] "Generic (PLEG): container finished" podID="a56a6b8e-233b-4256-b47f-48a323ce8f55" containerID="6b435e4d67a67a5d850dbda594960f74658c4d0e0e02c997672db7d7f641662f" exitCode=0 Nov 02 17:52:59.015219 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:59.015193 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" Nov 02 17:52:59.015424 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:59.015220 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" event={"ID":"a56a6b8e-233b-4256-b47f-48a323ce8f55","Type":"ContainerDied","Data":"6b435e4d67a67a5d850dbda594960f74658c4d0e0e02c997672db7d7f641662f"} Nov 02 17:52:59.015424 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:59.015300 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k" event={"ID":"a56a6b8e-233b-4256-b47f-48a323ce8f55","Type":"ContainerDied","Data":"95d146050b28ba4fbf5a7780a82c010436b0574d6e2d0c1ff12d63b45934badb"} Nov 02 17:52:59.015424 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:59.015322 5549 scope.go:122] "RemoveContainer" containerID="6b435e4d67a67a5d850dbda594960f74658c4d0e0e02c997672db7d7f641662f" Nov 02 17:52:59.026530 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:59.026514 5549 scope.go:122] "RemoveContainer" containerID="485d45edf1e797dcee8144203100e279ccda4ed50a42bde120abfb90bfda8207" Nov 02 17:52:59.036490 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:59.036471 5549 scope.go:122] "RemoveContainer" containerID="6b435e4d67a67a5d850dbda594960f74658c4d0e0e02c997672db7d7f641662f" Nov 02 17:52:59.036784 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:52:59.036763 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b435e4d67a67a5d850dbda594960f74658c4d0e0e02c997672db7d7f641662f\": container with ID starting with 6b435e4d67a67a5d850dbda594960f74658c4d0e0e02c997672db7d7f641662f not found: ID does not exist" containerID="6b435e4d67a67a5d850dbda594960f74658c4d0e0e02c997672db7d7f641662f" Nov 02 17:52:59.036864 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:59.036793 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b435e4d67a67a5d850dbda594960f74658c4d0e0e02c997672db7d7f641662f"} err="failed to get container status \"6b435e4d67a67a5d850dbda594960f74658c4d0e0e02c997672db7d7f641662f\": rpc error: code = NotFound desc = could not find container \"6b435e4d67a67a5d850dbda594960f74658c4d0e0e02c997672db7d7f641662f\": container with ID starting with 6b435e4d67a67a5d850dbda594960f74658c4d0e0e02c997672db7d7f641662f not found: ID does not exist" Nov 02 17:52:59.036864 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:59.036811 5549 scope.go:122] "RemoveContainer" containerID="485d45edf1e797dcee8144203100e279ccda4ed50a42bde120abfb90bfda8207" Nov 02 17:52:59.037063 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:52:59.037045 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"485d45edf1e797dcee8144203100e279ccda4ed50a42bde120abfb90bfda8207\": container with ID starting with 485d45edf1e797dcee8144203100e279ccda4ed50a42bde120abfb90bfda8207 not found: ID does not exist" containerID="485d45edf1e797dcee8144203100e279ccda4ed50a42bde120abfb90bfda8207" Nov 02 17:52:59.037135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:59.037067 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"485d45edf1e797dcee8144203100e279ccda4ed50a42bde120abfb90bfda8207"} err="failed to get container status \"485d45edf1e797dcee8144203100e279ccda4ed50a42bde120abfb90bfda8207\": rpc error: code = NotFound desc = could not find container \"485d45edf1e797dcee8144203100e279ccda4ed50a42bde120abfb90bfda8207\": container with ID starting with 485d45edf1e797dcee8144203100e279ccda4ed50a42bde120abfb90bfda8207 not found: ID does not exist" Nov 02 17:52:59.045794 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:59.045768 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k"] Nov 02 17:52:59.048641 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:59.048615 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-qzf9k"] Nov 02 17:52:59.132238 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:59.132205 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:59.645977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:52:59.645455 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:00.078967 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:00.078919 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:00.321769 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:00.321721 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a56a6b8e-233b-4256-b47f-48a323ce8f55" path="/var/lib/kubelet/pods/a56a6b8e-233b-4256-b47f-48a323ce8f55/volumes" Nov 02 17:53:00.489542 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:00.488923 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:00.548712 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:00.548676 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b"] Nov 02 17:53:00.557062 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:00.557032 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg"] Nov 02 17:53:00.690182 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:00.690151 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:00.857899 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:00.857811 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:01.040267 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:01.040224 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb" podUID="769444b9-fcd1-46d4-8753-27a9a8a7d3f2" containerName="payload-pre-processing" containerID="cri-o://efb049584f721aa17c804dbf2e33487f5cf3723f781b9bfb6a741d178162b89b" gracePeriod=25 Nov 02 17:53:01.044390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:01.044358 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69" podUID="007b76f2-5968-404f-a034-09bc84e9db73" containerName="payload-processing" containerID="cri-o://a669dc534e1866c9a43e42bbaca3d8b956128fbaceed0fbe3bc97b3045b2f2d7" gracePeriod=25 Nov 02 17:53:01.336892 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:01.336857 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb" Nov 02 17:53:01.360124 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:01.360093 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69" Nov 02 17:53:01.399340 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:01.399265 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007b76f2-5968-404f-a034-09bc84e9db73-kube-api-access-mwggg\" (UniqueName: \"kubernetes.io/projected/007b76f2-5968-404f-a034-09bc84e9db73-kube-api-access-mwggg\") pod \"007b76f2-5968-404f-a034-09bc84e9db73\" (UID: \"007b76f2-5968-404f-a034-09bc84e9db73\") " Nov 02 17:53:01.399488 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:01.399414 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/769444b9-fcd1-46d4-8753-27a9a8a7d3f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/769444b9-fcd1-46d4-8753-27a9a8a7d3f2-plugins-config-volume\") pod \"769444b9-fcd1-46d4-8753-27a9a8a7d3f2\" (UID: \"769444b9-fcd1-46d4-8753-27a9a8a7d3f2\") " Nov 02 17:53:01.399488 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:01.399460 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769444b9-fcd1-46d4-8753-27a9a8a7d3f2-kube-api-access-cf97k\" (UniqueName: \"kubernetes.io/projected/769444b9-fcd1-46d4-8753-27a9a8a7d3f2-kube-api-access-cf97k\") pod \"769444b9-fcd1-46d4-8753-27a9a8a7d3f2\" (UID: \"769444b9-fcd1-46d4-8753-27a9a8a7d3f2\") " Nov 02 17:53:01.399583 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:01.399526 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007b76f2-5968-404f-a034-09bc84e9db73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007b76f2-5968-404f-a034-09bc84e9db73-plugins-config-volume\") pod \"007b76f2-5968-404f-a034-09bc84e9db73\" (UID: \"007b76f2-5968-404f-a034-09bc84e9db73\") " Nov 02 17:53:01.400004 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:01.399940 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769444b9-fcd1-46d4-8753-27a9a8a7d3f2-plugins-config-volume" pod "769444b9-fcd1-46d4-8753-27a9a8a7d3f2" (UID: "769444b9-fcd1-46d4-8753-27a9a8a7d3f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:01.400134 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:01.400046 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007b76f2-5968-404f-a034-09bc84e9db73-plugins-config-volume" pod "007b76f2-5968-404f-a034-09bc84e9db73" (UID: "007b76f2-5968-404f-a034-09bc84e9db73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:01.400375 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:01.400352 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/769444b9-fcd1-46d4-8753-27a9a8a7d3f2-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:01.400458 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:01.400382 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007b76f2-5968-404f-a034-09bc84e9db73-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:01.402025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:01.401997 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007b76f2-5968-404f-a034-09bc84e9db73-kube-api-access-mwggg" pod "007b76f2-5968-404f-a034-09bc84e9db73" (UID: "007b76f2-5968-404f-a034-09bc84e9db73"). InnerVolumeSpecName "kube-api-access-mwggg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:01.402197 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:01.402179 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769444b9-fcd1-46d4-8753-27a9a8a7d3f2-kube-api-access-cf97k" pod "769444b9-fcd1-46d4-8753-27a9a8a7d3f2" (UID: "769444b9-fcd1-46d4-8753-27a9a8a7d3f2"). InnerVolumeSpecName "kube-api-access-cf97k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:01.501806 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:01.501764 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwggg\" (UniqueName: \"kubernetes.io/projected/007b76f2-5968-404f-a034-09bc84e9db73-kube-api-access-mwggg\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:01.501806 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:01.501800 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf97k\" (UniqueName: \"kubernetes.io/projected/769444b9-fcd1-46d4-8753-27a9a8a7d3f2-kube-api-access-cf97k\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:01.576660 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:01.576626 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:01.744784 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:01.744750 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:02.042865 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.042767 5549 generic.go:356] "Generic (PLEG): container finished" podID="007b76f2-5968-404f-a034-09bc84e9db73" containerID="a669dc534e1866c9a43e42bbaca3d8b956128fbaceed0fbe3bc97b3045b2f2d7" exitCode=0 Nov 02 17:53:02.042865 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.042822 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69" Nov 02 17:53:02.042865 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.042854 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69" event={"ID":"007b76f2-5968-404f-a034-09bc84e9db73","Type":"ContainerDied","Data":"a669dc534e1866c9a43e42bbaca3d8b956128fbaceed0fbe3bc97b3045b2f2d7"} Nov 02 17:53:02.043237 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.042925 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69" event={"ID":"007b76f2-5968-404f-a034-09bc84e9db73","Type":"ContainerDied","Data":"80cc23f396436c70175602ffc8aaeba727ac8079492cd1bc496c71e8a248b72f"} Nov 02 17:53:02.043237 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.042965 5549 scope.go:122] "RemoveContainer" containerID="a669dc534e1866c9a43e42bbaca3d8b956128fbaceed0fbe3bc97b3045b2f2d7" Nov 02 17:53:02.044516 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.044493 5549 generic.go:356] "Generic (PLEG): container finished" podID="769444b9-fcd1-46d4-8753-27a9a8a7d3f2" containerID="efb049584f721aa17c804dbf2e33487f5cf3723f781b9bfb6a741d178162b89b" exitCode=0 Nov 02 17:53:02.044621 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.044548 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb" Nov 02 17:53:02.044621 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.044568 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb" event={"ID":"769444b9-fcd1-46d4-8753-27a9a8a7d3f2","Type":"ContainerDied","Data":"efb049584f721aa17c804dbf2e33487f5cf3723f781b9bfb6a741d178162b89b"} Nov 02 17:53:02.044724 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.044622 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb" event={"ID":"769444b9-fcd1-46d4-8753-27a9a8a7d3f2","Type":"ContainerDied","Data":"d001ab103ae59e5c1a7606fd987dc2c2615dfcae1f52f2b785a56ded6cdc474b"} Nov 02 17:53:02.055266 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.055242 5549 scope.go:122] "RemoveContainer" containerID="a669dc534e1866c9a43e42bbaca3d8b956128fbaceed0fbe3bc97b3045b2f2d7" Nov 02 17:53:02.055597 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:53:02.055577 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a669dc534e1866c9a43e42bbaca3d8b956128fbaceed0fbe3bc97b3045b2f2d7\": container with ID starting with a669dc534e1866c9a43e42bbaca3d8b956128fbaceed0fbe3bc97b3045b2f2d7 not found: ID does not exist" containerID="a669dc534e1866c9a43e42bbaca3d8b956128fbaceed0fbe3bc97b3045b2f2d7" Nov 02 17:53:02.055662 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.055606 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a669dc534e1866c9a43e42bbaca3d8b956128fbaceed0fbe3bc97b3045b2f2d7"} err="failed to get container status \"a669dc534e1866c9a43e42bbaca3d8b956128fbaceed0fbe3bc97b3045b2f2d7\": rpc error: code = NotFound desc = could not find container \"a669dc534e1866c9a43e42bbaca3d8b956128fbaceed0fbe3bc97b3045b2f2d7\": container with ID starting with a669dc534e1866c9a43e42bbaca3d8b956128fbaceed0fbe3bc97b3045b2f2d7 not found: ID does not exist" Nov 02 17:53:02.055662 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.055622 5549 scope.go:122] "RemoveContainer" containerID="efb049584f721aa17c804dbf2e33487f5cf3723f781b9bfb6a741d178162b89b" Nov 02 17:53:02.070219 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.070195 5549 scope.go:122] "RemoveContainer" containerID="efb049584f721aa17c804dbf2e33487f5cf3723f781b9bfb6a741d178162b89b" Nov 02 17:53:02.070577 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:53:02.070556 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb049584f721aa17c804dbf2e33487f5cf3723f781b9bfb6a741d178162b89b\": container with ID starting with efb049584f721aa17c804dbf2e33487f5cf3723f781b9bfb6a741d178162b89b not found: ID does not exist" containerID="efb049584f721aa17c804dbf2e33487f5cf3723f781b9bfb6a741d178162b89b" Nov 02 17:53:02.070658 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.070584 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb049584f721aa17c804dbf2e33487f5cf3723f781b9bfb6a741d178162b89b"} err="failed to get container status \"efb049584f721aa17c804dbf2e33487f5cf3723f781b9bfb6a741d178162b89b\": rpc error: code = NotFound desc = could not find container \"efb049584f721aa17c804dbf2e33487f5cf3723f781b9bfb6a741d178162b89b\": container with ID starting with efb049584f721aa17c804dbf2e33487f5cf3723f781b9bfb6a741d178162b89b not found: ID does not exist" Nov 02 17:53:02.089624 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.089588 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb"] Nov 02 17:53:02.095297 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.095252 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-6764675cc5-c8qcb"] Nov 02 17:53:02.110749 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.110713 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69"] Nov 02 17:53:02.120516 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.120485 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-9dc6895df-hfl69"] Nov 02 17:53:02.320926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.320841 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="007b76f2-5968-404f-a034-09bc84e9db73" path="/var/lib/kubelet/pods/007b76f2-5968-404f-a034-09bc84e9db73/volumes" Nov 02 17:53:02.321322 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.321308 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="769444b9-fcd1-46d4-8753-27a9a8a7d3f2" path="/var/lib/kubelet/pods/769444b9-fcd1-46d4-8753-27a9a8a7d3f2/volumes" Nov 02 17:53:02.616205 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.616091 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5" podUID="c4c1e656-62c3-41d7-b29d-c120c194315e" containerName="payload-pre-processing" containerID="cri-o://e0e2e009bf01d8995d01a42ad47691055f95eb0a9800828222cd142d9e20406d" gracePeriod=25 Nov 02 17:53:02.658389 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.658339 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:02.921745 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.917666 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:02.951723 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:02.951694 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5" Nov 02 17:53:03.053753 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:03.053717 5549 generic.go:356] "Generic (PLEG): container finished" podID="c4c1e656-62c3-41d7-b29d-c120c194315e" containerID="e0e2e009bf01d8995d01a42ad47691055f95eb0a9800828222cd142d9e20406d" exitCode=0 Nov 02 17:53:03.054032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:03.053769 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5" Nov 02 17:53:03.054032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:03.053792 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5" event={"ID":"c4c1e656-62c3-41d7-b29d-c120c194315e","Type":"ContainerDied","Data":"e0e2e009bf01d8995d01a42ad47691055f95eb0a9800828222cd142d9e20406d"} Nov 02 17:53:03.054032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:03.053886 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5" event={"ID":"c4c1e656-62c3-41d7-b29d-c120c194315e","Type":"ContainerDied","Data":"547dbefa23be7df52fd80da249328683d91b4a7413512b2f5dd267588a63696e"} Nov 02 17:53:03.054032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:03.053918 5549 scope.go:122] "RemoveContainer" containerID="e0e2e009bf01d8995d01a42ad47691055f95eb0a9800828222cd142d9e20406d" Nov 02 17:53:03.057037 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:03.057005 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4c1e656-62c3-41d7-b29d-c120c194315e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c1e656-62c3-41d7-b29d-c120c194315e-plugins-config-volume\") pod \"c4c1e656-62c3-41d7-b29d-c120c194315e\" (UID: \"c4c1e656-62c3-41d7-b29d-c120c194315e\") " Nov 02 17:53:03.057249 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:03.057229 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4c1e656-62c3-41d7-b29d-c120c194315e-kube-api-access-n466v\" (UniqueName: \"kubernetes.io/projected/c4c1e656-62c3-41d7-b29d-c120c194315e-kube-api-access-n466v\") pod \"c4c1e656-62c3-41d7-b29d-c120c194315e\" (UID: \"c4c1e656-62c3-41d7-b29d-c120c194315e\") " Nov 02 17:53:03.057408 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:03.057382 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4c1e656-62c3-41d7-b29d-c120c194315e-plugins-config-volume" pod "c4c1e656-62c3-41d7-b29d-c120c194315e" (UID: "c4c1e656-62c3-41d7-b29d-c120c194315e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:03.057933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:03.057909 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c1e656-62c3-41d7-b29d-c120c194315e-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:03.059347 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:03.059325 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4c1e656-62c3-41d7-b29d-c120c194315e-kube-api-access-n466v" pod "c4c1e656-62c3-41d7-b29d-c120c194315e" (UID: "c4c1e656-62c3-41d7-b29d-c120c194315e"). InnerVolumeSpecName "kube-api-access-n466v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:03.081818 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:03.081796 5549 scope.go:122] "RemoveContainer" containerID="e0e2e009bf01d8995d01a42ad47691055f95eb0a9800828222cd142d9e20406d" Nov 02 17:53:03.082160 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:53:03.082138 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e2e009bf01d8995d01a42ad47691055f95eb0a9800828222cd142d9e20406d\": container with ID starting with e0e2e009bf01d8995d01a42ad47691055f95eb0a9800828222cd142d9e20406d not found: ID does not exist" containerID="e0e2e009bf01d8995d01a42ad47691055f95eb0a9800828222cd142d9e20406d" Nov 02 17:53:03.082226 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:03.082170 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e2e009bf01d8995d01a42ad47691055f95eb0a9800828222cd142d9e20406d"} err="failed to get container status \"e0e2e009bf01d8995d01a42ad47691055f95eb0a9800828222cd142d9e20406d\": rpc error: code = NotFound desc = could not find container \"e0e2e009bf01d8995d01a42ad47691055f95eb0a9800828222cd142d9e20406d\": container with ID starting with e0e2e009bf01d8995d01a42ad47691055f95eb0a9800828222cd142d9e20406d not found: ID does not exist" Nov 02 17:53:03.159476 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:03.159392 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n466v\" (UniqueName: \"kubernetes.io/projected/c4c1e656-62c3-41d7-b29d-c120c194315e-kube-api-access-n466v\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:03.380679 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:03.380648 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5"] Nov 02 17:53:03.385554 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:03.385531 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4a5f09-cc78df9f5-nnsw5"] Nov 02 17:53:03.483976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:03.477503 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:03.967567 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:03.967534 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:04.025311 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.025280 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l"] Nov 02 17:53:04.025989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.025973 5549 cpu_manager.go:414] "containerMap: removing container" podUID="007b76f2-5968-404f-a034-09bc84e9db73" containerName="payload-processing" Nov 02 17:53:04.026040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.025990 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007b76f2-5968-404f-a034-09bc84e9db73" containerName="payload-processing" Nov 02 17:53:04.026040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026008 5549 cpu_manager.go:414] "containerMap: removing container" podUID="98455203-d4a6-4bc7-af34-b901edb31b83" containerName="istio-proxy" Nov 02 17:53:04.026040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026014 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98455203-d4a6-4bc7-af34-b901edb31b83" containerName="istio-proxy" Nov 02 17:53:04.026040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026030 5549 cpu_manager.go:414] "containerMap: removing container" podUID="316538fa-77cd-4d89-8844-0be9cb16cc5c" containerName="maas-api" Nov 02 17:53:04.026040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026037 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="316538fa-77cd-4d89-8844-0be9cb16cc5c" containerName="maas-api" Nov 02 17:53:04.026235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026053 5549 cpu_manager.go:414] "containerMap: removing container" podUID="a56a6b8e-233b-4256-b47f-48a323ce8f55" containerName="istio-proxy" Nov 02 17:53:04.026235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026058 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a56a6b8e-233b-4256-b47f-48a323ce8f55" containerName="istio-proxy" Nov 02 17:53:04.026235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026065 5549 cpu_manager.go:414] "containerMap: removing container" podUID="c4c1e656-62c3-41d7-b29d-c120c194315e" containerName="payload-pre-processing" Nov 02 17:53:04.026235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026070 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4c1e656-62c3-41d7-b29d-c120c194315e" containerName="payload-pre-processing" Nov 02 17:53:04.026235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026080 5549 cpu_manager.go:414] "containerMap: removing container" podUID="a56a6b8e-233b-4256-b47f-48a323ce8f55" containerName="istio-proxy" Nov 02 17:53:04.026235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026085 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a56a6b8e-233b-4256-b47f-48a323ce8f55" containerName="istio-proxy" Nov 02 17:53:04.026235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026093 5549 cpu_manager.go:414] "containerMap: removing container" podUID="655f5af4-c96f-4657-a926-3b8ef89a002b" containerName="payload-processing" Nov 02 17:53:04.026235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026098 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="655f5af4-c96f-4657-a926-3b8ef89a002b" containerName="payload-processing" Nov 02 17:53:04.026235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026112 5549 cpu_manager.go:414] "containerMap: removing container" podUID="6c6c7ac6-b524-465f-8195-447bbc3e5695" containerName="payload-pre-processing" Nov 02 17:53:04.026235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026118 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c6c7ac6-b524-465f-8195-447bbc3e5695" containerName="payload-pre-processing" Nov 02 17:53:04.026235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026142 5549 cpu_manager.go:414] "containerMap: removing container" podUID="769444b9-fcd1-46d4-8753-27a9a8a7d3f2" containerName="payload-pre-processing" Nov 02 17:53:04.026235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026147 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769444b9-fcd1-46d4-8753-27a9a8a7d3f2" containerName="payload-pre-processing" Nov 02 17:53:04.026235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026154 5549 cpu_manager.go:414] "containerMap: removing container" podUID="31bd29fd-f13a-4516-913a-45ea08661487" containerName="maas-api" Nov 02 17:53:04.026235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026158 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31bd29fd-f13a-4516-913a-45ea08661487" containerName="maas-api" Nov 02 17:53:04.026802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026267 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c6c7ac6-b524-465f-8195-447bbc3e5695" containerName="payload-pre-processing" Nov 02 17:53:04.026802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026281 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="769444b9-fcd1-46d4-8753-27a9a8a7d3f2" containerName="payload-pre-processing" Nov 02 17:53:04.026802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026294 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="31bd29fd-f13a-4516-913a-45ea08661487" containerName="maas-api" Nov 02 17:53:04.026802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026302 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="98455203-d4a6-4bc7-af34-b901edb31b83" containerName="istio-proxy" Nov 02 17:53:04.026802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026309 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="007b76f2-5968-404f-a034-09bc84e9db73" containerName="payload-processing" Nov 02 17:53:04.026802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026322 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="316538fa-77cd-4d89-8844-0be9cb16cc5c" containerName="maas-api" Nov 02 17:53:04.026802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026332 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a56a6b8e-233b-4256-b47f-48a323ce8f55" containerName="istio-proxy" Nov 02 17:53:04.026802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026340 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4c1e656-62c3-41d7-b29d-c120c194315e" containerName="payload-pre-processing" Nov 02 17:53:04.026802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026352 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a56a6b8e-233b-4256-b47f-48a323ce8f55" containerName="istio-proxy" Nov 02 17:53:04.026802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.026361 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="655f5af4-c96f-4657-a926-3b8ef89a002b" containerName="payload-processing" Nov 02 17:53:04.031566 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.031548 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l" Nov 02 17:53:04.041610 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.041581 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eda8e2\"" Nov 02 17:53:04.042974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.042935 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l"] Nov 02 17:53:04.056483 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.056454 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eda8e2-dockercfg-ksz57\"" Nov 02 17:53:04.227055 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.212337 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0986e16-44a0-471e-9467-5373643adae6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l\" (UID: \"f0986e16-44a0-471e-9467-5373643adae6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l" Nov 02 17:53:04.228518 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.212529 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bprs6\" (UniqueName: \"kubernetes.io/projected/f0986e16-44a0-471e-9467-5373643adae6-kube-api-access-bprs6\") pod \"payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l\" (UID: \"f0986e16-44a0-471e-9467-5373643adae6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l" Nov 02 17:53:04.262057 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.253574 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:04.348527 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.340595 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bprs6\" (UniqueName: \"kubernetes.io/projected/f0986e16-44a0-471e-9467-5373643adae6-kube-api-access-bprs6\") pod \"payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l\" (UID: \"f0986e16-44a0-471e-9467-5373643adae6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l" Nov 02 17:53:04.353986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.350262 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0986e16-44a0-471e-9467-5373643adae6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l\" (UID: \"f0986e16-44a0-471e-9467-5373643adae6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l" Nov 02 17:53:04.368179 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.358884 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0986e16-44a0-471e-9467-5373643adae6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l\" (UID: \"f0986e16-44a0-471e-9467-5373643adae6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l" Nov 02 17:53:04.401801 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.398477 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4c1e656-62c3-41d7-b29d-c120c194315e" path="/var/lib/kubelet/pods/c4c1e656-62c3-41d7-b29d-c120c194315e/volumes" Nov 02 17:53:04.402993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.402465 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bprs6\" (UniqueName: \"kubernetes.io/projected/f0986e16-44a0-471e-9467-5373643adae6-kube-api-access-bprs6\") pod \"payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l\" (UID: \"f0986e16-44a0-471e-9467-5373643adae6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l" Nov 02 17:53:04.626190 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.626087 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:04.652932 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.652899 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l" Nov 02 17:53:04.846982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:04.838835 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:05.147279 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:05.147248 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l"] Nov 02 17:53:05.148738 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:53:05.148711 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0986e16_44a0_471e_9467_5373643adae6.slice/crio-00cd00b37e4938a4e600c30c54aa155c04d2d2d774fa8d41ad7f96d2ea22932b WatchSource:0}: Error finding container 00cd00b37e4938a4e600c30c54aa155c04d2d2d774fa8d41ad7f96d2ea22932b: Status 404 returned error can't find the container with id 00cd00b37e4938a4e600c30c54aa155c04d2d2d774fa8d41ad7f96d2ea22932b Nov 02 17:53:05.552071 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:05.552019 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b" podUID="c724e9bd-faa6-4867-9e87-499fad60a383" containerName="payload-processing" containerID="cri-o://a077a2500ea5c50a08359789ec595f6ac8fee24a9bf9d33f185e28dd1ee5cca5" gracePeriod=25 Nov 02 17:53:05.581221 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:05.575050 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg" podUID="899469e3-9cf7-455e-9481-4637653d73af" containerName="payload-pre-processing" containerID="cri-o://9102b9f3453608648d94244fd2c2dbe45dfea621f0eb0a96e0df2b69d91be886" gracePeriod=25 Nov 02 17:53:05.603431 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:05.601674 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:05.890627 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:05.890596 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b" Nov 02 17:53:05.933134 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:05.933098 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c724e9bd-faa6-4867-9e87-499fad60a383-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c724e9bd-faa6-4867-9e87-499fad60a383-plugins-config-volume\") pod \"c724e9bd-faa6-4867-9e87-499fad60a383\" (UID: \"c724e9bd-faa6-4867-9e87-499fad60a383\") " Nov 02 17:53:05.933332 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:05.933313 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c724e9bd-faa6-4867-9e87-499fad60a383-kube-api-access-gn4hw\" (UniqueName: \"kubernetes.io/projected/c724e9bd-faa6-4867-9e87-499fad60a383-kube-api-access-gn4hw\") pod \"c724e9bd-faa6-4867-9e87-499fad60a383\" (UID: \"c724e9bd-faa6-4867-9e87-499fad60a383\") " Nov 02 17:53:05.933522 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:05.933499 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c724e9bd-faa6-4867-9e87-499fad60a383-plugins-config-volume" pod "c724e9bd-faa6-4867-9e87-499fad60a383" (UID: "c724e9bd-faa6-4867-9e87-499fad60a383"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:05.933889 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:05.933858 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg" Nov 02 17:53:05.934077 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:05.933910 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c724e9bd-faa6-4867-9e87-499fad60a383-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:05.935667 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:05.935638 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c724e9bd-faa6-4867-9e87-499fad60a383-kube-api-access-gn4hw" pod "c724e9bd-faa6-4867-9e87-499fad60a383" (UID: "c724e9bd-faa6-4867-9e87-499fad60a383"). InnerVolumeSpecName "kube-api-access-gn4hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:06.034480 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.034441 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/899469e3-9cf7-455e-9481-4637653d73af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899469e3-9cf7-455e-9481-4637653d73af-plugins-config-volume\") pod \"899469e3-9cf7-455e-9481-4637653d73af\" (UID: \"899469e3-9cf7-455e-9481-4637653d73af\") " Nov 02 17:53:06.034629 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.034556 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/899469e3-9cf7-455e-9481-4637653d73af-kube-api-access-b8v6q\" (UniqueName: \"kubernetes.io/projected/899469e3-9cf7-455e-9481-4637653d73af-kube-api-access-b8v6q\") pod \"899469e3-9cf7-455e-9481-4637653d73af\" (UID: \"899469e3-9cf7-455e-9481-4637653d73af\") " Nov 02 17:53:06.034910 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.034884 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/899469e3-9cf7-455e-9481-4637653d73af-plugins-config-volume" pod "899469e3-9cf7-455e-9481-4637653d73af" (UID: "899469e3-9cf7-455e-9481-4637653d73af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:06.035211 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.035189 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899469e3-9cf7-455e-9481-4637653d73af-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:06.035272 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.035213 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn4hw\" (UniqueName: \"kubernetes.io/projected/c724e9bd-faa6-4867-9e87-499fad60a383-kube-api-access-gn4hw\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:06.036668 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.036642 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/899469e3-9cf7-455e-9481-4637653d73af-kube-api-access-b8v6q" pod "899469e3-9cf7-455e-9481-4637653d73af" (UID: "899469e3-9cf7-455e-9481-4637653d73af"). InnerVolumeSpecName "kube-api-access-b8v6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:06.084021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.083893 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l" event={"ID":"f0986e16-44a0-471e-9467-5373643adae6","Type":"ContainerStarted","Data":"f8e9777b93e64ea46a2da39718343dfb85001fa3f24e962ecfe149227eec9345"} Nov 02 17:53:06.084021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.084005 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l" Nov 02 17:53:06.084276 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.084031 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l" event={"ID":"f0986e16-44a0-471e-9467-5373643adae6","Type":"ContainerStarted","Data":"00cd00b37e4938a4e600c30c54aa155c04d2d2d774fa8d41ad7f96d2ea22932b"} Nov 02 17:53:06.085477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.085418 5549 generic.go:356] "Generic (PLEG): container finished" podID="c724e9bd-faa6-4867-9e87-499fad60a383" containerID="a077a2500ea5c50a08359789ec595f6ac8fee24a9bf9d33f185e28dd1ee5cca5" exitCode=0 Nov 02 17:53:06.085618 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.085472 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b" Nov 02 17:53:06.085618 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.085477 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b" event={"ID":"c724e9bd-faa6-4867-9e87-499fad60a383","Type":"ContainerDied","Data":"a077a2500ea5c50a08359789ec595f6ac8fee24a9bf9d33f185e28dd1ee5cca5"} Nov 02 17:53:06.085618 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.085597 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b" event={"ID":"c724e9bd-faa6-4867-9e87-499fad60a383","Type":"ContainerDied","Data":"8b6e41c1968805e58452a954779a5cba8bcaf140fdc75cf958759033e05aec7b"} Nov 02 17:53:06.085618 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.085614 5549 scope.go:122] "RemoveContainer" containerID="a077a2500ea5c50a08359789ec595f6ac8fee24a9bf9d33f185e28dd1ee5cca5" Nov 02 17:53:06.087209 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.087184 5549 generic.go:356] "Generic (PLEG): container finished" podID="899469e3-9cf7-455e-9481-4637653d73af" containerID="9102b9f3453608648d94244fd2c2dbe45dfea621f0eb0a96e0df2b69d91be886" exitCode=0 Nov 02 17:53:06.087304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.087248 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg" Nov 02 17:53:06.087304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.087264 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg" event={"ID":"899469e3-9cf7-455e-9481-4637653d73af","Type":"ContainerDied","Data":"9102b9f3453608648d94244fd2c2dbe45dfea621f0eb0a96e0df2b69d91be886"} Nov 02 17:53:06.087398 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.087311 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg" event={"ID":"899469e3-9cf7-455e-9481-4637653d73af","Type":"ContainerDied","Data":"21fa98de8d8fa1778d41814ed9cf5e5903b1faa09c5d1d4f288f8823d116d92c"} Nov 02 17:53:06.097500 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.097477 5549 scope.go:122] "RemoveContainer" containerID="a077a2500ea5c50a08359789ec595f6ac8fee24a9bf9d33f185e28dd1ee5cca5" Nov 02 17:53:06.097907 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:53:06.097883 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a077a2500ea5c50a08359789ec595f6ac8fee24a9bf9d33f185e28dd1ee5cca5\": container with ID starting with a077a2500ea5c50a08359789ec595f6ac8fee24a9bf9d33f185e28dd1ee5cca5 not found: ID does not exist" containerID="a077a2500ea5c50a08359789ec595f6ac8fee24a9bf9d33f185e28dd1ee5cca5" Nov 02 17:53:06.098024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.097916 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a077a2500ea5c50a08359789ec595f6ac8fee24a9bf9d33f185e28dd1ee5cca5"} err="failed to get container status \"a077a2500ea5c50a08359789ec595f6ac8fee24a9bf9d33f185e28dd1ee5cca5\": rpc error: code = NotFound desc = could not find container \"a077a2500ea5c50a08359789ec595f6ac8fee24a9bf9d33f185e28dd1ee5cca5\": container with ID starting with a077a2500ea5c50a08359789ec595f6ac8fee24a9bf9d33f185e28dd1ee5cca5 not found: ID does not exist" Nov 02 17:53:06.098024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.097935 5549 scope.go:122] "RemoveContainer" containerID="9102b9f3453608648d94244fd2c2dbe45dfea621f0eb0a96e0df2b69d91be886" Nov 02 17:53:06.107584 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.107564 5549 scope.go:122] "RemoveContainer" containerID="9102b9f3453608648d94244fd2c2dbe45dfea621f0eb0a96e0df2b69d91be886" Nov 02 17:53:06.107866 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:53:06.107845 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9102b9f3453608648d94244fd2c2dbe45dfea621f0eb0a96e0df2b69d91be886\": container with ID starting with 9102b9f3453608648d94244fd2c2dbe45dfea621f0eb0a96e0df2b69d91be886 not found: ID does not exist" containerID="9102b9f3453608648d94244fd2c2dbe45dfea621f0eb0a96e0df2b69d91be886" Nov 02 17:53:06.107924 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.107873 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9102b9f3453608648d94244fd2c2dbe45dfea621f0eb0a96e0df2b69d91be886"} err="failed to get container status \"9102b9f3453608648d94244fd2c2dbe45dfea621f0eb0a96e0df2b69d91be886\": rpc error: code = NotFound desc = could not find container \"9102b9f3453608648d94244fd2c2dbe45dfea621f0eb0a96e0df2b69d91be886\": container with ID starting with 9102b9f3453608648d94244fd2c2dbe45dfea621f0eb0a96e0df2b69d91be886 not found: ID does not exist" Nov 02 17:53:06.117418 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.117371 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l" podStartSLOduration=2.117358336 podStartE2EDuration="2.117358336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:06.114596279 +0000 UTC m=+1157.903101808" watchObservedRunningTime="2026-11-02 17:53:06.117358336 +0000 UTC m=+1157.905863864" Nov 02 17:53:06.136381 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.136343 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8v6q\" (UniqueName: \"kubernetes.io/projected/899469e3-9cf7-455e-9481-4637653d73af-kube-api-access-b8v6q\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:06.138746 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.138717 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg"] Nov 02 17:53:06.145742 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.145716 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-009a5f-945d46bf5-bd7wg"] Nov 02 17:53:06.173992 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.173935 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b"] Nov 02 17:53:06.179481 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.179451 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-009a5f-5cbdb6898c-2h44b"] Nov 02 17:53:06.319696 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.319663 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="899469e3-9cf7-455e-9481-4637653d73af" path="/var/lib/kubelet/pods/899469e3-9cf7-455e-9481-4637653d73af/volumes" Nov 02 17:53:06.320268 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.320242 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c724e9bd-faa6-4867-9e87-499fad60a383" path="/var/lib/kubelet/pods/c724e9bd-faa6-4867-9e87-499fad60a383/volumes" Nov 02 17:53:06.418070 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.417967 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j"] Nov 02 17:53:06.418281 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.418255 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" podUID="fad6f82c-b39c-4b16-92b2-64964826ad15" containerName="maas-api" containerID="cri-o://032d08caefd2ddac1885c6e1cdd31c00f0b405e4a6ac48427e150949e9e0a594" gracePeriod=30 Nov 02 17:53:06.511479 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.511432 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:06.845682 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:06.845528 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:07.069569 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.069178 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" Nov 02 17:53:07.086189 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.086147 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad6f82c-b39c-4b16-92b2-64964826ad15-kube-api-access-qcklb\" (UniqueName: \"kubernetes.io/projected/fad6f82c-b39c-4b16-92b2-64964826ad15-kube-api-access-qcklb\") pod \"fad6f82c-b39c-4b16-92b2-64964826ad15\" (UID: \"fad6f82c-b39c-4b16-92b2-64964826ad15\") " Nov 02 17:53:07.087045 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.087020 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad6f82c-b39c-4b16-92b2-64964826ad15-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad6f82c-b39c-4b16-92b2-64964826ad15-openshift-service-ca\") pod \"fad6f82c-b39c-4b16-92b2-64964826ad15\" (UID: \"fad6f82c-b39c-4b16-92b2-64964826ad15\") " Nov 02 17:53:07.111057 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.100510 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fad6f82c-b39c-4b16-92b2-64964826ad15-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad6f82c-b39c-4b16-92b2-64964826ad15-metrics-certs\") pod \"fad6f82c-b39c-4b16-92b2-64964826ad15\" (UID: \"fad6f82c-b39c-4b16-92b2-64964826ad15\") " Nov 02 17:53:07.111057 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.100696 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fad6f82c-b39c-4b16-92b2-64964826ad15-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fad6f82c-b39c-4b16-92b2-64964826ad15-maas-api-tls\") pod \"fad6f82c-b39c-4b16-92b2-64964826ad15\" (UID: \"fad6f82c-b39c-4b16-92b2-64964826ad15\") " Nov 02 17:53:07.111057 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.090522 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad6f82c-b39c-4b16-92b2-64964826ad15-openshift-service-ca" pod "fad6f82c-b39c-4b16-92b2-64964826ad15" (UID: "fad6f82c-b39c-4b16-92b2-64964826ad15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:07.111057 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.106542 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fad6f82c-b39c-4b16-92b2-64964826ad15-maas-api-tls" pod "fad6f82c-b39c-4b16-92b2-64964826ad15" (UID: "fad6f82c-b39c-4b16-92b2-64964826ad15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:07.113579 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.113286 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad6f82c-b39c-4b16-92b2-64964826ad15-kube-api-access-qcklb" pod "fad6f82c-b39c-4b16-92b2-64964826ad15" (UID: "fad6f82c-b39c-4b16-92b2-64964826ad15"). InnerVolumeSpecName "kube-api-access-qcklb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:07.121755 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.121636 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fad6f82c-b39c-4b16-92b2-64964826ad15-metrics-certs" pod "fad6f82c-b39c-4b16-92b2-64964826ad15" (UID: "fad6f82c-b39c-4b16-92b2-64964826ad15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:07.129986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.124363 5549 generic.go:356] "Generic (PLEG): container finished" podID="fad6f82c-b39c-4b16-92b2-64964826ad15" containerID="032d08caefd2ddac1885c6e1cdd31c00f0b405e4a6ac48427e150949e9e0a594" exitCode=0 Nov 02 17:53:07.129986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.124569 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" event={"ID":"fad6f82c-b39c-4b16-92b2-64964826ad15","Type":"ContainerDied","Data":"032d08caefd2ddac1885c6e1cdd31c00f0b405e4a6ac48427e150949e9e0a594"} Nov 02 17:53:07.129986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.124619 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" event={"ID":"fad6f82c-b39c-4b16-92b2-64964826ad15","Type":"ContainerDied","Data":"0f1bfe6deb4d9a31f71a43f147a430db356d49b595e1ada690c109ef0a159184"} Nov 02 17:53:07.129986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.124650 5549 scope.go:122] "RemoveContainer" containerID="032d08caefd2ddac1885c6e1cdd31c00f0b405e4a6ac48427e150949e9e0a594" Nov 02 17:53:07.129986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.124762 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j" Nov 02 17:53:07.160188 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.160158 5549 scope.go:122] "RemoveContainer" containerID="032d08caefd2ddac1885c6e1cdd31c00f0b405e4a6ac48427e150949e9e0a594" Nov 02 17:53:07.160600 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:53:07.160573 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"032d08caefd2ddac1885c6e1cdd31c00f0b405e4a6ac48427e150949e9e0a594\": container with ID starting with 032d08caefd2ddac1885c6e1cdd31c00f0b405e4a6ac48427e150949e9e0a594 not found: ID does not exist" containerID="032d08caefd2ddac1885c6e1cdd31c00f0b405e4a6ac48427e150949e9e0a594" Nov 02 17:53:07.160690 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.160607 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"032d08caefd2ddac1885c6e1cdd31c00f0b405e4a6ac48427e150949e9e0a594"} err="failed to get container status \"032d08caefd2ddac1885c6e1cdd31c00f0b405e4a6ac48427e150949e9e0a594\": rpc error: code = NotFound desc = could not find container \"032d08caefd2ddac1885c6e1cdd31c00f0b405e4a6ac48427e150949e9e0a594\": container with ID starting with 032d08caefd2ddac1885c6e1cdd31c00f0b405e4a6ac48427e150949e9e0a594 not found: ID does not exist" Nov 02 17:53:07.179622 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.179581 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j"] Nov 02 17:53:07.187743 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.187710 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a5f09-67678cf57b-sts4j"] Nov 02 17:53:07.203095 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.203060 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcklb\" (UniqueName: \"kubernetes.io/projected/fad6f82c-b39c-4b16-92b2-64964826ad15-kube-api-access-qcklb\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:07.203095 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.203095 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad6f82c-b39c-4b16-92b2-64964826ad15-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:07.203340 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.203112 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad6f82c-b39c-4b16-92b2-64964826ad15-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:07.203340 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.203126 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fad6f82c-b39c-4b16-92b2-64964826ad15-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:07.968998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:07.967801 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:08.321531 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:08.321450 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad6f82c-b39c-4b16-92b2-64964826ad15" path="/var/lib/kubelet/pods/fad6f82c-b39c-4b16-92b2-64964826ad15/volumes" Nov 02 17:53:09.040404 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:09.040370 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:09.257467 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:09.257423 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:09.446809 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:09.446773 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:09.686124 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:09.686084 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:10.156117 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.156076 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv"] Nov 02 17:53:10.156410 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.156380 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" podUID="454db7ea-3a85-4bb3-aa88-5f9dd74466ce" containerName="maas-api" containerID="cri-o://b71c12ac3236aaf9fd5526150bdecfd3588901233ee66377a7a842cabe86df3e" gracePeriod=30 Nov 02 17:53:10.261229 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.261185 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:10.280870 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.280839 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5"] Nov 02 17:53:10.281171 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.281142 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5" podUID="7ca344ab-a3fb-498c-9bc9-703ea0816597" containerName="payload-pre-processing" containerID="cri-o://ae8496ae029403bd64bcc2340a77acd8d868e7aa26db2829c24eb9e6304888f1" gracePeriod=30 Nov 02 17:53:10.316602 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.316567 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc"] Nov 02 17:53:10.317888 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.317859 5549 cpu_manager.go:414] "containerMap: removing container" podUID="98455203-d4a6-4bc7-af34-b901edb31b83" containerName="istio-proxy" Nov 02 17:53:10.317888 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.317885 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98455203-d4a6-4bc7-af34-b901edb31b83" containerName="istio-proxy" Nov 02 17:53:10.318093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.317923 5549 cpu_manager.go:414] "containerMap: removing container" podUID="899469e3-9cf7-455e-9481-4637653d73af" containerName="payload-pre-processing" Nov 02 17:53:10.318093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.317932 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="899469e3-9cf7-455e-9481-4637653d73af" containerName="payload-pre-processing" Nov 02 17:53:10.318093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.318006 5549 cpu_manager.go:414] "containerMap: removing container" podUID="c724e9bd-faa6-4867-9e87-499fad60a383" containerName="payload-processing" Nov 02 17:53:10.318093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.318015 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c724e9bd-faa6-4867-9e87-499fad60a383" containerName="payload-processing" Nov 02 17:53:10.318093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.318033 5549 cpu_manager.go:414] "containerMap: removing container" podUID="fad6f82c-b39c-4b16-92b2-64964826ad15" containerName="maas-api" Nov 02 17:53:10.318093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.318053 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad6f82c-b39c-4b16-92b2-64964826ad15" containerName="maas-api" Nov 02 17:53:10.318455 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.318232 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c724e9bd-faa6-4867-9e87-499fad60a383" containerName="payload-processing" Nov 02 17:53:10.318455 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.318245 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad6f82c-b39c-4b16-92b2-64964826ad15" containerName="maas-api" Nov 02 17:53:10.318455 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.318257 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="98455203-d4a6-4bc7-af34-b901edb31b83" containerName="istio-proxy" Nov 02 17:53:10.318455 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.318284 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="899469e3-9cf7-455e-9481-4637653d73af" containerName="payload-pre-processing" Nov 02 17:53:10.325372 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.325347 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc" Nov 02 17:53:10.338296 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.337096 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-6f78de\"" Nov 02 17:53:10.338296 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.338254 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-6f78de-dockercfg-cp876\"" Nov 02 17:53:10.345639 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.345609 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc"] Nov 02 17:53:10.354804 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.354773 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn"] Nov 02 17:53:10.355732 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.355610 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f268007-ed65-4c71-a091-9a8fa16f4e43-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc\" (UID: \"6f268007-ed65-4c71-a091-9a8fa16f4e43\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc" Nov 02 17:53:10.356596 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.355818 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75nmx\" (UniqueName: \"kubernetes.io/projected/6f268007-ed65-4c71-a091-9a8fa16f4e43-kube-api-access-75nmx\") pod \"payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc\" (UID: \"6f268007-ed65-4c71-a091-9a8fa16f4e43\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc" Nov 02 17:53:10.368519 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.368478 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn"] Nov 02 17:53:10.368826 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.368812 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn" Nov 02 17:53:10.457925 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.457097 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f268007-ed65-4c71-a091-9a8fa16f4e43-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc\" (UID: \"6f268007-ed65-4c71-a091-9a8fa16f4e43\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc" Nov 02 17:53:10.457925 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.457169 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84xhx\" (UniqueName: \"kubernetes.io/projected/da93fd4f-c062-43b0-add5-167738bd7489-kube-api-access-84xhx\") pod \"payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn\" (UID: \"da93fd4f-c062-43b0-add5-167738bd7489\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn" Nov 02 17:53:10.457925 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.457209 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da93fd4f-c062-43b0-add5-167738bd7489-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn\" (UID: \"da93fd4f-c062-43b0-add5-167738bd7489\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn" Nov 02 17:53:10.465828 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.458831 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p"] Nov 02 17:53:10.465828 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.459096 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" podUID="932f3be9-81c1-4473-8b41-46870d0fc7fa" containerName="maas-api" containerID="cri-o://c50e625665d23d90366261dc7016c64c6a4f9ac0a748ee82965ab2b58cb9fdb6" gracePeriod=30 Nov 02 17:53:10.465828 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.464200 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f268007-ed65-4c71-a091-9a8fa16f4e43-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc\" (UID: \"6f268007-ed65-4c71-a091-9a8fa16f4e43\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc" Nov 02 17:53:10.465828 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.465729 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:10.467882 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.467445 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75nmx\" (UniqueName: \"kubernetes.io/projected/6f268007-ed65-4c71-a091-9a8fa16f4e43-kube-api-access-75nmx\") pod \"payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc\" (UID: \"6f268007-ed65-4c71-a091-9a8fa16f4e43\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc" Nov 02 17:53:10.480621 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.480583 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75nmx\" (UniqueName: \"kubernetes.io/projected/6f268007-ed65-4c71-a091-9a8fa16f4e43-kube-api-access-75nmx\") pod \"payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc\" (UID: \"6f268007-ed65-4c71-a091-9a8fa16f4e43\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc" Nov 02 17:53:10.569507 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.569381 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84xhx\" (UniqueName: \"kubernetes.io/projected/da93fd4f-c062-43b0-add5-167738bd7489-kube-api-access-84xhx\") pod \"payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn\" (UID: \"da93fd4f-c062-43b0-add5-167738bd7489\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn" Nov 02 17:53:10.569507 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.569438 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da93fd4f-c062-43b0-add5-167738bd7489-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn\" (UID: \"da93fd4f-c062-43b0-add5-167738bd7489\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn" Nov 02 17:53:10.570577 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.570549 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da93fd4f-c062-43b0-add5-167738bd7489-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn\" (UID: \"da93fd4f-c062-43b0-add5-167738bd7489\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn" Nov 02 17:53:10.579896 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.579872 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84xhx\" (UniqueName: \"kubernetes.io/projected/da93fd4f-c062-43b0-add5-167738bd7489-kube-api-access-84xhx\") pod \"payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn\" (UID: \"da93fd4f-c062-43b0-add5-167738bd7489\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn" Nov 02 17:53:10.591434 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.591404 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" Nov 02 17:53:10.597405 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.597375 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5" Nov 02 17:53:10.671050 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.671011 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-openshift-service-ca\") pod \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\" (UID: \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\") " Nov 02 17:53:10.671250 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.671227 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca344ab-a3fb-498c-9bc9-703ea0816597-kube-api-access-hxnjm\" (UniqueName: \"kubernetes.io/projected/7ca344ab-a3fb-498c-9bc9-703ea0816597-kube-api-access-hxnjm\") pod \"7ca344ab-a3fb-498c-9bc9-703ea0816597\" (UID: \"7ca344ab-a3fb-498c-9bc9-703ea0816597\") " Nov 02 17:53:10.671348 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.671307 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-metrics-certs\") pod \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\" (UID: \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\") " Nov 02 17:53:10.671475 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.671444 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-kube-api-access-btv9k\" (UniqueName: \"kubernetes.io/projected/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-kube-api-access-btv9k\") pod \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\" (UID: \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\") " Nov 02 17:53:10.671591 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.671478 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca344ab-a3fb-498c-9bc9-703ea0816597-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca344ab-a3fb-498c-9bc9-703ea0816597-plugins-config-volume\") pod \"7ca344ab-a3fb-498c-9bc9-703ea0816597\" (UID: \"7ca344ab-a3fb-498c-9bc9-703ea0816597\") " Nov 02 17:53:10.671591 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.671571 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-maas-api-tls\") pod \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\" (UID: \"454db7ea-3a85-4bb3-aa88-5f9dd74466ce\") " Nov 02 17:53:10.672144 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.672111 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-openshift-service-ca" pod "454db7ea-3a85-4bb3-aa88-5f9dd74466ce" (UID: "454db7ea-3a85-4bb3-aa88-5f9dd74466ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:10.672319 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.672299 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:10.672742 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.672720 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca344ab-a3fb-498c-9bc9-703ea0816597-plugins-config-volume" pod "7ca344ab-a3fb-498c-9bc9-703ea0816597" (UID: "7ca344ab-a3fb-498c-9bc9-703ea0816597"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:10.676522 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.676485 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-kube-api-access-btv9k" pod "454db7ea-3a85-4bb3-aa88-5f9dd74466ce" (UID: "454db7ea-3a85-4bb3-aa88-5f9dd74466ce"). InnerVolumeSpecName "kube-api-access-btv9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:10.677289 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.677225 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-metrics-certs" pod "454db7ea-3a85-4bb3-aa88-5f9dd74466ce" (UID: "454db7ea-3a85-4bb3-aa88-5f9dd74466ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:10.677595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.677568 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-maas-api-tls" pod "454db7ea-3a85-4bb3-aa88-5f9dd74466ce" (UID: "454db7ea-3a85-4bb3-aa88-5f9dd74466ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:10.679159 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.679128 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca344ab-a3fb-498c-9bc9-703ea0816597-kube-api-access-hxnjm" pod "7ca344ab-a3fb-498c-9bc9-703ea0816597" (UID: "7ca344ab-a3fb-498c-9bc9-703ea0816597"). InnerVolumeSpecName "kube-api-access-hxnjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:10.690196 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.690166 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc" Nov 02 17:53:10.701585 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.701554 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn" Nov 02 17:53:10.710537 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.710426 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:10.775546 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.773731 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" Nov 02 17:53:10.775847 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.775790 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxnjm\" (UniqueName: \"kubernetes.io/projected/7ca344ab-a3fb-498c-9bc9-703ea0816597-kube-api-access-hxnjm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:10.775847 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.775823 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:10.775847 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.775840 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btv9k\" (UniqueName: \"kubernetes.io/projected/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-kube-api-access-btv9k\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:10.776031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.775856 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca344ab-a3fb-498c-9bc9-703ea0816597-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:10.776031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:10.775871 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454db7ea-3a85-4bb3-aa88-5f9dd74466ce-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:11.069027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.062549 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/932f3be9-81c1-4473-8b41-46870d0fc7fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/932f3be9-81c1-4473-8b41-46870d0fc7fa-metrics-certs\") pod \"932f3be9-81c1-4473-8b41-46870d0fc7fa\" (UID: \"932f3be9-81c1-4473-8b41-46870d0fc7fa\") " Nov 02 17:53:11.069027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.062715 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932f3be9-81c1-4473-8b41-46870d0fc7fa-kube-api-access-948ql\" (UniqueName: \"kubernetes.io/projected/932f3be9-81c1-4473-8b41-46870d0fc7fa-kube-api-access-948ql\") pod \"932f3be9-81c1-4473-8b41-46870d0fc7fa\" (UID: \"932f3be9-81c1-4473-8b41-46870d0fc7fa\") " Nov 02 17:53:11.069027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.062757 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/932f3be9-81c1-4473-8b41-46870d0fc7fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/932f3be9-81c1-4473-8b41-46870d0fc7fa-openshift-service-ca\") pod \"932f3be9-81c1-4473-8b41-46870d0fc7fa\" (UID: \"932f3be9-81c1-4473-8b41-46870d0fc7fa\") " Nov 02 17:53:11.069027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.062797 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/932f3be9-81c1-4473-8b41-46870d0fc7fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/932f3be9-81c1-4473-8b41-46870d0fc7fa-maas-api-tls\") pod \"932f3be9-81c1-4473-8b41-46870d0fc7fa\" (UID: \"932f3be9-81c1-4473-8b41-46870d0fc7fa\") " Nov 02 17:53:11.107009 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.105898 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:11.107009 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.106462 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/932f3be9-81c1-4473-8b41-46870d0fc7fa-metrics-certs" pod "932f3be9-81c1-4473-8b41-46870d0fc7fa" (UID: "932f3be9-81c1-4473-8b41-46870d0fc7fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:11.107319 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.107288 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932f3be9-81c1-4473-8b41-46870d0fc7fa-kube-api-access-948ql" pod "932f3be9-81c1-4473-8b41-46870d0fc7fa" (UID: "932f3be9-81c1-4473-8b41-46870d0fc7fa"). InnerVolumeSpecName "kube-api-access-948ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:11.107395 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.107355 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932f3be9-81c1-4473-8b41-46870d0fc7fa-openshift-service-ca" pod "932f3be9-81c1-4473-8b41-46870d0fc7fa" (UID: "932f3be9-81c1-4473-8b41-46870d0fc7fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:11.112214 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.112180 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc"] Nov 02 17:53:11.117201 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.117166 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/932f3be9-81c1-4473-8b41-46870d0fc7fa-maas-api-tls" pod "932f3be9-81c1-4473-8b41-46870d0fc7fa" (UID: "932f3be9-81c1-4473-8b41-46870d0fc7fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:11.164282 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.164249 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932f3be9-81c1-4473-8b41-46870d0fc7fa-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:11.164282 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.164277 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-948ql\" (UniqueName: \"kubernetes.io/projected/932f3be9-81c1-4473-8b41-46870d0fc7fa-kube-api-access-948ql\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:11.164531 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.164292 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/932f3be9-81c1-4473-8b41-46870d0fc7fa-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:11.164531 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.164301 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/932f3be9-81c1-4473-8b41-46870d0fc7fa-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:11.185349 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.185298 5549 generic.go:356] "Generic (PLEG): container finished" podID="932f3be9-81c1-4473-8b41-46870d0fc7fa" containerID="c50e625665d23d90366261dc7016c64c6a4f9ac0a748ee82965ab2b58cb9fdb6" exitCode=0 Nov 02 17:53:11.185349 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.185344 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" Nov 02 17:53:11.185578 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.185430 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" event={"ID":"932f3be9-81c1-4473-8b41-46870d0fc7fa","Type":"ContainerDied","Data":"c50e625665d23d90366261dc7016c64c6a4f9ac0a748ee82965ab2b58cb9fdb6"} Nov 02 17:53:11.185578 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.185490 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p" event={"ID":"932f3be9-81c1-4473-8b41-46870d0fc7fa","Type":"ContainerDied","Data":"0f78348fa70a5379742761daf8204e48a1f101e70859361e715afef89a371d6e"} Nov 02 17:53:11.185578 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.185525 5549 scope.go:122] "RemoveContainer" containerID="c50e625665d23d90366261dc7016c64c6a4f9ac0a748ee82965ab2b58cb9fdb6" Nov 02 17:53:11.189607 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.189467 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc" event={"ID":"6f268007-ed65-4c71-a091-9a8fa16f4e43","Type":"ContainerStarted","Data":"be4bea94eaddc3c977e7866e341536d73de5d182370dea065a309e6431d17f95"} Nov 02 17:53:11.196251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.194042 5549 generic.go:356] "Generic (PLEG): container finished" podID="454db7ea-3a85-4bb3-aa88-5f9dd74466ce" containerID="b71c12ac3236aaf9fd5526150bdecfd3588901233ee66377a7a842cabe86df3e" exitCode=0 Nov 02 17:53:11.196251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.194131 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" event={"ID":"454db7ea-3a85-4bb3-aa88-5f9dd74466ce","Type":"ContainerDied","Data":"b71c12ac3236aaf9fd5526150bdecfd3588901233ee66377a7a842cabe86df3e"} Nov 02 17:53:11.196251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.194184 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" event={"ID":"454db7ea-3a85-4bb3-aa88-5f9dd74466ce","Type":"ContainerDied","Data":"13aa0369c7ccd96b7154152298a8531dcc57effaad04f7c2687938d521099eb6"} Nov 02 17:53:11.196251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.194262 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv" Nov 02 17:53:11.197789 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.197724 5549 generic.go:356] "Generic (PLEG): container finished" podID="7ca344ab-a3fb-498c-9bc9-703ea0816597" containerID="ae8496ae029403bd64bcc2340a77acd8d868e7aa26db2829c24eb9e6304888f1" exitCode=0 Nov 02 17:53:11.197891 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.197793 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5" event={"ID":"7ca344ab-a3fb-498c-9bc9-703ea0816597","Type":"ContainerDied","Data":"ae8496ae029403bd64bcc2340a77acd8d868e7aa26db2829c24eb9e6304888f1"} Nov 02 17:53:11.197891 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.197849 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5" event={"ID":"7ca344ab-a3fb-498c-9bc9-703ea0816597","Type":"ContainerDied","Data":"4c296f9b4fe9fb2f7c598d6d9adc9172e2df5665dc2fa04777febec2cab3416f"} Nov 02 17:53:11.198405 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.198124 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5" Nov 02 17:53:11.216467 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.216264 5549 scope.go:122] "RemoveContainer" containerID="c50e625665d23d90366261dc7016c64c6a4f9ac0a748ee82965ab2b58cb9fdb6" Nov 02 17:53:11.216827 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:53:11.216795 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c50e625665d23d90366261dc7016c64c6a4f9ac0a748ee82965ab2b58cb9fdb6\": container with ID starting with c50e625665d23d90366261dc7016c64c6a4f9ac0a748ee82965ab2b58cb9fdb6 not found: ID does not exist" containerID="c50e625665d23d90366261dc7016c64c6a4f9ac0a748ee82965ab2b58cb9fdb6" Nov 02 17:53:11.216900 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.216853 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c50e625665d23d90366261dc7016c64c6a4f9ac0a748ee82965ab2b58cb9fdb6"} err="failed to get container status \"c50e625665d23d90366261dc7016c64c6a4f9ac0a748ee82965ab2b58cb9fdb6\": rpc error: code = NotFound desc = could not find container \"c50e625665d23d90366261dc7016c64c6a4f9ac0a748ee82965ab2b58cb9fdb6\": container with ID starting with c50e625665d23d90366261dc7016c64c6a4f9ac0a748ee82965ab2b58cb9fdb6 not found: ID does not exist" Nov 02 17:53:11.216980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.216880 5549 scope.go:122] "RemoveContainer" containerID="b71c12ac3236aaf9fd5526150bdecfd3588901233ee66377a7a842cabe86df3e" Nov 02 17:53:11.218466 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.218091 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn"] Nov 02 17:53:11.223346 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:53:11.223316 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda93fd4f_c062_43b0_add5_167738bd7489.slice/crio-09d7fb3979c2b99c07e863492446813a9ff4ae7a97bbac118841b83a477ae79d WatchSource:0}: Error finding container 09d7fb3979c2b99c07e863492446813a9ff4ae7a97bbac118841b83a477ae79d: Status 404 returned error can't find the container with id 09d7fb3979c2b99c07e863492446813a9ff4ae7a97bbac118841b83a477ae79d Nov 02 17:53:11.248298 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.248261 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p"] Nov 02 17:53:11.252308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.252218 5549 scope.go:122] "RemoveContainer" containerID="b71c12ac3236aaf9fd5526150bdecfd3588901233ee66377a7a842cabe86df3e" Nov 02 17:53:11.264289 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.264245 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-009a5f-69cf75f4f4-9j75p"] Nov 02 17:53:11.267060 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:53:11.267027 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71c12ac3236aaf9fd5526150bdecfd3588901233ee66377a7a842cabe86df3e\": container with ID starting with b71c12ac3236aaf9fd5526150bdecfd3588901233ee66377a7a842cabe86df3e not found: ID does not exist" containerID="b71c12ac3236aaf9fd5526150bdecfd3588901233ee66377a7a842cabe86df3e" Nov 02 17:53:11.267251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.267224 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71c12ac3236aaf9fd5526150bdecfd3588901233ee66377a7a842cabe86df3e"} err="failed to get container status \"b71c12ac3236aaf9fd5526150bdecfd3588901233ee66377a7a842cabe86df3e\": rpc error: code = NotFound desc = could not find container \"b71c12ac3236aaf9fd5526150bdecfd3588901233ee66377a7a842cabe86df3e\": container with ID starting with b71c12ac3236aaf9fd5526150bdecfd3588901233ee66377a7a842cabe86df3e not found: ID does not exist" Nov 02 17:53:11.267341 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.267330 5549 scope.go:122] "RemoveContainer" containerID="ae8496ae029403bd64bcc2340a77acd8d868e7aa26db2829c24eb9e6304888f1" Nov 02 17:53:11.274936 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.274909 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv"] Nov 02 17:53:11.286745 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.286579 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8f7957-76c85bcc99-qhhnv"] Nov 02 17:53:11.288195 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.288142 5549 scope.go:122] "RemoveContainer" containerID="ae8496ae029403bd64bcc2340a77acd8d868e7aa26db2829c24eb9e6304888f1" Nov 02 17:53:11.288731 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:53:11.288704 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae8496ae029403bd64bcc2340a77acd8d868e7aa26db2829c24eb9e6304888f1\": container with ID starting with ae8496ae029403bd64bcc2340a77acd8d868e7aa26db2829c24eb9e6304888f1 not found: ID does not exist" containerID="ae8496ae029403bd64bcc2340a77acd8d868e7aa26db2829c24eb9e6304888f1" Nov 02 17:53:11.288862 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.288744 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8496ae029403bd64bcc2340a77acd8d868e7aa26db2829c24eb9e6304888f1"} err="failed to get container status \"ae8496ae029403bd64bcc2340a77acd8d868e7aa26db2829c24eb9e6304888f1\": rpc error: code = NotFound desc = could not find container \"ae8496ae029403bd64bcc2340a77acd8d868e7aa26db2829c24eb9e6304888f1\": container with ID starting with ae8496ae029403bd64bcc2340a77acd8d868e7aa26db2829c24eb9e6304888f1 not found: ID does not exist" Nov 02 17:53:11.294463 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.294436 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5"] Nov 02 17:53:11.298584 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.298561 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f7957-7c7cbff587-whkb5"] Nov 02 17:53:11.365121 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.365084 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:11.684047 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:11.667060 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:12.044184 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:12.044149 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:12.223641 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:12.223594 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc" event={"ID":"6f268007-ed65-4c71-a091-9a8fa16f4e43","Type":"ContainerStarted","Data":"57d70d71a596b3f3f18654bf0b2d482370045920ca9caac15a408fb0719f257e"} Nov 02 17:53:12.223884 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:12.223853 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc" Nov 02 17:53:12.225548 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:12.225518 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn" event={"ID":"da93fd4f-c062-43b0-add5-167738bd7489","Type":"ContainerStarted","Data":"2f3d129e8faf68e3e280507b71bd2ed741352e0bba1a43d69bf866ce5afa6b56"} Nov 02 17:53:12.225648 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:12.225565 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn" event={"ID":"da93fd4f-c062-43b0-add5-167738bd7489","Type":"ContainerStarted","Data":"09d7fb3979c2b99c07e863492446813a9ff4ae7a97bbac118841b83a477ae79d"} Nov 02 17:53:12.225648 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:12.225611 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn" Nov 02 17:53:12.247161 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:12.247107 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc" podStartSLOduration=2.2470946290000002 podStartE2EDuration="2.247094629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:12.24313543 +0000 UTC m=+1164.031640955" watchObservedRunningTime="2026-11-02 17:53:12.247094629 +0000 UTC m=+1164.035600180" Nov 02 17:53:12.264021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:12.263928 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn" podStartSLOduration=2.263913348 podStartE2EDuration="2.263913348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:12.263412963 +0000 UTC m=+1164.051918490" watchObservedRunningTime="2026-11-02 17:53:12.263913348 +0000 UTC m=+1164.052418876" Nov 02 17:53:12.264441 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:12.264411 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:12.323919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:12.323841 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="454db7ea-3a85-4bb3-aa88-5f9dd74466ce" path="/var/lib/kubelet/pods/454db7ea-3a85-4bb3-aa88-5f9dd74466ce/volumes" Nov 02 17:53:12.324424 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:12.324403 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca344ab-a3fb-498c-9bc9-703ea0816597" path="/var/lib/kubelet/pods/7ca344ab-a3fb-498c-9bc9-703ea0816597/volumes" Nov 02 17:53:12.324857 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:12.324843 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="932f3be9-81c1-4473-8b41-46870d0fc7fa" path="/var/lib/kubelet/pods/932f3be9-81c1-4473-8b41-46870d0fc7fa/volumes" Nov 02 17:53:12.678039 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:12.677456 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:12.895254 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:12.895213 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:13.843137 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:13.843078 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:14.906824 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:14.906789 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:16.236167 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:16.236133 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:16.911131 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:16.911041 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:17.142418 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.142387 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l" Nov 02 17:53:17.612465 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:53:17.612428 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:53:17.726928 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.726897 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf"] Nov 02 17:53:17.727996 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.727966 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7ca344ab-a3fb-498c-9bc9-703ea0816597" containerName="payload-pre-processing" Nov 02 17:53:17.728189 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.728171 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca344ab-a3fb-498c-9bc9-703ea0816597" containerName="payload-pre-processing" Nov 02 17:53:17.728327 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.728314 5549 cpu_manager.go:414] "containerMap: removing container" podUID="454db7ea-3a85-4bb3-aa88-5f9dd74466ce" containerName="maas-api" Nov 02 17:53:17.728428 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.728415 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="454db7ea-3a85-4bb3-aa88-5f9dd74466ce" containerName="maas-api" Nov 02 17:53:17.728539 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.728527 5549 cpu_manager.go:414] "containerMap: removing container" podUID="932f3be9-81c1-4473-8b41-46870d0fc7fa" containerName="maas-api" Nov 02 17:53:17.728628 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.728615 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932f3be9-81c1-4473-8b41-46870d0fc7fa" containerName="maas-api" Nov 02 17:53:17.728898 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.728880 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="454db7ea-3a85-4bb3-aa88-5f9dd74466ce" containerName="maas-api" Nov 02 17:53:17.729034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.729020 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="932f3be9-81c1-4473-8b41-46870d0fc7fa" containerName="maas-api" Nov 02 17:53:17.729149 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.729138 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca344ab-a3fb-498c-9bc9-703ea0816597" containerName="payload-pre-processing" Nov 02 17:53:17.735348 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.735330 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:53:17.747692 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.746487 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-6f78de-serving-cert\"" Nov 02 17:53:17.750814 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.750785 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf"] Nov 02 17:53:17.834909 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.833581 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2blgp\" (UniqueName: \"kubernetes.io/projected/dd40957c-0e44-445b-8274-afa6ba043fc5-kube-api-access-2blgp\") pod \"maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf\" (UID: \"dd40957c-0e44-445b-8274-afa6ba043fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:53:17.834909 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.833751 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd40957c-0e44-445b-8274-afa6ba043fc5-metrics-certs\") pod \"maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf\" (UID: \"dd40957c-0e44-445b-8274-afa6ba043fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:53:17.834909 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.833817 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd40957c-0e44-445b-8274-afa6ba043fc5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf\" (UID: \"dd40957c-0e44-445b-8274-afa6ba043fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:53:17.834909 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.833872 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd40957c-0e44-445b-8274-afa6ba043fc5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf\" (UID: \"dd40957c-0e44-445b-8274-afa6ba043fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:53:17.898104 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.894731 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:17.951985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.935874 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd40957c-0e44-445b-8274-afa6ba043fc5-metrics-certs\") pod \"maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf\" (UID: \"dd40957c-0e44-445b-8274-afa6ba043fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:53:17.952309 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.952279 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd40957c-0e44-445b-8274-afa6ba043fc5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf\" (UID: \"dd40957c-0e44-445b-8274-afa6ba043fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:53:17.952525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.952505 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd40957c-0e44-445b-8274-afa6ba043fc5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf\" (UID: \"dd40957c-0e44-445b-8274-afa6ba043fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:53:17.952711 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.952694 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2blgp\" (UniqueName: \"kubernetes.io/projected/dd40957c-0e44-445b-8274-afa6ba043fc5-kube-api-access-2blgp\") pod \"maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf\" (UID: \"dd40957c-0e44-445b-8274-afa6ba043fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:53:17.954111 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.954083 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd40957c-0e44-445b-8274-afa6ba043fc5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf\" (UID: \"dd40957c-0e44-445b-8274-afa6ba043fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:53:17.954335 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:53:17.954320 5549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-serving-cert: secret "maas-api-e2e-praxis-b-6f78de-serving-cert" not found Nov 02 17:53:17.954465 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:53:17.954456 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd40957c-0e44-445b-8274-afa6ba043fc5-maas-api-tls podName:dd40957c-0e44-445b-8274-afa6ba043fc5 nodeName:}" failed. No retries permitted until 2026-11-02 17:53:18.454435878 +0000 UTC m=+1170.242941389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dd40957c-0e44-445b-8274-afa6ba043fc5-maas-api-tls") pod "maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" (UID: "dd40957c-0e44-445b-8274-afa6ba043fc5") : secret "maas-api-e2e-praxis-b-6f78de-serving-cert" not found Nov 02 17:53:17.988020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:17.984278 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd40957c-0e44-445b-8274-afa6ba043fc5-metrics-certs\") pod \"maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf\" (UID: \"dd40957c-0e44-445b-8274-afa6ba043fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:53:18.021016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:18.019645 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2blgp\" (UniqueName: \"kubernetes.io/projected/dd40957c-0e44-445b-8274-afa6ba043fc5-kube-api-access-2blgp\") pod \"maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf\" (UID: \"dd40957c-0e44-445b-8274-afa6ba043fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:53:18.171620 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:18.171502 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:18.400978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:18.384526 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:18.462613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:18.462563 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd40957c-0e44-445b-8274-afa6ba043fc5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf\" (UID: \"dd40957c-0e44-445b-8274-afa6ba043fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:53:18.465145 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:18.465121 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd40957c-0e44-445b-8274-afa6ba043fc5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf\" (UID: \"dd40957c-0e44-445b-8274-afa6ba043fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:53:18.659178 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:18.657836 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:53:18.953025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:18.950366 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:19.047698 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:19.047670 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf"] Nov 02 17:53:19.048834 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:53:19.048802 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd40957c_0e44_445b_8274_afa6ba043fc5.slice/crio-b3e404084617bd4a74983c7780b244063c5ddc07257ef87cd58a0764adb15448 WatchSource:0}: Error finding container b3e404084617bd4a74983c7780b244063c5ddc07257ef87cd58a0764adb15448: Status 404 returned error can't find the container with id b3e404084617bd4a74983c7780b244063c5ddc07257ef87cd58a0764adb15448 Nov 02 17:53:19.355028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:19.349665 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" event={"ID":"dd40957c-0e44-445b-8274-afa6ba043fc5","Type":"ContainerStarted","Data":"b3e404084617bd4a74983c7780b244063c5ddc07257ef87cd58a0764adb15448"} Nov 02 17:53:19.376241 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:19.376202 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:19.807075 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:19.807038 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:20.110841 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:20.110717 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:20.218434 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:20.218396 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46"] Nov 02 17:53:20.219045 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:20.218967 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" podUID="36f413ea-e51a-4d63-a07f-0f1a701e6357" containerName="istio-proxy" containerID="cri-o://b1517fb466caa9b047f2e65911c9c24762e79ca7305a99d324c2658da080d820" gracePeriod=30 Nov 02 17:53:20.432966 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:20.432865 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:20.452050 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:20.451499 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-5hb2w_b693ebb5-876a-426b-bf1d-6fdf00d3e96a/payload-processing/0.log" Nov 02 17:53:20.470446 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:20.469146 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" event={"ID":"dd40957c-0e44-445b-8274-afa6ba043fc5","Type":"ContainerStarted","Data":"904d9019cfb836a3cc1505dd8769b8cea124b90207ab934f445206c8ffd35a11"} Nov 02 17:53:20.470446 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:20.470318 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:53:20.728490 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:20.728457 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-5hb2w_b693ebb5-876a-426b-bf1d-6fdf00d3e96a/payload-processing/0.log" Nov 02 17:53:20.905721 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:20.905691 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l_f0986e16-44a0-471e-9467-5373643adae6/payload-processing/0.log" Nov 02 17:53:21.345097 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:21.341587 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:21.628771 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:21.628683 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:21.847843 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:21.847799 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:22.139942 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:22.139903 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:22.428032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:22.414784 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:22.621527 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:22.621493 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:23.249068 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:23.249033 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc" Nov 02 17:53:23.256367 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:23.256338 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn" Nov 02 17:53:23.263769 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:23.263740 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:23.285715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:23.285641 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" podStartSLOduration=5.617975372 podStartE2EDuration="6.28562509s" totalImagesPullingTime="667.649718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:20.511229928 +0000 UTC m=+1172.299735460" watchObservedRunningTime="2026-11-02 17:53:23.28562509 +0000 UTC m=+1175.074130618" Nov 02 17:53:23.326464 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:23.326427 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:53:24.353762 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:24.350055 5549 patch_prober.go:29] interesting pod/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": read tcp 10.133.0.2:35446->10.133.0.143:15021: read: connection reset by peer" start-of-body= Nov 02 17:53:24.353762 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:24.350115 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" podUID="c9f78f64-e109-490b-a5ae-4fcec9644276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": read tcp 10.133.0.2:35446->10.133.0.143:15021: read: connection reset by peer" Nov 02 17:53:24.516588 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:24.516508 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:24.610991 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:24.610896 5549 generic.go:356] "Generic (PLEG): container finished" podID="c9f78f64-e109-490b-a5ae-4fcec9644276" containerID="84931c92ab04e142b5e666ece6dac8e8ff0cfb0155929d5c4f286fdcf0cda1cb" exitCode=0 Nov 02 17:53:24.610991 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:24.610980 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" event={"ID":"c9f78f64-e109-490b-a5ae-4fcec9644276","Type":"ContainerDied","Data":"84931c92ab04e142b5e666ece6dac8e8ff0cfb0155929d5c4f286fdcf0cda1cb"} Nov 02 17:53:24.611438 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:24.611423 5549 scope.go:122] "RemoveContainer" containerID="84931c92ab04e142b5e666ece6dac8e8ff0cfb0155929d5c4f286fdcf0cda1cb" Nov 02 17:53:24.612938 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:24.612909 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:53:24.613022 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:24.612990 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:53:24.613064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:24.613025 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:53:25.540568 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.540541 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:53:25.610284 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.610244 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f413ea-e51a-4d63-a07f-0f1a701e6357-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f413ea-e51a-4d63-a07f-0f1a701e6357-istiod-ca-cert\") pod \"36f413ea-e51a-4d63-a07f-0f1a701e6357\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " Nov 02 17:53:25.610487 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.610393 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-envoy\") pod \"36f413ea-e51a-4d63-a07f-0f1a701e6357\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " Nov 02 17:53:25.610487 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.610425 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-token\" (UniqueName: \"kubernetes.io/projected/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-token\") pod \"36f413ea-e51a-4d63-a07f-0f1a701e6357\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " Nov 02 17:53:25.610603 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.610497 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-credential-socket\") pod \"36f413ea-e51a-4d63-a07f-0f1a701e6357\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " Nov 02 17:53:25.610603 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.610535 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-data\") pod \"36f413ea-e51a-4d63-a07f-0f1a701e6357\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " Nov 02 17:53:25.610709 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.610618 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f413ea-e51a-4d63-a07f-0f1a701e6357-kube-api-access-mhnc6\" (UniqueName: \"kubernetes.io/projected/36f413ea-e51a-4d63-a07f-0f1a701e6357-kube-api-access-mhnc6\") pod \"36f413ea-e51a-4d63-a07f-0f1a701e6357\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " Nov 02 17:53:25.610762 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.610721 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-podinfo\") pod \"36f413ea-e51a-4d63-a07f-0f1a701e6357\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " Nov 02 17:53:25.610817 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.610793 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-workload-certs\") pod \"36f413ea-e51a-4d63-a07f-0f1a701e6357\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " Nov 02 17:53:25.610884 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.610822 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-workload-socket\") pod \"36f413ea-e51a-4d63-a07f-0f1a701e6357\" (UID: \"36f413ea-e51a-4d63-a07f-0f1a701e6357\") " Nov 02 17:53:25.611442 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.611239 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-credential-socket" pod "36f413ea-e51a-4d63-a07f-0f1a701e6357" (UID: "36f413ea-e51a-4d63-a07f-0f1a701e6357"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:25.611565 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.611537 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:25.611854 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.611830 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f413ea-e51a-4d63-a07f-0f1a701e6357-istiod-ca-cert" pod "36f413ea-e51a-4d63-a07f-0f1a701e6357" (UID: "36f413ea-e51a-4d63-a07f-0f1a701e6357"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:25.612164 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.612138 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-data" pod "36f413ea-e51a-4d63-a07f-0f1a701e6357" (UID: "36f413ea-e51a-4d63-a07f-0f1a701e6357"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:25.613330 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.612824 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-workload-socket" pod "36f413ea-e51a-4d63-a07f-0f1a701e6357" (UID: "36f413ea-e51a-4d63-a07f-0f1a701e6357"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:25.613330 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.613098 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-workload-certs" pod "36f413ea-e51a-4d63-a07f-0f1a701e6357" (UID: "36f413ea-e51a-4d63-a07f-0f1a701e6357"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:25.615743 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.614544 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-envoy" pod "36f413ea-e51a-4d63-a07f-0f1a701e6357" (UID: "36f413ea-e51a-4d63-a07f-0f1a701e6357"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:25.619470 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.619418 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-podinfo" pod "36f413ea-e51a-4d63-a07f-0f1a701e6357" (UID: "36f413ea-e51a-4d63-a07f-0f1a701e6357"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:53:25.619470 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.619461 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f413ea-e51a-4d63-a07f-0f1a701e6357-kube-api-access-mhnc6" pod "36f413ea-e51a-4d63-a07f-0f1a701e6357" (UID: "36f413ea-e51a-4d63-a07f-0f1a701e6357"). InnerVolumeSpecName "kube-api-access-mhnc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:25.621152 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.621125 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-token" pod "36f413ea-e51a-4d63-a07f-0f1a701e6357" (UID: "36f413ea-e51a-4d63-a07f-0f1a701e6357"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:25.623220 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.623197 5549 generic.go:356] "Generic (PLEG): container finished" podID="36f413ea-e51a-4d63-a07f-0f1a701e6357" containerID="b1517fb466caa9b047f2e65911c9c24762e79ca7305a99d324c2658da080d820" exitCode=0 Nov 02 17:53:25.623310 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.623262 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" Nov 02 17:53:25.623310 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.623277 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" event={"ID":"36f413ea-e51a-4d63-a07f-0f1a701e6357","Type":"ContainerDied","Data":"b1517fb466caa9b047f2e65911c9c24762e79ca7305a99d324c2658da080d820"} Nov 02 17:53:25.623431 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.623325 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46" event={"ID":"36f413ea-e51a-4d63-a07f-0f1a701e6357","Type":"ContainerDied","Data":"dcb56d396cfad917a59f3cb78faaf194f2a7e25c8f4916317cb8cf863296709c"} Nov 02 17:53:25.623431 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.623346 5549 scope.go:122] "RemoveContainer" containerID="b1517fb466caa9b047f2e65911c9c24762e79ca7305a99d324c2658da080d820" Nov 02 17:53:25.626454 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.626424 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" event={"ID":"c9f78f64-e109-490b-a5ae-4fcec9644276","Type":"ContainerStarted","Data":"c4f33b887ecfa6e06a68d6b1fda29f8fae035855e414f8f78435a9fb6980610c"} Nov 02 17:53:25.659139 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.658594 5549 scope.go:122] "RemoveContainer" containerID="a0fa576d446c3351778e82c3e6ad8b2fed1c7bc88ec3fe03e7a3ba89bd055e7c" Nov 02 17:53:25.670016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.669938 5549 scope.go:122] "RemoveContainer" containerID="b1517fb466caa9b047f2e65911c9c24762e79ca7305a99d324c2658da080d820" Nov 02 17:53:25.670356 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:53:25.670301 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1517fb466caa9b047f2e65911c9c24762e79ca7305a99d324c2658da080d820\": container with ID starting with b1517fb466caa9b047f2e65911c9c24762e79ca7305a99d324c2658da080d820 not found: ID does not exist" containerID="b1517fb466caa9b047f2e65911c9c24762e79ca7305a99d324c2658da080d820" Nov 02 17:53:25.670356 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.670331 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1517fb466caa9b047f2e65911c9c24762e79ca7305a99d324c2658da080d820"} err="failed to get container status \"b1517fb466caa9b047f2e65911c9c24762e79ca7305a99d324c2658da080d820\": rpc error: code = NotFound desc = could not find container \"b1517fb466caa9b047f2e65911c9c24762e79ca7305a99d324c2658da080d820\": container with ID starting with b1517fb466caa9b047f2e65911c9c24762e79ca7305a99d324c2658da080d820 not found: ID does not exist" Nov 02 17:53:25.670356 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.670352 5549 scope.go:122] "RemoveContainer" containerID="a0fa576d446c3351778e82c3e6ad8b2fed1c7bc88ec3fe03e7a3ba89bd055e7c" Nov 02 17:53:25.670713 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:53:25.670676 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0fa576d446c3351778e82c3e6ad8b2fed1c7bc88ec3fe03e7a3ba89bd055e7c\": container with ID starting with a0fa576d446c3351778e82c3e6ad8b2fed1c7bc88ec3fe03e7a3ba89bd055e7c not found: ID does not exist" containerID="a0fa576d446c3351778e82c3e6ad8b2fed1c7bc88ec3fe03e7a3ba89bd055e7c" Nov 02 17:53:25.670830 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.670710 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0fa576d446c3351778e82c3e6ad8b2fed1c7bc88ec3fe03e7a3ba89bd055e7c"} err="failed to get container status \"a0fa576d446c3351778e82c3e6ad8b2fed1c7bc88ec3fe03e7a3ba89bd055e7c\": rpc error: code = NotFound desc = could not find container \"a0fa576d446c3351778e82c3e6ad8b2fed1c7bc88ec3fe03e7a3ba89bd055e7c\": container with ID starting with a0fa576d446c3351778e82c3e6ad8b2fed1c7bc88ec3fe03e7a3ba89bd055e7c not found: ID does not exist" Nov 02 17:53:25.712515 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.712478 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:25.712515 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.712514 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:25.712751 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.712531 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:25.712751 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.712548 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhnc6\" (UniqueName: \"kubernetes.io/projected/36f413ea-e51a-4d63-a07f-0f1a701e6357-kube-api-access-mhnc6\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:25.712751 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.712565 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f413ea-e51a-4d63-a07f-0f1a701e6357-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:25.712751 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.712580 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:25.712751 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.712594 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f413ea-e51a-4d63-a07f-0f1a701e6357-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:25.712751 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.712609 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f413ea-e51a-4d63-a07f-0f1a701e6357-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:25.713003 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.712974 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46"] Nov 02 17:53:25.724743 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.724712 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f7957-openshift-default-8dc7996fc-bxs46"] Nov 02 17:53:25.954745 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.954157 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:53:25.954745 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.954297 5549 patch_prober.go:29] interesting pod/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 17:53:25.954745 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:25.954342 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" podUID="c9f78f64-e109-490b-a5ae-4fcec9644276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 17:53:26.227552 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:26.227407 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:26.320719 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:26.320688 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f413ea-e51a-4d63-a07f-0f1a701e6357" path="/var/lib/kubelet/pods/36f413ea-e51a-4d63-a07f-0f1a701e6357/volumes" Nov 02 17:53:26.535096 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:26.534995 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:26.954632 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:26.954600 5549 patch_prober.go:29] interesting pod/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 17:53:26.955082 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:26.954655 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" podUID="c9f78f64-e109-490b-a5ae-4fcec9644276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 17:53:27.304452 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.304353 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:27.611650 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.611574 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:53:27.711096 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.711060 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8"] Nov 02 17:53:27.712338 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.712308 5549 cpu_manager.go:414] "containerMap: removing container" podUID="36f413ea-e51a-4d63-a07f-0f1a701e6357" containerName="istio-proxy" Nov 02 17:53:27.712519 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.712503 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f413ea-e51a-4d63-a07f-0f1a701e6357" containerName="istio-proxy" Nov 02 17:53:27.712677 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.712662 5549 cpu_manager.go:414] "containerMap: removing container" podUID="36f413ea-e51a-4d63-a07f-0f1a701e6357" containerName="istio-proxy" Nov 02 17:53:27.712807 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.712792 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f413ea-e51a-4d63-a07f-0f1a701e6357" containerName="istio-proxy" Nov 02 17:53:27.713130 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.713110 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f413ea-e51a-4d63-a07f-0f1a701e6357" containerName="istio-proxy" Nov 02 17:53:27.713258 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.713245 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f413ea-e51a-4d63-a07f-0f1a701e6357" containerName="istio-proxy" Nov 02 17:53:27.719243 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.719221 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8" Nov 02 17:53:27.728558 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.728529 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8"] Nov 02 17:53:27.743968 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.743923 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-4e6649\"" Nov 02 17:53:27.763363 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.763334 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df"] Nov 02 17:53:27.765238 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.765211 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd8zx\" (UniqueName: \"kubernetes.io/projected/e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd-kube-api-access-vd8zx\") pod \"payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8\" (UID: \"e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8" Nov 02 17:53:27.766808 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.766779 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8\" (UID: \"e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8" Nov 02 17:53:27.767514 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.767454 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:27.771161 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.771135 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df" Nov 02 17:53:27.797257 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.781715 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df"] Nov 02 17:53:27.807254 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.802301 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-4e6649-dockercfg-62xjw\"" Nov 02 17:53:27.871410 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.871336 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8\" (UID: \"e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8" Nov 02 17:53:27.871410 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.871379 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe57717-8ea3-4d44-90bd-13e943adfa80-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df\" (UID: \"bfe57717-8ea3-4d44-90bd-13e943adfa80\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df" Nov 02 17:53:27.871595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.871454 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rggqf\" (UniqueName: \"kubernetes.io/projected/bfe57717-8ea3-4d44-90bd-13e943adfa80-kube-api-access-rggqf\") pod \"payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df\" (UID: \"bfe57717-8ea3-4d44-90bd-13e943adfa80\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df" Nov 02 17:53:27.871735 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.871715 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd8zx\" (UniqueName: \"kubernetes.io/projected/e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd-kube-api-access-vd8zx\") pod \"payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8\" (UID: \"e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8" Nov 02 17:53:27.872145 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.872127 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8\" (UID: \"e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8" Nov 02 17:53:27.906990 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.906920 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd8zx\" (UniqueName: \"kubernetes.io/projected/e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd-kube-api-access-vd8zx\") pod \"payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8\" (UID: \"e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8" Nov 02 17:53:27.957237 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.957198 5549 patch_prober.go:29] interesting pod/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 17:53:27.957793 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.957757 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" podUID="c9f78f64-e109-490b-a5ae-4fcec9644276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 17:53:27.996979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.974782 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe57717-8ea3-4d44-90bd-13e943adfa80-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df\" (UID: \"bfe57717-8ea3-4d44-90bd-13e943adfa80\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df" Nov 02 17:53:27.997398 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.997359 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rggqf\" (UniqueName: \"kubernetes.io/projected/bfe57717-8ea3-4d44-90bd-13e943adfa80-kube-api-access-rggqf\") pod \"payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df\" (UID: \"bfe57717-8ea3-4d44-90bd-13e943adfa80\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df" Nov 02 17:53:27.998173 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:27.979684 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe57717-8ea3-4d44-90bd-13e943adfa80-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df\" (UID: \"bfe57717-8ea3-4d44-90bd-13e943adfa80\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df" Nov 02 17:53:28.017982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:28.015873 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:28.017982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:28.016836 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rggqf\" (UniqueName: \"kubernetes.io/projected/bfe57717-8ea3-4d44-90bd-13e943adfa80-kube-api-access-rggqf\") pod \"payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df\" (UID: \"bfe57717-8ea3-4d44-90bd-13e943adfa80\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df" Nov 02 17:53:28.049215 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:28.049184 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8" Nov 02 17:53:28.125078 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:28.118251 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df" Nov 02 17:53:28.513045 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:28.512974 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:28.750250 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:53:28.750218 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d0274f_2eb1_4c72_8140_ae9e8b8d22cd.slice/crio-818c41b99e9f97b004eefca7e227c7095d2d1b2cdd003f5aaae4aef79c0546f9 WatchSource:0}: Error finding container 818c41b99e9f97b004eefca7e227c7095d2d1b2cdd003f5aaae4aef79c0546f9: Status 404 returned error can't find the container with id 818c41b99e9f97b004eefca7e227c7095d2d1b2cdd003f5aaae4aef79c0546f9 Nov 02 17:53:28.750442 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:28.750417 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8"] Nov 02 17:53:28.809486 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:53:28.809456 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe57717_8ea3_4d44_90bd_13e943adfa80.slice/crio-abb174c0c7d74eb87a11b90e5bfd2169d8c152ec9ee75d62a8f6e493d968f7b7 WatchSource:0}: Error finding container abb174c0c7d74eb87a11b90e5bfd2169d8c152ec9ee75d62a8f6e493d968f7b7: Status 404 returned error can't find the container with id abb174c0c7d74eb87a11b90e5bfd2169d8c152ec9ee75d62a8f6e493d968f7b7 Nov 02 17:53:28.809940 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:28.809919 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df"] Nov 02 17:53:28.954804 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:28.954773 5549 patch_prober.go:29] interesting pod/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 17:53:28.955000 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:28.954820 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" podUID="c9f78f64-e109-490b-a5ae-4fcec9644276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 17:53:29.655727 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:29.638484 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:29.716929 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:29.716881 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df" event={"ID":"bfe57717-8ea3-4d44-90bd-13e943adfa80","Type":"ContainerStarted","Data":"d3e7a0d45096171a3e5ad3417990f09f4a1ff5229e25c063db5509648910eb2a"} Nov 02 17:53:29.717092 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:29.716998 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df" event={"ID":"bfe57717-8ea3-4d44-90bd-13e943adfa80","Type":"ContainerStarted","Data":"abb174c0c7d74eb87a11b90e5bfd2169d8c152ec9ee75d62a8f6e493d968f7b7"} Nov 02 17:53:29.717244 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:29.717223 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df" Nov 02 17:53:29.720947 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:29.720914 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8" event={"ID":"e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd","Type":"ContainerStarted","Data":"9ac07c951ce9dc6e887b28c57e5f8ac7cddd5ab7867dbb574bce0617746034a7"} Nov 02 17:53:29.721113 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:29.720988 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8" event={"ID":"e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd","Type":"ContainerStarted","Data":"818c41b99e9f97b004eefca7e227c7095d2d1b2cdd003f5aaae4aef79c0546f9"} Nov 02 17:53:29.721735 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:29.721711 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8" Nov 02 17:53:29.777023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:29.776974 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df" podStartSLOduration=2.77694481 podStartE2EDuration="2.77694481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:29.773217543 +0000 UTC m=+1181.561723070" watchObservedRunningTime="2026-11-02 17:53:29.77694481 +0000 UTC m=+1181.565450337" Nov 02 17:53:29.822931 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:29.822862 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8" podStartSLOduration=2.822848337 podStartE2EDuration="2.822848337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:29.822646416 +0000 UTC m=+1181.611151936" watchObservedRunningTime="2026-11-02 17:53:29.822848337 +0000 UTC m=+1181.611353865" Nov 02 17:53:29.966976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:29.959449 5549 patch_prober.go:29] interesting pod/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 17:53:29.966976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:29.959521 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" podUID="c9f78f64-e109-490b-a5ae-4fcec9644276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 17:53:30.415896 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:30.415775 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx"] Nov 02 17:53:30.416098 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:30.416070 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx" podUID="fa505463-b683-461b-a1d3-2a0d4cb1e693" containerName="payload-processing" containerID="cri-o://ab8d70086dd5a807c03e1683965178f427c25ade413b7446bf7dbd8861b87745" gracePeriod=30 Nov 02 17:53:30.420686 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:30.420651 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq"] Nov 02 17:53:30.420933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:30.420902 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq" podUID="b63ebb7d-159f-4648-8d9d-a96c608d3799" containerName="payload-pre-processing" containerID="cri-o://3cab779930a20d895e4464537bfe2528955873a6c10c96660bf81518c42455b6" gracePeriod=30 Nov 02 17:53:30.443778 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:30.443745 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:30.643026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:30.632540 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:30.893683 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:30.893410 5549 generic.go:356] "Generic (PLEG): container finished" podID="b63ebb7d-159f-4648-8d9d-a96c608d3799" containerID="3cab779930a20d895e4464537bfe2528955873a6c10c96660bf81518c42455b6" exitCode=0 Nov 02 17:53:30.893683 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:30.893566 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq" event={"ID":"b63ebb7d-159f-4648-8d9d-a96c608d3799","Type":"ContainerDied","Data":"3cab779930a20d895e4464537bfe2528955873a6c10c96660bf81518c42455b6"} Nov 02 17:53:30.899018 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:30.897134 5549 generic.go:356] "Generic (PLEG): container finished" podID="fa505463-b683-461b-a1d3-2a0d4cb1e693" containerID="ab8d70086dd5a807c03e1683965178f427c25ade413b7446bf7dbd8861b87745" exitCode=0 Nov 02 17:53:30.899018 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:30.898187 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx" event={"ID":"fa505463-b683-461b-a1d3-2a0d4cb1e693","Type":"ContainerDied","Data":"ab8d70086dd5a807c03e1683965178f427c25ade413b7446bf7dbd8861b87745"} Nov 02 17:53:30.982303 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:30.982259 5549 patch_prober.go:29] interesting pod/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 17:53:30.982497 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:30.982341 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" podUID="c9f78f64-e109-490b-a5ae-4fcec9644276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 17:53:31.025864 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.011256 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:31.047121 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.046035 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq" Nov 02 17:53:31.137388 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.137364 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx" Nov 02 17:53:31.142706 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.142682 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b63ebb7d-159f-4648-8d9d-a96c608d3799-kube-api-access-b8q9d\" (UniqueName: \"kubernetes.io/projected/b63ebb7d-159f-4648-8d9d-a96c608d3799-kube-api-access-b8q9d\") pod \"b63ebb7d-159f-4648-8d9d-a96c608d3799\" (UID: \"b63ebb7d-159f-4648-8d9d-a96c608d3799\") " Nov 02 17:53:31.142851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.142831 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b63ebb7d-159f-4648-8d9d-a96c608d3799-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b63ebb7d-159f-4648-8d9d-a96c608d3799-plugins-config-volume\") pod \"b63ebb7d-159f-4648-8d9d-a96c608d3799\" (UID: \"b63ebb7d-159f-4648-8d9d-a96c608d3799\") " Nov 02 17:53:31.143254 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.143227 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b63ebb7d-159f-4648-8d9d-a96c608d3799-plugins-config-volume" pod "b63ebb7d-159f-4648-8d9d-a96c608d3799" (UID: "b63ebb7d-159f-4648-8d9d-a96c608d3799"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:31.143476 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.143438 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b63ebb7d-159f-4648-8d9d-a96c608d3799-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:31.144813 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.144767 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b63ebb7d-159f-4648-8d9d-a96c608d3799-kube-api-access-b8q9d" pod "b63ebb7d-159f-4648-8d9d-a96c608d3799" (UID: "b63ebb7d-159f-4648-8d9d-a96c608d3799"). InnerVolumeSpecName "kube-api-access-b8q9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:31.244617 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.244577 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa505463-b683-461b-a1d3-2a0d4cb1e693-kube-api-access-7tztq\" (UniqueName: \"kubernetes.io/projected/fa505463-b683-461b-a1d3-2a0d4cb1e693-kube-api-access-7tztq\") pod \"fa505463-b683-461b-a1d3-2a0d4cb1e693\" (UID: \"fa505463-b683-461b-a1d3-2a0d4cb1e693\") " Nov 02 17:53:31.244617 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.244625 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa505463-b683-461b-a1d3-2a0d4cb1e693-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa505463-b683-461b-a1d3-2a0d4cb1e693-plugins-config-volume\") pod \"fa505463-b683-461b-a1d3-2a0d4cb1e693\" (UID: \"fa505463-b683-461b-a1d3-2a0d4cb1e693\") " Nov 02 17:53:31.245125 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.245097 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa505463-b683-461b-a1d3-2a0d4cb1e693-plugins-config-volume" pod "fa505463-b683-461b-a1d3-2a0d4cb1e693" (UID: "fa505463-b683-461b-a1d3-2a0d4cb1e693"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:31.245254 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.245240 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa505463-b683-461b-a1d3-2a0d4cb1e693-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:31.245294 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.245257 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8q9d\" (UniqueName: \"kubernetes.io/projected/b63ebb7d-159f-4648-8d9d-a96c608d3799-kube-api-access-b8q9d\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:31.246832 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.246809 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa505463-b683-461b-a1d3-2a0d4cb1e693-kube-api-access-7tztq" pod "fa505463-b683-461b-a1d3-2a0d4cb1e693" (UID: "fa505463-b683-461b-a1d3-2a0d4cb1e693"). InnerVolumeSpecName "kube-api-access-7tztq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:31.346404 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.346367 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tztq\" (UniqueName: \"kubernetes.io/projected/fa505463-b683-461b-a1d3-2a0d4cb1e693-kube-api-access-7tztq\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:31.786843 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.786810 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:31.905232 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.905203 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq" Nov 02 17:53:31.905725 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.905208 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq" event={"ID":"b63ebb7d-159f-4648-8d9d-a96c608d3799","Type":"ContainerDied","Data":"95704fc595743149b303afca7b5c8dcebf856c7ae801e3d7a6bdfd6eedaae02e"} Nov 02 17:53:31.905725 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.905359 5549 scope.go:122] "RemoveContainer" containerID="3cab779930a20d895e4464537bfe2528955873a6c10c96660bf81518c42455b6" Nov 02 17:53:31.906811 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.906787 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx" Nov 02 17:53:31.906924 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.906841 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx" event={"ID":"fa505463-b683-461b-a1d3-2a0d4cb1e693","Type":"ContainerDied","Data":"f53204edfda31932e2105baa2573b3d9313cc3ee7d51ecdfc48060bd59926033"} Nov 02 17:53:31.918334 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.917562 5549 scope.go:122] "RemoveContainer" containerID="ab8d70086dd5a807c03e1683965178f427c25ade413b7446bf7dbd8861b87745" Nov 02 17:53:31.932295 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.932267 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq"] Nov 02 17:53:31.937244 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.937219 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-8579456655-vt6dq"] Nov 02 17:53:31.952851 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.952823 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx"] Nov 02 17:53:31.958256 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.958231 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6098d0-54bf87cbc6-snhcx"] Nov 02 17:53:31.959462 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.959442 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:53:31.959825 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.959803 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:53:31.960610 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:31.960590 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:53:32.321882 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:32.321852 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b63ebb7d-159f-4648-8d9d-a96c608d3799" path="/var/lib/kubelet/pods/b63ebb7d-159f-4648-8d9d-a96c608d3799/volumes" Nov 02 17:53:32.322321 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:32.322306 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa505463-b683-461b-a1d3-2a0d4cb1e693" path="/var/lib/kubelet/pods/fa505463-b683-461b-a1d3-2a0d4cb1e693/volumes" Nov 02 17:53:32.333386 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:32.333354 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:32.517532 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:32.517497 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:33.644739 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:33.644706 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:34.239466 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:34.239429 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:34.474136 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:34.474095 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:35.066151 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.066106 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd"] Nov 02 17:53:35.067196 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.067167 5549 cpu_manager.go:414] "containerMap: removing container" podUID="b63ebb7d-159f-4648-8d9d-a96c608d3799" containerName="payload-pre-processing" Nov 02 17:53:35.067196 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.067193 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b63ebb7d-159f-4648-8d9d-a96c608d3799" containerName="payload-pre-processing" Nov 02 17:53:35.067854 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.067269 5549 cpu_manager.go:414] "containerMap: removing container" podUID="fa505463-b683-461b-a1d3-2a0d4cb1e693" containerName="payload-processing" Nov 02 17:53:35.067854 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.067277 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa505463-b683-461b-a1d3-2a0d4cb1e693" containerName="payload-processing" Nov 02 17:53:35.067854 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.067604 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa505463-b683-461b-a1d3-2a0d4cb1e693" containerName="payload-processing" Nov 02 17:53:35.067854 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.067624 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="b63ebb7d-159f-4648-8d9d-a96c608d3799" containerName="payload-pre-processing" Nov 02 17:53:35.072818 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.072796 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" Nov 02 17:53:35.082994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.080124 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd"] Nov 02 17:53:35.084218 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.084192 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-4e6649-serving-cert\"" Nov 02 17:53:35.149892 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.149851 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvcdg\" (UniqueName: \"kubernetes.io/projected/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-kube-api-access-zvcdg\") pod \"maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd\" (UID: \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" Nov 02 17:53:35.150132 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.149929 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd\" (UID: \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" Nov 02 17:53:35.150132 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.150048 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-metrics-certs\") pod \"maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd\" (UID: \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" Nov 02 17:53:35.150132 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.150122 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd\" (UID: \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" Nov 02 17:53:35.214907 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.214862 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:35.251273 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.251235 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-metrics-certs\") pod \"maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd\" (UID: \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" Nov 02 17:53:35.251435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.251297 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd\" (UID: \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" Nov 02 17:53:35.251482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.251432 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvcdg\" (UniqueName: \"kubernetes.io/projected/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-kube-api-access-zvcdg\") pod \"maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd\" (UID: \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" Nov 02 17:53:35.251518 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.251480 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd\" (UID: \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" Nov 02 17:53:35.252149 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.252124 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd\" (UID: \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" Nov 02 17:53:35.253796 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.253767 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd\" (UID: \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" Nov 02 17:53:35.253899 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.253827 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-metrics-certs\") pod \"maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd\" (UID: \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" Nov 02 17:53:35.288978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.264848 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvcdg\" (UniqueName: \"kubernetes.io/projected/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-kube-api-access-zvcdg\") pod \"maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd\" (UID: \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" Nov 02 17:53:35.393245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.393158 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" Nov 02 17:53:35.498510 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.497827 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:35.689266 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.689227 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd"] Nov 02 17:53:35.690739 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:53:35.690711 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0ba7ba_b19f_4f6b_9997_1860d1b9ca1d.slice/crio-76b2fff4cec1f8722410c2063ac4b4a0d27ca0666caaa3e51a065ca71f9a0081 WatchSource:0}: Error finding container 76b2fff4cec1f8722410c2063ac4b4a0d27ca0666caaa3e51a065ca71f9a0081: Status 404 returned error can't find the container with id 76b2fff4cec1f8722410c2063ac4b4a0d27ca0666caaa3e51a065ca71f9a0081 Nov 02 17:53:35.964483 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:35.964446 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" event={"ID":"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d","Type":"ContainerStarted","Data":"76b2fff4cec1f8722410c2063ac4b4a0d27ca0666caaa3e51a065ca71f9a0081"} Nov 02 17:53:36.259254 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:36.259169 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:36.972939 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:36.972898 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" event={"ID":"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d","Type":"ContainerStarted","Data":"1d842fb803b5f473cdfb8c9183c2e4ff0eaf36e53a58e7a02c5ca95d02ae6a92"} Nov 02 17:53:36.973158 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:36.973005 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" Nov 02 17:53:36.999559 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:36.999500 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" podStartSLOduration=1.409429177 podStartE2EDuration="1.999485631s" totalImagesPullingTime="590.056454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:36.99736534 +0000 UTC m=+1188.785870867" watchObservedRunningTime="2026-11-02 17:53:36.999485631 +0000 UTC m=+1188.787991159" Nov 02 17:53:37.267557 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:37.267464 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:38.237230 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:38.237194 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:39.039321 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.039269 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh"] Nov 02 17:53:39.044090 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.044062 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.056129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.056098 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-d8e217\"/\"kube-root-ca.crt\"" Nov 02 17:53:39.057487 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.057461 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-d8e217\"/\"openshift-service-ca.crt\"" Nov 02 17:53:39.057623 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.057513 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-d8e217\"/\"praxis-route-d8e217-kserve-self-signed-certs\"" Nov 02 17:53:39.057623 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.057564 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-d8e217\"/\"default-dockercfg-7n52b\"" Nov 02 17:53:39.128138 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.128097 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-tmp-dir\") pod \"praxis-route-d8e217-kserve-754885c495-rgfjh\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.128325 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.128166 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqsf\" (UniqueName: \"kubernetes.io/projected/3827bd23-547c-4ae6-83c5-90c71860f814-kube-api-access-shqsf\") pod \"praxis-route-d8e217-kserve-754885c495-rgfjh\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.128383 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.128336 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-dshm\") pod \"praxis-route-d8e217-kserve-754885c495-rgfjh\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.128463 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.128445 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-home\") pod \"praxis-route-d8e217-kserve-754885c495-rgfjh\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.128599 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.128585 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3827bd23-547c-4ae6-83c5-90c71860f814-tls-certs\") pod \"praxis-route-d8e217-kserve-754885c495-rgfjh\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.128643 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.128610 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-model-cache\") pod \"praxis-route-d8e217-kserve-754885c495-rgfjh\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.157247 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.157203 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh"] Nov 02 17:53:39.167826 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.167785 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:39.229969 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.229925 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-tmp-dir\") pod \"praxis-route-d8e217-kserve-754885c495-rgfjh\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.230155 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.230057 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqsf\" (UniqueName: \"kubernetes.io/projected/3827bd23-547c-4ae6-83c5-90c71860f814-kube-api-access-shqsf\") pod \"praxis-route-d8e217-kserve-754885c495-rgfjh\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.230222 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.230190 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-dshm\") pod \"praxis-route-d8e217-kserve-754885c495-rgfjh\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.230303 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.230264 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-home\") pod \"praxis-route-d8e217-kserve-754885c495-rgfjh\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.230380 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.230365 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3827bd23-547c-4ae6-83c5-90c71860f814-tls-certs\") pod \"praxis-route-d8e217-kserve-754885c495-rgfjh\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.230439 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.230376 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-tmp-dir\") pod \"praxis-route-d8e217-kserve-754885c495-rgfjh\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.230439 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.230400 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-model-cache\") pod \"praxis-route-d8e217-kserve-754885c495-rgfjh\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.230621 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.230600 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-home\") pod \"praxis-route-d8e217-kserve-754885c495-rgfjh\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.230701 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.230680 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-model-cache\") pod \"praxis-route-d8e217-kserve-754885c495-rgfjh\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.232390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.232369 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-dshm\") pod \"praxis-route-d8e217-kserve-754885c495-rgfjh\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.232720 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.232701 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3827bd23-547c-4ae6-83c5-90c71860f814-tls-certs\") pod \"praxis-route-d8e217-kserve-754885c495-rgfjh\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.241353 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.241327 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqsf\" (UniqueName: \"kubernetes.io/projected/3827bd23-547c-4ae6-83c5-90c71860f814-kube-api-access-shqsf\") pod \"praxis-route-d8e217-kserve-754885c495-rgfjh\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.360024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.359518 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:39.438062 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.437966 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:39.747143 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:39.747112 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh"] Nov 02 17:53:39.748540 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:53:39.748511 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3827bd23_547c_4ae6_83c5_90c71860f814.slice/crio-15325dbb4691fe82f7f937660926eb388b2e4c7581b7aad44a76ba75d3ebc5f7 WatchSource:0}: Error finding container 15325dbb4691fe82f7f937660926eb388b2e4c7581b7aad44a76ba75d3ebc5f7: Status 404 returned error can't find the container with id 15325dbb4691fe82f7f937660926eb388b2e4c7581b7aad44a76ba75d3ebc5f7 Nov 02 17:53:40.014195 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:40.014108 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" event={"ID":"3827bd23-547c-4ae6-83c5-90c71860f814","Type":"ContainerStarted","Data":"7740a846d6bd96f7ed598d166db9313b29b8e68997874ed5b7e690059de16267"} Nov 02 17:53:40.014195 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:40.014154 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" event={"ID":"3827bd23-547c-4ae6-83c5-90c71860f814","Type":"ContainerStarted","Data":"15325dbb4691fe82f7f937660926eb388b2e4c7581b7aad44a76ba75d3ebc5f7"} Nov 02 17:53:40.062976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:40.055407 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" podStartSLOduration=1.055385991 podStartE2EDuration="1.055385991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:40.054444963 +0000 UTC m=+1191.842950492" watchObservedRunningTime="2026-11-02 17:53:40.055385991 +0000 UTC m=+1191.843891520" Nov 02 17:53:40.102006 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:40.101970 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:40.361457 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:40.361367 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:40.902479 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:40.902433 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df" Nov 02 17:53:40.997411 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:40.996183 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:41.874218 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:41.874173 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:41.911627 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:41.911590 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8" Nov 02 17:53:42.491010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:42.490402 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:42.764029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:42.762566 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:43.029255 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:43.020806 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" Nov 02 17:53:43.069486 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:43.069446 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:43.471535 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:43.471491 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:43.824308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:43.824220 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:44.157905 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:44.157794 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:44.409326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:44.409222 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:45.099000 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:45.098938 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:45.978756 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:45.978709 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:46.271774 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:46.271683 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:53:46.373999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:46.372453 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:46.724723 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:46.724672 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:47.654384 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:47.654343 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:47.656332 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:53:47.656304 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:53:47.908087 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:47.906189 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:48.252979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:48.226894 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:48.734997 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:48.734935 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:49.282978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:49.274130 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 17:53:49.379978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:49.379841 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:49.610005 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:49.609968 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:49.786309 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:49.785757 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 17:53:49.928101 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:49.922453 5549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:50.163995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:50.157606 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:50.235977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:50.232889 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:50.235977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:50.232987 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" podUID="9a22d1c3-76ca-445d-ab56-052577f4fcff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:50.270354 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:50.267643 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:50.444576 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:50.444546 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:53:51.061157 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:51.061042 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:51.852814 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:51.852777 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:52.031447 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:52.031409 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:52.843976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:52.837861 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:53.738966 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:53.736849 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:54.445351 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:54.445307 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:54.503389 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:54.503357 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l"] Nov 02 17:53:54.503588 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:54.503566 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l" podUID="f0986e16-44a0-471e-9467-5373643adae6" containerName="payload-processing" containerID="cri-o://f8e9777b93e64ea46a2da39718343dfb85001fa3f24e962ecfe149227eec9345" gracePeriod=30 Nov 02 17:53:54.616215 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:54.616185 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:54.960895 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:54.960572 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l" Nov 02 17:53:55.000797 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:55.000751 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:55.056477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:55.056444 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0986e16-44a0-471e-9467-5373643adae6-kube-api-access-bprs6\" (UniqueName: \"kubernetes.io/projected/f0986e16-44a0-471e-9467-5373643adae6-kube-api-access-bprs6\") pod \"f0986e16-44a0-471e-9467-5373643adae6\" (UID: \"f0986e16-44a0-471e-9467-5373643adae6\") " Nov 02 17:53:55.056692 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:55.056675 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0986e16-44a0-471e-9467-5373643adae6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0986e16-44a0-471e-9467-5373643adae6-plugins-config-volume\") pod \"f0986e16-44a0-471e-9467-5373643adae6\" (UID: \"f0986e16-44a0-471e-9467-5373643adae6\") " Nov 02 17:53:55.057178 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:55.057149 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0986e16-44a0-471e-9467-5373643adae6-plugins-config-volume" pod "f0986e16-44a0-471e-9467-5373643adae6" (UID: "f0986e16-44a0-471e-9467-5373643adae6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:55.059193 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:55.059169 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0986e16-44a0-471e-9467-5373643adae6-kube-api-access-bprs6" pod "f0986e16-44a0-471e-9467-5373643adae6" (UID: "f0986e16-44a0-471e-9467-5373643adae6"). InnerVolumeSpecName "kube-api-access-bprs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:55.160304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:55.160171 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0986e16-44a0-471e-9467-5373643adae6-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:55.160304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:55.160212 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bprs6\" (UniqueName: \"kubernetes.io/projected/f0986e16-44a0-471e-9467-5373643adae6-kube-api-access-bprs6\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:55.213783 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:55.213649 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:55.508999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:55.508929 5549 generic.go:356] "Generic (PLEG): container finished" podID="f0986e16-44a0-471e-9467-5373643adae6" containerID="f8e9777b93e64ea46a2da39718343dfb85001fa3f24e962ecfe149227eec9345" exitCode=0 Nov 02 17:53:55.508999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:55.508994 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l" Nov 02 17:53:55.509513 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:55.508983 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l" event={"ID":"f0986e16-44a0-471e-9467-5373643adae6","Type":"ContainerDied","Data":"f8e9777b93e64ea46a2da39718343dfb85001fa3f24e962ecfe149227eec9345"} Nov 02 17:53:55.509513 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:55.509126 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l" event={"ID":"f0986e16-44a0-471e-9467-5373643adae6","Type":"ContainerDied","Data":"00cd00b37e4938a4e600c30c54aa155c04d2d2d774fa8d41ad7f96d2ea22932b"} Nov 02 17:53:55.509513 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:55.509161 5549 scope.go:122] "RemoveContainer" containerID="f8e9777b93e64ea46a2da39718343dfb85001fa3f24e962ecfe149227eec9345" Nov 02 17:53:55.520010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:55.519991 5549 scope.go:122] "RemoveContainer" containerID="f8e9777b93e64ea46a2da39718343dfb85001fa3f24e962ecfe149227eec9345" Nov 02 17:53:55.520266 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:53:55.520250 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e9777b93e64ea46a2da39718343dfb85001fa3f24e962ecfe149227eec9345\": container with ID starting with f8e9777b93e64ea46a2da39718343dfb85001fa3f24e962ecfe149227eec9345 not found: ID does not exist" containerID="f8e9777b93e64ea46a2da39718343dfb85001fa3f24e962ecfe149227eec9345" Nov 02 17:53:55.520311 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:55.520273 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e9777b93e64ea46a2da39718343dfb85001fa3f24e962ecfe149227eec9345"} err="failed to get container status \"f8e9777b93e64ea46a2da39718343dfb85001fa3f24e962ecfe149227eec9345\": rpc error: code = NotFound desc = could not find container \"f8e9777b93e64ea46a2da39718343dfb85001fa3f24e962ecfe149227eec9345\": container with ID starting with f8e9777b93e64ea46a2da39718343dfb85001fa3f24e962ecfe149227eec9345 not found: ID does not exist" Nov 02 17:53:55.535116 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:55.535086 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l"] Nov 02 17:53:55.542110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:55.542084 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eda8e2-c494b4ddb-b286l"] Nov 02 17:53:56.308267 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.308228 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:56.371854 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.365888 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0986e16-44a0-471e-9467-5373643adae6" path="/var/lib/kubelet/pods/f0986e16-44a0-471e-9467-5373643adae6/volumes" Nov 02 17:53:56.684451 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.684368 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:56.760245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.760202 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn"] Nov 02 17:53:56.762015 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.761980 5549 cpu_manager.go:414] "containerMap: removing container" podUID="f0986e16-44a0-471e-9467-5373643adae6" containerName="payload-processing" Nov 02 17:53:56.762015 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.762004 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0986e16-44a0-471e-9467-5373643adae6" containerName="payload-processing" Nov 02 17:53:56.762323 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.762303 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0986e16-44a0-471e-9467-5373643adae6" containerName="payload-processing" Nov 02 17:53:56.769614 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.769588 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn" Nov 02 17:53:56.776293 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.776266 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn"] Nov 02 17:53:56.781716 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.781680 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-e198eb\"" Nov 02 17:53:56.785342 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.785313 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-e198eb-dockercfg-nn8sr\"" Nov 02 17:53:56.789014 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.788988 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z"] Nov 02 17:53:56.793254 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.793225 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" Nov 02 17:53:56.819272 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.819240 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z"] Nov 02 17:53:56.958030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.936380 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lcx6\" (UniqueName: \"kubernetes.io/projected/f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d-kube-api-access-4lcx6\") pod \"payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn\" (UID: \"f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn" Nov 02 17:53:56.958030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.936465 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slkj4\" (UniqueName: \"kubernetes.io/projected/34b73845-9bed-45b5-b698-12759713f944-kube-api-access-slkj4\") pod \"payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z\" (UID: \"34b73845-9bed-45b5-b698-12759713f944\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" Nov 02 17:53:56.958030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.936601 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn\" (UID: \"f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn" Nov 02 17:53:56.958030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.936663 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b73845-9bed-45b5-b698-12759713f944-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z\" (UID: \"34b73845-9bed-45b5-b698-12759713f944\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" Nov 02 17:53:56.958030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:56.940630 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:57.044035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.038423 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lcx6\" (UniqueName: \"kubernetes.io/projected/f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d-kube-api-access-4lcx6\") pod \"payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn\" (UID: \"f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn" Nov 02 17:53:57.044035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.038528 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slkj4\" (UniqueName: \"kubernetes.io/projected/34b73845-9bed-45b5-b698-12759713f944-kube-api-access-slkj4\") pod \"payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z\" (UID: \"34b73845-9bed-45b5-b698-12759713f944\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" Nov 02 17:53:57.044035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.038684 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn\" (UID: \"f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn" Nov 02 17:53:57.044035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.038764 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b73845-9bed-45b5-b698-12759713f944-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z\" (UID: \"34b73845-9bed-45b5-b698-12759713f944\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" Nov 02 17:53:57.044035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.040924 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn\" (UID: \"f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn" Nov 02 17:53:57.044035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.041520 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b73845-9bed-45b5-b698-12759713f944-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z\" (UID: \"34b73845-9bed-45b5-b698-12759713f944\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" Nov 02 17:53:57.052635 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.052575 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lcx6\" (UniqueName: \"kubernetes.io/projected/f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d-kube-api-access-4lcx6\") pod \"payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn\" (UID: \"f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn" Nov 02 17:53:57.063454 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.063425 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slkj4\" (UniqueName: \"kubernetes.io/projected/34b73845-9bed-45b5-b698-12759713f944-kube-api-access-slkj4\") pod \"payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z\" (UID: \"34b73845-9bed-45b5-b698-12759713f944\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" Nov 02 17:53:57.088285 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.088251 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn" Nov 02 17:53:57.160428 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.160182 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" Nov 02 17:53:57.361180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.361146 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:57.413373 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.413246 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn"] Nov 02 17:53:57.416528 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:53:57.416502 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf586b4c3_d237_4d83_a2bb_5fbc85c3bc6d.slice/crio-0d68b8545f81a0ca6c9d34f745890db10cbf65efe75b65751b6dcd96845276df WatchSource:0}: Error finding container 0d68b8545f81a0ca6c9d34f745890db10cbf65efe75b65751b6dcd96845276df: Status 404 returned error can't find the container with id 0d68b8545f81a0ca6c9d34f745890db10cbf65efe75b65751b6dcd96845276df Nov 02 17:53:57.461126 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.461099 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z"] Nov 02 17:53:57.461408 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:53:57.461382 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b73845_9bed_45b5_b698_12759713f944.slice/crio-b28f80e140ec2b2382a76b945bcd65e6c291b7116fcd043e42ed605970623114 WatchSource:0}: Error finding container b28f80e140ec2b2382a76b945bcd65e6c291b7116fcd043e42ed605970623114: Status 404 returned error can't find the container with id b28f80e140ec2b2382a76b945bcd65e6c291b7116fcd043e42ed605970623114 Nov 02 17:53:57.525375 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.525341 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" event={"ID":"34b73845-9bed-45b5-b698-12759713f944","Type":"ContainerStarted","Data":"f1b20e39185f9185d922b4fb586d649c2fa26e9f593484feaf60339d185e3ba2"} Nov 02 17:53:57.525508 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.525402 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" event={"ID":"34b73845-9bed-45b5-b698-12759713f944","Type":"ContainerStarted","Data":"b28f80e140ec2b2382a76b945bcd65e6c291b7116fcd043e42ed605970623114"} Nov 02 17:53:57.525508 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.525431 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" Nov 02 17:53:57.527150 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.527127 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn" event={"ID":"f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d","Type":"ContainerStarted","Data":"65f253ae6df5be8111db4829ec9cb3968246f1b48138d74fc32c6e1b8304e087"} Nov 02 17:53:57.527255 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.527173 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn" Nov 02 17:53:57.527255 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.527191 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn" event={"ID":"f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d","Type":"ContainerStarted","Data":"0d68b8545f81a0ca6c9d34f745890db10cbf65efe75b65751b6dcd96845276df"} Nov 02 17:53:57.547238 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.547196 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" podStartSLOduration=1.547182443 podStartE2EDuration="1.547182443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:57.545151782 +0000 UTC m=+1209.333657334" watchObservedRunningTime="2026-11-02 17:53:57.547182443 +0000 UTC m=+1209.335687971" Nov 02 17:53:57.566033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:57.565993 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn" podStartSLOduration=1.565982062 podStartE2EDuration="1.565982062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:57.565448328 +0000 UTC m=+1209.353953868" watchObservedRunningTime="2026-11-02 17:53:57.565982062 +0000 UTC m=+1209.354487590" Nov 02 17:53:58.166415 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:58.166376 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:58.660334 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:58.660297 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:58.992470 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:58.992424 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:59.220772 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:59.220704 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:59.415532 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:59.415421 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:59.781834 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:53:59.781777 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:00.090025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:00.083149 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:00.100042 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:00.098299 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd"] Nov 02 17:54:00.100042 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:00.098607 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" podUID="c9f78f64-e109-490b-a5ae-4fcec9644276" containerName="istio-proxy" containerID="cri-o://c4f33b887ecfa6e06a68d6b1fda29f8fae035855e414f8f78435a9fb6980610c" gracePeriod=30 Nov 02 17:54:00.326009 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:00.321552 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:00.967002 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:00.966944 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:01.662753 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:01.662718 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:01.833915 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:01.833878 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:01.962734 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:01.962694 5549 patch_prober.go:29] interesting pod/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:54:01.962906 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:01.962751 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" podUID="c9f78f64-e109-490b-a5ae-4fcec9644276" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:54:01.980072 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:01.980038 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878"] Nov 02 17:54:01.986315 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:01.986289 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:01.994586 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:01.994557 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878"] Nov 02 17:54:02.017373 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.017346 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-dfd356-openshift-default-dockercfg-thxt9\"" Nov 02 17:54:02.068998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.068174 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:02.125975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.085830 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-envoy\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.126398 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.126367 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-credential-socket\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.126653 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.126631 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-workload-socket\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.126882 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.126852 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mltdc\" (UniqueName: \"kubernetes.io/projected/c8d28e37-d453-4fb6-878e-68b551f73dde-kube-api-access-mltdc\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.127062 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.127043 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-data\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.127210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.127194 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-token\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.127361 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.127345 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d28e37-d453-4fb6-878e-68b551f73dde-istiod-ca-cert\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.127492 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.127477 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-workload-certs\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.127660 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.127642 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-podinfo\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.234024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.233049 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mltdc\" (UniqueName: \"kubernetes.io/projected/c8d28e37-d453-4fb6-878e-68b551f73dde-kube-api-access-mltdc\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.234024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.233284 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-data\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.234024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.233341 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-token\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.234024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.233381 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d28e37-d453-4fb6-878e-68b551f73dde-istiod-ca-cert\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.234024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.233425 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-workload-certs\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.234024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.233462 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-podinfo\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.234500 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.234224 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-envoy\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.234500 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.234386 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-credential-socket\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.234614 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.234550 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-workload-socket\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.235492 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.235466 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-workload-socket\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.235614 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.235530 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-data\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.235987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.235967 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-credential-socket\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.236346 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.236289 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-workload-certs\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.245214 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.245177 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d28e37-d453-4fb6-878e-68b551f73dde-istiod-ca-cert\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.258399 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.258316 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-podinfo\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.264036 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.262049 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-token\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.268742 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.268634 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-envoy\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.269738 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.269687 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mltdc\" (UniqueName: \"kubernetes.io/projected/c8d28e37-d453-4fb6-878e-68b551f73dde-kube-api-access-mltdc\") pod \"e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.304794 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.304762 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:02.352076 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.347017 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:02.569824 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.564244 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:02.676719 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.676686 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878"] Nov 02 17:54:02.677866 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:54:02.677838 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d28e37_d453_4fb6_878e_68b551f73dde.slice/crio-f04a60a933036a571a38800ac648993b9c25c216e8635a7514a8cbce4c42cbb8 WatchSource:0}: Error finding container f04a60a933036a571a38800ac648993b9c25c216e8635a7514a8cbce4c42cbb8: Status 404 returned error can't find the container with id f04a60a933036a571a38800ac648993b9c25c216e8635a7514a8cbce4c42cbb8 Nov 02 17:54:02.680503 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.680467 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:54:02.680613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.680549 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:54:02.680613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:02.680594 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:54:03.111033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.110999 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb"] Nov 02 17:54:03.118377 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.118348 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" Nov 02 17:54:03.132794 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.132761 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-e198eb-serving-cert\"" Nov 02 17:54:03.138375 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.135855 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb"] Nov 02 17:54:03.279989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.271475 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36810b4f-4409-4a63-96fa-8c4565058368-metrics-certs\") pod \"maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb\" (UID: \"36810b4f-4409-4a63-96fa-8c4565058368\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" Nov 02 17:54:03.279989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.271590 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36810b4f-4409-4a63-96fa-8c4565058368-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb\" (UID: \"36810b4f-4409-4a63-96fa-8c4565058368\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" Nov 02 17:54:03.279989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.271765 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36810b4f-4409-4a63-96fa-8c4565058368-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb\" (UID: \"36810b4f-4409-4a63-96fa-8c4565058368\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" Nov 02 17:54:03.279989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.271808 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwsqb\" (UniqueName: \"kubernetes.io/projected/36810b4f-4409-4a63-96fa-8c4565058368-kube-api-access-vwsqb\") pod \"maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb\" (UID: \"36810b4f-4409-4a63-96fa-8c4565058368\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" Nov 02 17:54:03.279989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.272938 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:03.373723 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.373640 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36810b4f-4409-4a63-96fa-8c4565058368-metrics-certs\") pod \"maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb\" (UID: \"36810b4f-4409-4a63-96fa-8c4565058368\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" Nov 02 17:54:03.373723 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.373707 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36810b4f-4409-4a63-96fa-8c4565058368-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb\" (UID: \"36810b4f-4409-4a63-96fa-8c4565058368\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" Nov 02 17:54:03.373985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.373828 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36810b4f-4409-4a63-96fa-8c4565058368-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb\" (UID: \"36810b4f-4409-4a63-96fa-8c4565058368\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" Nov 02 17:54:03.373985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.373871 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwsqb\" (UniqueName: \"kubernetes.io/projected/36810b4f-4409-4a63-96fa-8c4565058368-kube-api-access-vwsqb\") pod \"maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb\" (UID: \"36810b4f-4409-4a63-96fa-8c4565058368\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" Nov 02 17:54:03.374570 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.374539 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36810b4f-4409-4a63-96fa-8c4565058368-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb\" (UID: \"36810b4f-4409-4a63-96fa-8c4565058368\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" Nov 02 17:54:03.376335 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.376309 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36810b4f-4409-4a63-96fa-8c4565058368-metrics-certs\") pod \"maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb\" (UID: \"36810b4f-4409-4a63-96fa-8c4565058368\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" Nov 02 17:54:03.376523 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.376498 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36810b4f-4409-4a63-96fa-8c4565058368-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb\" (UID: \"36810b4f-4409-4a63-96fa-8c4565058368\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" Nov 02 17:54:03.394647 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.394616 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwsqb\" (UniqueName: \"kubernetes.io/projected/36810b4f-4409-4a63-96fa-8c4565058368-kube-api-access-vwsqb\") pod \"maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb\" (UID: \"36810b4f-4409-4a63-96fa-8c4565058368\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" Nov 02 17:54:03.430445 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.430408 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:03.447636 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.447603 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" Nov 02 17:54:03.584145 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.584106 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:03.608109 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.608072 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" event={"ID":"c8d28e37-d453-4fb6-878e-68b551f73dde","Type":"ContainerStarted","Data":"42805b3aacbf92b7dd51e641348c5c1609f271d8f994d885c42f1c10843542fb"} Nov 02 17:54:03.608280 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.608129 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" event={"ID":"c8d28e37-d453-4fb6-878e-68b551f73dde","Type":"ContainerStarted","Data":"f04a60a933036a571a38800ac648993b9c25c216e8635a7514a8cbce4c42cbb8"} Nov 02 17:54:03.634306 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.634214 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" podStartSLOduration=2.634201298 podStartE2EDuration="2.634201298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:03.629910997 +0000 UTC m=+1215.418416544" watchObservedRunningTime="2026-11-02 17:54:03.634201298 +0000 UTC m=+1215.422706826" Nov 02 17:54:03.733776 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:03.733746 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb"] Nov 02 17:54:03.734896 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:54:03.734870 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36810b4f_4409_4a63_96fa_8c4565058368.slice/crio-a64d28b229e42a10c36b46e090815cb9c885cf98871f04d352eb7fb8c4a0a9de WatchSource:0}: Error finding container a64d28b229e42a10c36b46e090815cb9c885cf98871f04d352eb7fb8c4a0a9de: Status 404 returned error can't find the container with id a64d28b229e42a10c36b46e090815cb9c885cf98871f04d352eb7fb8c4a0a9de Nov 02 17:54:04.197222 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:04.193723 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:04.305868 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:04.305835 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:04.323479 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:04.323454 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:04.616479 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:04.616384 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" event={"ID":"36810b4f-4409-4a63-96fa-8c4565058368","Type":"ContainerStarted","Data":"c073892f66176455dfd091074f698eee87dd99233f7d724b0742f22d519af0cf"} Nov 02 17:54:04.616479 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:04.616450 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" event={"ID":"36810b4f-4409-4a63-96fa-8c4565058368","Type":"ContainerStarted","Data":"a64d28b229e42a10c36b46e090815cb9c885cf98871f04d352eb7fb8c4a0a9de"} Nov 02 17:54:04.616685 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:04.616483 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" Nov 02 17:54:04.616685 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:04.616625 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:04.617501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:04.617480 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:54:04.635406 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:04.635363 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" podStartSLOduration=1.110181985 podStartE2EDuration="1.635351927s" totalImagesPullingTime="525.169942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:04.633924379 +0000 UTC m=+1216.422429907" watchObservedRunningTime="2026-11-02 17:54:04.635351927 +0000 UTC m=+1216.423857454" Nov 02 17:54:04.889200 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:04.889114 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:05.390863 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.390830 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:54:05.466759 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.466720 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-workload-socket\") pod \"c9f78f64-e109-490b-a5ae-4fcec9644276\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " Nov 02 17:54:05.466994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.466798 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-envoy\") pod \"c9f78f64-e109-490b-a5ae-4fcec9644276\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " Nov 02 17:54:05.466994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.466851 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-workload-certs\") pod \"c9f78f64-e109-490b-a5ae-4fcec9644276\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " Nov 02 17:54:05.467134 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.467002 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9f78f64-e109-490b-a5ae-4fcec9644276-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f78f64-e109-490b-a5ae-4fcec9644276-istiod-ca-cert\") pod \"c9f78f64-e109-490b-a5ae-4fcec9644276\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " Nov 02 17:54:05.467134 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.467081 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-workload-socket" pod "c9f78f64-e109-490b-a5ae-4fcec9644276" (UID: "c9f78f64-e109-490b-a5ae-4fcec9644276"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:05.467134 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.467103 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-podinfo\") pod \"c9f78f64-e109-490b-a5ae-4fcec9644276\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " Nov 02 17:54:05.467348 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.467185 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-token\" (UniqueName: \"kubernetes.io/projected/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-token\") pod \"c9f78f64-e109-490b-a5ae-4fcec9644276\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " Nov 02 17:54:05.467348 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.467220 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-workload-certs" pod "c9f78f64-e109-490b-a5ae-4fcec9644276" (UID: "c9f78f64-e109-490b-a5ae-4fcec9644276"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:05.467348 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.467242 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f78f64-e109-490b-a5ae-4fcec9644276-kube-api-access-vx85z\" (UniqueName: \"kubernetes.io/projected/c9f78f64-e109-490b-a5ae-4fcec9644276-kube-api-access-vx85z\") pod \"c9f78f64-e109-490b-a5ae-4fcec9644276\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " Nov 02 17:54:05.467348 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.467293 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-credential-socket\") pod \"c9f78f64-e109-490b-a5ae-4fcec9644276\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " Nov 02 17:54:05.467550 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.467411 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-data\") pod \"c9f78f64-e109-490b-a5ae-4fcec9644276\" (UID: \"c9f78f64-e109-490b-a5ae-4fcec9644276\") " Nov 02 17:54:05.467550 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.467415 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f78f64-e109-490b-a5ae-4fcec9644276-istiod-ca-cert" pod "c9f78f64-e109-490b-a5ae-4fcec9644276" (UID: "c9f78f64-e109-490b-a5ae-4fcec9644276"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:05.467641 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.467577 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-credential-socket" pod "c9f78f64-e109-490b-a5ae-4fcec9644276" (UID: "c9f78f64-e109-490b-a5ae-4fcec9644276"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:05.467990 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.467753 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-data" pod "c9f78f64-e109-490b-a5ae-4fcec9644276" (UID: "c9f78f64-e109-490b-a5ae-4fcec9644276"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:05.468177 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.468154 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:05.468251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.468183 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:05.468251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.468198 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:05.468251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.468213 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f78f64-e109-490b-a5ae-4fcec9644276-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:05.468251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.468226 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:05.469394 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.469374 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-token" pod "c9f78f64-e109-490b-a5ae-4fcec9644276" (UID: "c9f78f64-e109-490b-a5ae-4fcec9644276"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:05.469520 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.469423 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-podinfo" pod "c9f78f64-e109-490b-a5ae-4fcec9644276" (UID: "c9f78f64-e109-490b-a5ae-4fcec9644276"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:54:05.469573 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.469520 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-envoy" pod "c9f78f64-e109-490b-a5ae-4fcec9644276" (UID: "c9f78f64-e109-490b-a5ae-4fcec9644276"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:05.469611 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.469579 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f78f64-e109-490b-a5ae-4fcec9644276-kube-api-access-vx85z" pod "c9f78f64-e109-490b-a5ae-4fcec9644276" (UID: "c9f78f64-e109-490b-a5ae-4fcec9644276"). InnerVolumeSpecName "kube-api-access-vx85z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:05.569620 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.569584 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:05.569620 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.569614 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:05.569620 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.569624 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx85z\" (UniqueName: \"kubernetes.io/projected/c9f78f64-e109-490b-a5ae-4fcec9644276-kube-api-access-vx85z\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:05.569826 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.569635 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f78f64-e109-490b-a5ae-4fcec9644276-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:05.625100 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.625059 5549 generic.go:356] "Generic (PLEG): container finished" podID="c9f78f64-e109-490b-a5ae-4fcec9644276" containerID="c4f33b887ecfa6e06a68d6b1fda29f8fae035855e414f8f78435a9fb6980610c" exitCode=0 Nov 02 17:54:05.625309 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.625125 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" Nov 02 17:54:05.625309 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.625148 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" event={"ID":"c9f78f64-e109-490b-a5ae-4fcec9644276","Type":"ContainerDied","Data":"c4f33b887ecfa6e06a68d6b1fda29f8fae035855e414f8f78435a9fb6980610c"} Nov 02 17:54:05.625309 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.625219 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd" event={"ID":"c9f78f64-e109-490b-a5ae-4fcec9644276","Type":"ContainerDied","Data":"edd8c45727174994ee67d35c8ecc34be4ef0bd2ab36165594d3bfbb963aee10a"} Nov 02 17:54:05.625309 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.625254 5549 scope.go:122] "RemoveContainer" containerID="c4f33b887ecfa6e06a68d6b1fda29f8fae035855e414f8f78435a9fb6980610c" Nov 02 17:54:05.636351 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.636330 5549 scope.go:122] "RemoveContainer" containerID="84931c92ab04e142b5e666ece6dac8e8ff0cfb0155929d5c4f286fdcf0cda1cb" Nov 02 17:54:05.645576 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.645557 5549 scope.go:122] "RemoveContainer" containerID="c4f33b887ecfa6e06a68d6b1fda29f8fae035855e414f8f78435a9fb6980610c" Nov 02 17:54:05.645842 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:54:05.645823 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f33b887ecfa6e06a68d6b1fda29f8fae035855e414f8f78435a9fb6980610c\": container with ID starting with c4f33b887ecfa6e06a68d6b1fda29f8fae035855e414f8f78435a9fb6980610c not found: ID does not exist" containerID="c4f33b887ecfa6e06a68d6b1fda29f8fae035855e414f8f78435a9fb6980610c" Nov 02 17:54:05.645916 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.645853 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f33b887ecfa6e06a68d6b1fda29f8fae035855e414f8f78435a9fb6980610c"} err="failed to get container status \"c4f33b887ecfa6e06a68d6b1fda29f8fae035855e414f8f78435a9fb6980610c\": rpc error: code = NotFound desc = could not find container \"c4f33b887ecfa6e06a68d6b1fda29f8fae035855e414f8f78435a9fb6980610c\": container with ID starting with c4f33b887ecfa6e06a68d6b1fda29f8fae035855e414f8f78435a9fb6980610c not found: ID does not exist" Nov 02 17:54:05.645916 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.645877 5549 scope.go:122] "RemoveContainer" containerID="84931c92ab04e142b5e666ece6dac8e8ff0cfb0155929d5c4f286fdcf0cda1cb" Nov 02 17:54:05.646196 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:54:05.646178 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84931c92ab04e142b5e666ece6dac8e8ff0cfb0155929d5c4f286fdcf0cda1cb\": container with ID starting with 84931c92ab04e142b5e666ece6dac8e8ff0cfb0155929d5c4f286fdcf0cda1cb not found: ID does not exist" containerID="84931c92ab04e142b5e666ece6dac8e8ff0cfb0155929d5c4f286fdcf0cda1cb" Nov 02 17:54:05.646248 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.646201 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84931c92ab04e142b5e666ece6dac8e8ff0cfb0155929d5c4f286fdcf0cda1cb"} err="failed to get container status \"84931c92ab04e142b5e666ece6dac8e8ff0cfb0155929d5c4f286fdcf0cda1cb\": rpc error: code = NotFound desc = could not find container \"84931c92ab04e142b5e666ece6dac8e8ff0cfb0155929d5c4f286fdcf0cda1cb\": container with ID starting with 84931c92ab04e142b5e666ece6dac8e8ff0cfb0155929d5c4f286fdcf0cda1cb not found: ID does not exist" Nov 02 17:54:05.651757 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.651733 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd"] Nov 02 17:54:05.656840 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.656819 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-eda8e2-openshift-default-c8f649447-rggwd"] Nov 02 17:54:05.717978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:05.717104 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:06.017213 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.017178 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:06.175235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.175199 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:06.321218 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.321143 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9f78f64-e109-490b-a5ae-4fcec9644276" path="/var/lib/kubelet/pods/c9f78f64-e109-490b-a5ae-4fcec9644276/volumes" Nov 02 17:54:06.570976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.565492 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:06.669916 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.669841 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b"] Nov 02 17:54:06.670530 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.670516 5549 cpu_manager.go:414] "containerMap: removing container" podUID="c9f78f64-e109-490b-a5ae-4fcec9644276" containerName="istio-proxy" Nov 02 17:54:06.670572 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.670531 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f78f64-e109-490b-a5ae-4fcec9644276" containerName="istio-proxy" Nov 02 17:54:06.670572 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.670544 5549 cpu_manager.go:414] "containerMap: removing container" podUID="c9f78f64-e109-490b-a5ae-4fcec9644276" containerName="istio-proxy" Nov 02 17:54:06.670572 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.670550 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f78f64-e109-490b-a5ae-4fcec9644276" containerName="istio-proxy" Nov 02 17:54:06.670675 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.670645 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f78f64-e109-490b-a5ae-4fcec9644276" containerName="istio-proxy" Nov 02 17:54:06.670675 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.670653 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f78f64-e109-490b-a5ae-4fcec9644276" containerName="istio-proxy" Nov 02 17:54:06.675730 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.675706 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.690029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.690001 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b"] Nov 02 17:54:06.793631 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.793587 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67nfr\" (UniqueName: \"kubernetes.io/projected/4952505b-a6ee-469c-b3ad-39f897429fb4-kube-api-access-67nfr\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.793792 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.793680 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.793792 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.793738 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.793792 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.793763 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.793926 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.793869 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.794000 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.793977 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.794113 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.794095 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.794163 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.794146 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4952505b-a6ee-469c-b3ad-39f897429fb4-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.794288 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.794218 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.895815 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.895773 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67nfr\" (UniqueName: \"kubernetes.io/projected/4952505b-a6ee-469c-b3ad-39f897429fb4-kube-api-access-67nfr\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.896029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.895838 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.896029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.895883 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.896029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.895918 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.896029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.896014 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.896209 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.896056 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.896209 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.896132 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.896209 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.896177 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4952505b-a6ee-469c-b3ad-39f897429fb4-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.896362 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.896293 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.896362 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.896331 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.896535 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.896503 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.896621 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.896576 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.896621 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.896604 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.896912 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.896888 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4952505b-a6ee-469c-b3ad-39f897429fb4-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.898477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.898450 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.898781 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.898755 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.907449 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.907422 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67nfr\" (UniqueName: \"kubernetes.io/projected/4952505b-a6ee-469c-b3ad-39f897429fb4-kube-api-access-67nfr\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.908082 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.908065 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:06.991732 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:06.991690 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:07.252421 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:07.252329 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:07.285608 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:07.285572 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b"] Nov 02 17:54:07.287468 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:54:07.287430 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4952505b_a6ee_469c_b3ad_39f897429fb4.slice/crio-bdb581fc99ca0fe31958f71dc16a911e98c4a90f473354c81daefc80c0ba2abf WatchSource:0}: Error finding container bdb581fc99ca0fe31958f71dc16a911e98c4a90f473354c81daefc80c0ba2abf: Status 404 returned error can't find the container with id bdb581fc99ca0fe31958f71dc16a911e98c4a90f473354c81daefc80c0ba2abf Nov 02 17:54:07.291082 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:07.291042 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:54:07.291177 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:07.291133 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:54:07.291265 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:07.291179 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:54:07.427539 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:07.427448 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:07.595807 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:07.595713 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:07.688971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:07.688132 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" event={"ID":"4952505b-a6ee-469c-b3ad-39f897429fb4","Type":"ContainerStarted","Data":"d61b233561e01a4e4acb17d924b420f0737c1dc82571c10afc1195ee03918b33"} Nov 02 17:54:07.688971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:07.688209 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" event={"ID":"4952505b-a6ee-469c-b3ad-39f897429fb4","Type":"ContainerStarted","Data":"bdb581fc99ca0fe31958f71dc16a911e98c4a90f473354c81daefc80c0ba2abf"} Nov 02 17:54:07.733339 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:07.733265 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" podStartSLOduration=1.733242145 podStartE2EDuration="1.733242145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:07.731720373 +0000 UTC m=+1219.520225912" watchObservedRunningTime="2026-11-02 17:54:07.733242145 +0000 UTC m=+1219.521747675" Nov 02 17:54:07.992188 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:07.992146 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:08.536435 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:08.536398 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" Nov 02 17:54:08.536879 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:08.536470 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn" Nov 02 17:54:08.659245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:08.659210 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:09.006739 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:09.006697 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 17:54:09.006908 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:09.006758 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" podUID="4952505b-a6ee-469c-b3ad-39f897429fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 17:54:09.439029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:09.435186 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:10.002211 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:10.001762 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 17:54:10.002211 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:10.001823 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" podUID="4952505b-a6ee-469c-b3ad-39f897429fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 17:54:10.395385 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:10.395282 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:10.631326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:10.631298 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" Nov 02 17:54:10.998234 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:10.992366 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 17:54:11.003002 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:10.998523 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" podUID="4952505b-a6ee-469c-b3ad-39f897429fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 17:54:11.080230 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:11.080183 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:11.743022 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:11.742988 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:12.004399 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:12.004302 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 17:54:12.004399 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:12.004358 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" podUID="4952505b-a6ee-469c-b3ad-39f897429fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 17:54:12.529683 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:12.525675 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:13.004406 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.004368 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 17:54:13.005024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.004942 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" podUID="4952505b-a6ee-469c-b3ad-39f897429fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 17:54:13.312587 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.312505 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml"] Nov 02 17:54:13.319109 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.319079 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml"] Nov 02 17:54:13.319331 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.319198 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" Nov 02 17:54:13.323693 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.323659 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-dfd356-serving-cert\"" Nov 02 17:54:13.405476 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.405446 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:13.474983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.458781 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml\" (UID: \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" Nov 02 17:54:13.474983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.458937 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml\" (UID: \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" Nov 02 17:54:13.474983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.459056 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-metrics-certs\") pod \"maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml\" (UID: \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" Nov 02 17:54:13.474983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.459080 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvrm5\" (UniqueName: \"kubernetes.io/projected/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-kube-api-access-wvrm5\") pod \"maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml\" (UID: \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" Nov 02 17:54:13.560974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.560360 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml\" (UID: \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" Nov 02 17:54:13.560974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.560506 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-metrics-certs\") pod \"maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml\" (UID: \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" Nov 02 17:54:13.560974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.560535 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvrm5\" (UniqueName: \"kubernetes.io/projected/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-kube-api-access-wvrm5\") pod \"maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml\" (UID: \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" Nov 02 17:54:13.560974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.560681 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml\" (UID: \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" Nov 02 17:54:13.563177 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.561609 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml\" (UID: \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" Nov 02 17:54:13.566283 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.566255 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml\" (UID: \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" Nov 02 17:54:13.586974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.583115 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:13.586974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.585015 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-metrics-certs\") pod \"maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml\" (UID: \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" Nov 02 17:54:13.593430 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.593397 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvrm5\" (UniqueName: \"kubernetes.io/projected/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-kube-api-access-wvrm5\") pod \"maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml\" (UID: \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" Nov 02 17:54:13.637164 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.637129 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" Nov 02 17:54:13.999622 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:13.999484 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml"] Nov 02 17:54:14.004253 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:54:14.001421 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26e1b21_d8a8_4bd6_bc52_0052582bfb9d.slice/crio-b8135b0ce1ac68003a7b522aa506202ac5ce01eda557d39bf3fae3e94402b895 WatchSource:0}: Error finding container b8135b0ce1ac68003a7b522aa506202ac5ce01eda557d39bf3fae3e94402b895: Status 404 returned error can't find the container with id b8135b0ce1ac68003a7b522aa506202ac5ce01eda557d39bf3fae3e94402b895 Nov 02 17:54:14.004253 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:14.002992 5549 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 17:54:14.004253 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:14.003042 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" podUID="4952505b-a6ee-469c-b3ad-39f897429fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 17:54:14.222997 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:14.216384 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:14.920249 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:14.920140 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" event={"ID":"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d","Type":"ContainerStarted","Data":"c691557de8307107de58aabc62c063ceda0562ca4f8c429f6ddd4b9270beb4dd"} Nov 02 17:54:14.920249 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:14.920215 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" event={"ID":"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d","Type":"ContainerStarted","Data":"b8135b0ce1ac68003a7b522aa506202ac5ce01eda557d39bf3fae3e94402b895"} Nov 02 17:54:14.920249 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:14.920245 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" Nov 02 17:54:14.943273 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:14.943213 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" podStartSLOduration=1.49250693 podStartE2EDuration="1.943194515s" totalImagesPullingTime="450.687585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:14.940669555 +0000 UTC m=+1226.729175082" watchObservedRunningTime="2026-11-02 17:54:14.943194515 +0000 UTC m=+1226.731700048" Nov 02 17:54:14.986252 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:14.986209 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:15.154485 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:15.154445 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:15.357703 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:15.357669 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:15.358017 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:15.357993 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:15.358749 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:15.358729 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 17:54:15.842984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:15.842747 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:16.638067 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:16.638014 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:17.140648 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:17.140606 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:17.198002 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:17.192523 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:54:17.643336 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:54:17.643302 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:54:17.651360 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:17.651331 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:18.504974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:18.500466 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:19.565977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:19.548220 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:20.316657 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:20.316621 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:20.934256 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:20.934223 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" Nov 02 17:54:21.123218 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:21.111535 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:21.778489 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:21.778438 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:22.451110 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:22.451070 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:23.281271 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.281236 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb"] Nov 02 17:54:23.289013 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.288983 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.300252 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.300220 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb"] Nov 02 17:54:23.312896 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.312858 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.314553 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.314527 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.315568 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.315541 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk95s\" (UniqueName: \"kubernetes.io/projected/a3822c1d-660b-4c73-9104-9bde5105e0c7-kube-api-access-xk95s\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.315785 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.315766 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.319749 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.319733 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.319980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.319946 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3822c1d-660b-4c73-9104-9bde5105e0c7-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.320127 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.320114 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.320260 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.320235 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.320314 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.320297 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.380664 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.380612 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:23.422121 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.422082 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.422324 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.422215 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk95s\" (UniqueName: \"kubernetes.io/projected/a3822c1d-660b-4c73-9104-9bde5105e0c7-kube-api-access-xk95s\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.422324 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.422273 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.422447 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.422333 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.422599 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.422566 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3822c1d-660b-4c73-9104-9bde5105e0c7-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.422745 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.422722 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.422811 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.422762 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.422872 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.422809 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.423008 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.422982 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.423146 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.423019 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.423553 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.423520 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3822c1d-660b-4c73-9104-9bde5105e0c7-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.423855 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.423833 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.424192 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.424167 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.424269 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.424201 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.425161 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.425123 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.425498 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.425467 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.434988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.434948 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk95s\" (UniqueName: \"kubernetes.io/projected/a3822c1d-660b-4c73-9104-9bde5105e0c7-kube-api-access-xk95s\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.435118 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.434991 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.620754 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.620660 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:23.911066 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.911032 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb"] Nov 02 17:54:23.912170 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:54:23.912132 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3822c1d_660b_4c73_9104_9bde5105e0c7.slice/crio-28ed7723c2498b1c21c966605ae2de7714f58b484b6403a478abc25e01b03538 WatchSource:0}: Error finding container 28ed7723c2498b1c21c966605ae2de7714f58b484b6403a478abc25e01b03538: Status 404 returned error can't find the container with id 28ed7723c2498b1c21c966605ae2de7714f58b484b6403a478abc25e01b03538 Nov 02 17:54:23.915028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.914993 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:54:23.915133 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.915075 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:54:23.915133 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:23.915119 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:54:24.016997 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:24.016946 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" event={"ID":"a3822c1d-660b-4c73-9104-9bde5105e0c7","Type":"ContainerStarted","Data":"c9fe04711861759b38d73fb23296027177dfa6a449654b1330e47f5d0c49a7f2"} Nov 02 17:54:24.017147 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:24.017026 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" event={"ID":"a3822c1d-660b-4c73-9104-9bde5105e0c7","Type":"ContainerStarted","Data":"28ed7723c2498b1c21c966605ae2de7714f58b484b6403a478abc25e01b03538"} Nov 02 17:54:24.038830 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:24.038745 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" podStartSLOduration=1.038726197 podStartE2EDuration="1.038726197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:24.036390016 +0000 UTC m=+1235.824895547" watchObservedRunningTime="2026-11-02 17:54:24.038726197 +0000 UTC m=+1235.827231730" Nov 02 17:54:24.295987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:24.295003 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:24.621233 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:24.621134 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:24.622838 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:24.622807 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 17:54:24.622946 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:24.622876 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" podUID="a3822c1d-660b-4c73-9104-9bde5105e0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 17:54:25.005052 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:25.003676 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:25.008088 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:25.008061 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df"] Nov 02 17:54:25.024236 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:25.024202 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8"] Nov 02 17:54:25.173703 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:25.173670 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:25.333155 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:25.329743 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:25.621590 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:25.621506 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 17:54:25.621590 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:25.621557 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" podUID="a3822c1d-660b-4c73-9104-9bde5105e0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 17:54:26.005768 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:26.005734 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:26.142646 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:26.142602 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:26.621493 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:26.621453 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 17:54:26.621662 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:26.621515 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" podUID="a3822c1d-660b-4c73-9104-9bde5105e0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 17:54:26.919024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:26.917497 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:27.247472 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:27.247440 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:27.622150 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:27.622066 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 17:54:27.622150 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:27.622126 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" podUID="a3822c1d-660b-4c73-9104-9bde5105e0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 17:54:27.655311 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:27.655249 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:27.861323 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:27.861274 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:28.223978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:28.215236 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:28.511023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:28.510406 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:28.622160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:28.622111 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 17:54:28.622667 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:28.622180 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" podUID="a3822c1d-660b-4c73-9104-9bde5105e0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 17:54:29.291976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:29.287805 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:29.622090 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:29.621991 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 17:54:29.622090 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:29.622053 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" podUID="a3822c1d-660b-4c73-9104-9bde5105e0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 17:54:30.010164 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.010083 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df" podUID="bfe57717-8ea3-4d44-90bd-13e943adfa80" containerName="payload-processing" containerID="cri-o://d3e7a0d45096171a3e5ad3417990f09f4a1ff5229e25c063db5509648910eb2a" gracePeriod=25 Nov 02 17:54:30.025196 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.025160 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8" podUID="e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd" containerName="payload-pre-processing" containerID="cri-o://9ac07c951ce9dc6e887b28c57e5f8ac7cddd5ab7867dbb574bce0617746034a7" gracePeriod=25 Nov 02 17:54:30.304355 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.302613 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:30.439162 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.439127 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df" Nov 02 17:54:30.472504 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.472470 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8" Nov 02 17:54:30.521698 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.521657 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe57717-8ea3-4d44-90bd-13e943adfa80-kube-api-access-rggqf\" (UniqueName: \"kubernetes.io/projected/bfe57717-8ea3-4d44-90bd-13e943adfa80-kube-api-access-rggqf\") pod \"bfe57717-8ea3-4d44-90bd-13e943adfa80\" (UID: \"bfe57717-8ea3-4d44-90bd-13e943adfa80\") " Nov 02 17:54:30.521887 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.521867 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe57717-8ea3-4d44-90bd-13e943adfa80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe57717-8ea3-4d44-90bd-13e943adfa80-plugins-config-volume\") pod \"bfe57717-8ea3-4d44-90bd-13e943adfa80\" (UID: \"bfe57717-8ea3-4d44-90bd-13e943adfa80\") " Nov 02 17:54:30.523151 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.523123 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe57717-8ea3-4d44-90bd-13e943adfa80-plugins-config-volume" pod "bfe57717-8ea3-4d44-90bd-13e943adfa80" (UID: "bfe57717-8ea3-4d44-90bd-13e943adfa80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:30.525379 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.525348 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe57717-8ea3-4d44-90bd-13e943adfa80-kube-api-access-rggqf" pod "bfe57717-8ea3-4d44-90bd-13e943adfa80" (UID: "bfe57717-8ea3-4d44-90bd-13e943adfa80"). InnerVolumeSpecName "kube-api-access-rggqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:30.621298 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.621213 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 17:54:30.621482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.621302 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" podUID="a3822c1d-660b-4c73-9104-9bde5105e0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 17:54:30.623766 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.623742 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd-plugins-config-volume\") pod \"e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd\" (UID: \"e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd\") " Nov 02 17:54:30.624081 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.624047 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd-kube-api-access-vd8zx\" (UniqueName: \"kubernetes.io/projected/e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd-kube-api-access-vd8zx\") pod \"e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd\" (UID: \"e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd\") " Nov 02 17:54:30.624172 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.624151 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd-plugins-config-volume" pod "e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd" (UID: "e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:30.624644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.624627 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:30.624683 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.624648 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rggqf\" (UniqueName: \"kubernetes.io/projected/bfe57717-8ea3-4d44-90bd-13e943adfa80-kube-api-access-rggqf\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:30.624683 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.624657 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe57717-8ea3-4d44-90bd-13e943adfa80-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:30.626203 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.626170 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd-kube-api-access-vd8zx" pod "e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd" (UID: "e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd"). InnerVolumeSpecName "kube-api-access-vd8zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:30.726333 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.726292 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd8zx\" (UniqueName: \"kubernetes.io/projected/e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd-kube-api-access-vd8zx\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:30.766752 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.766713 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd"] Nov 02 17:54:30.767036 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.767005 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" podUID="2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d" containerName="maas-api" containerID="cri-o://1d842fb803b5f473cdfb8c9183c2e4ff0eaf36e53a58e7a02c5ca95d02ae6a92" gracePeriod=30 Nov 02 17:54:30.868353 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:30.868320 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:31.189636 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.188167 5549 generic.go:356] "Generic (PLEG): container finished" podID="bfe57717-8ea3-4d44-90bd-13e943adfa80" containerID="d3e7a0d45096171a3e5ad3417990f09f4a1ff5229e25c063db5509648910eb2a" exitCode=0 Nov 02 17:54:31.189636 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.188331 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df" event={"ID":"bfe57717-8ea3-4d44-90bd-13e943adfa80","Type":"ContainerDied","Data":"d3e7a0d45096171a3e5ad3417990f09f4a1ff5229e25c063db5509648910eb2a"} Nov 02 17:54:31.189636 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.188383 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df" event={"ID":"bfe57717-8ea3-4d44-90bd-13e943adfa80","Type":"ContainerDied","Data":"abb174c0c7d74eb87a11b90e5bfd2169d8c152ec9ee75d62a8f6e493d968f7b7"} Nov 02 17:54:31.189636 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.188415 5549 scope.go:122] "RemoveContainer" containerID="d3e7a0d45096171a3e5ad3417990f09f4a1ff5229e25c063db5509648910eb2a" Nov 02 17:54:31.189636 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.188606 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df" Nov 02 17:54:31.224052 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.215556 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:31.364509 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.364460 5549 generic.go:356] "Generic (PLEG): container finished" podID="e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd" containerID="9ac07c951ce9dc6e887b28c57e5f8ac7cddd5ab7867dbb574bce0617746034a7" exitCode=0 Nov 02 17:54:31.364760 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.364740 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8" Nov 02 17:54:31.365639 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.365502 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8" event={"ID":"e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd","Type":"ContainerDied","Data":"9ac07c951ce9dc6e887b28c57e5f8ac7cddd5ab7867dbb574bce0617746034a7"} Nov 02 17:54:31.365639 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.365573 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8" event={"ID":"e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd","Type":"ContainerDied","Data":"818c41b99e9f97b004eefca7e227c7095d2d1b2cdd003f5aaae4aef79c0546f9"} Nov 02 17:54:31.384870 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.384845 5549 generic.go:356] "Generic (PLEG): container finished" podID="2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d" containerID="1d842fb803b5f473cdfb8c9183c2e4ff0eaf36e53a58e7a02c5ca95d02ae6a92" exitCode=0 Nov 02 17:54:31.385029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.384926 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" event={"ID":"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d","Type":"ContainerDied","Data":"1d842fb803b5f473cdfb8c9183c2e4ff0eaf36e53a58e7a02c5ca95d02ae6a92"} Nov 02 17:54:31.402589 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.402561 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df"] Nov 02 17:54:31.415502 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.415472 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4e6649-86c856c59-zc2df"] Nov 02 17:54:31.425390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.425356 5549 scope.go:122] "RemoveContainer" containerID="d3e7a0d45096171a3e5ad3417990f09f4a1ff5229e25c063db5509648910eb2a" Nov 02 17:54:31.425729 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:54:31.425704 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e7a0d45096171a3e5ad3417990f09f4a1ff5229e25c063db5509648910eb2a\": container with ID starting with d3e7a0d45096171a3e5ad3417990f09f4a1ff5229e25c063db5509648910eb2a not found: ID does not exist" containerID="d3e7a0d45096171a3e5ad3417990f09f4a1ff5229e25c063db5509648910eb2a" Nov 02 17:54:31.425807 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.425740 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e7a0d45096171a3e5ad3417990f09f4a1ff5229e25c063db5509648910eb2a"} err="failed to get container status \"d3e7a0d45096171a3e5ad3417990f09f4a1ff5229e25c063db5509648910eb2a\": rpc error: code = NotFound desc = could not find container \"d3e7a0d45096171a3e5ad3417990f09f4a1ff5229e25c063db5509648910eb2a\": container with ID starting with d3e7a0d45096171a3e5ad3417990f09f4a1ff5229e25c063db5509648910eb2a not found: ID does not exist" Nov 02 17:54:31.425807 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.425759 5549 scope.go:122] "RemoveContainer" containerID="9ac07c951ce9dc6e887b28c57e5f8ac7cddd5ab7867dbb574bce0617746034a7" Nov 02 17:54:31.431541 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.431517 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" Nov 02 17:54:31.437783 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.437753 5549 scope.go:122] "RemoveContainer" containerID="9ac07c951ce9dc6e887b28c57e5f8ac7cddd5ab7867dbb574bce0617746034a7" Nov 02 17:54:31.438102 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:54:31.438081 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac07c951ce9dc6e887b28c57e5f8ac7cddd5ab7867dbb574bce0617746034a7\": container with ID starting with 9ac07c951ce9dc6e887b28c57e5f8ac7cddd5ab7867dbb574bce0617746034a7 not found: ID does not exist" containerID="9ac07c951ce9dc6e887b28c57e5f8ac7cddd5ab7867dbb574bce0617746034a7" Nov 02 17:54:31.438180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.438111 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac07c951ce9dc6e887b28c57e5f8ac7cddd5ab7867dbb574bce0617746034a7"} err="failed to get container status \"9ac07c951ce9dc6e887b28c57e5f8ac7cddd5ab7867dbb574bce0617746034a7\": rpc error: code = NotFound desc = could not find container \"9ac07c951ce9dc6e887b28c57e5f8ac7cddd5ab7867dbb574bce0617746034a7\": container with ID starting with 9ac07c951ce9dc6e887b28c57e5f8ac7cddd5ab7867dbb574bce0617746034a7 not found: ID does not exist" Nov 02 17:54:31.447303 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.447276 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8"] Nov 02 17:54:31.450239 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.450215 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4e6649-655b959b67-9n7b8"] Nov 02 17:54:31.466791 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.466695 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:31.495123 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.495090 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-maas-api-tls\") pod \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\" (UID: \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\") " Nov 02 17:54:31.495295 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.495256 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-openshift-service-ca\") pod \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\" (UID: \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\") " Nov 02 17:54:31.495437 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.495420 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-metrics-certs\") pod \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\" (UID: \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\") " Nov 02 17:54:31.495575 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.495560 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-kube-api-access-zvcdg\" (UniqueName: \"kubernetes.io/projected/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-kube-api-access-zvcdg\") pod \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\" (UID: \"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d\") " Nov 02 17:54:31.496889 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.496860 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-openshift-service-ca" pod "2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d" (UID: "2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:31.499390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.499360 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-metrics-certs" pod "2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d" (UID: "2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:31.499762 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.499732 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-kube-api-access-zvcdg" pod "2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d" (UID: "2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d"). InnerVolumeSpecName "kube-api-access-zvcdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:31.500284 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.500260 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-maas-api-tls" pod "2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d" (UID: "2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:31.621640 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.606124 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:31.621640 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.606168 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvcdg\" (UniqueName: \"kubernetes.io/projected/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-kube-api-access-zvcdg\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:31.621640 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.606189 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:31.621640 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.606203 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:31.650975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:31.646271 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:32.060982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.060926 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:32.240288 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.240241 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:32.240673 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.240645 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:32.241392 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.241374 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 17:54:32.342755 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.342677 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe57717-8ea3-4d44-90bd-13e943adfa80" path="/var/lib/kubelet/pods/bfe57717-8ea3-4d44-90bd-13e943adfa80/volumes" Nov 02 17:54:32.343365 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.343344 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd" path="/var/lib/kubelet/pods/e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd/volumes" Nov 02 17:54:32.394003 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.393965 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" Nov 02 17:54:32.394188 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.394011 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd" event={"ID":"2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d","Type":"ContainerDied","Data":"76b2fff4cec1f8722410c2063ac4b4a0d27ca0666caaa3e51a065ca71f9a0081"} Nov 02 17:54:32.394188 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.394083 5549 scope.go:122] "RemoveContainer" containerID="1d842fb803b5f473cdfb8c9183c2e4ff0eaf36e53a58e7a02c5ca95d02ae6a92" Nov 02 17:54:32.427621 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.427587 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd"] Nov 02 17:54:32.432765 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.432739 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4e6649-58d49545d8-svprd"] Nov 02 17:54:32.666574 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.666484 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:32.683252 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.683219 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j"] Nov 02 17:54:32.683931 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.683915 5549 cpu_manager.go:414] "containerMap: removing container" podUID="e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd" containerName="payload-pre-processing" Nov 02 17:54:32.683986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.683934 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd" containerName="payload-pre-processing" Nov 02 17:54:32.683986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.683962 5549 cpu_manager.go:414] "containerMap: removing container" podUID="2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d" containerName="maas-api" Nov 02 17:54:32.683986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.683968 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d" containerName="maas-api" Nov 02 17:54:32.683986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.683978 5549 cpu_manager.go:414] "containerMap: removing container" podUID="bfe57717-8ea3-4d44-90bd-13e943adfa80" containerName="payload-processing" Nov 02 17:54:32.683986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.683983 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe57717-8ea3-4d44-90bd-13e943adfa80" containerName="payload-processing" Nov 02 17:54:32.684170 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.684091 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d" containerName="maas-api" Nov 02 17:54:32.684170 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.684102 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe57717-8ea3-4d44-90bd-13e943adfa80" containerName="payload-processing" Nov 02 17:54:32.684170 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.684117 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2d0274f-2eb1-4c72-8140-ae9e8b8d22cd" containerName="payload-pre-processing" Nov 02 17:54:32.688882 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.688865 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j" Nov 02 17:54:32.693797 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.693772 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-56e6db\"" Nov 02 17:54:32.693922 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.693867 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-56e6db-dockercfg-kzh7p\"" Nov 02 17:54:32.697570 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.697545 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j"] Nov 02 17:54:32.712125 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.712096 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk"] Nov 02 17:54:32.717433 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.717403 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk" Nov 02 17:54:32.726804 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.726779 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk"] Nov 02 17:54:32.944031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.839883 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvxpw\" (UniqueName: \"kubernetes.io/projected/1bb46ea2-bbe1-43c6-88b8-346276e9b7d0-kube-api-access-zvxpw\") pod \"payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j\" (UID: \"1bb46ea2-bbe1-43c6-88b8-346276e9b7d0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j" Nov 02 17:54:32.944031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.839985 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156ce927-ebce-4824-ae14-36712fe15948-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk\" (UID: \"156ce927-ebce-4824-ae14-36712fe15948\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk" Nov 02 17:54:32.944031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.840167 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb46ea2-bbe1-43c6-88b8-346276e9b7d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j\" (UID: \"1bb46ea2-bbe1-43c6-88b8-346276e9b7d0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j" Nov 02 17:54:32.944031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.840251 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7wbk\" (UniqueName: \"kubernetes.io/projected/156ce927-ebce-4824-ae14-36712fe15948-kube-api-access-w7wbk\") pod \"payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk\" (UID: \"156ce927-ebce-4824-ae14-36712fe15948\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk" Nov 02 17:54:32.944031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.845877 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:32.944031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.943353 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb46ea2-bbe1-43c6-88b8-346276e9b7d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j\" (UID: \"1bb46ea2-bbe1-43c6-88b8-346276e9b7d0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j" Nov 02 17:54:32.944031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.943458 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wbk\" (UniqueName: \"kubernetes.io/projected/156ce927-ebce-4824-ae14-36712fe15948-kube-api-access-w7wbk\") pod \"payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk\" (UID: \"156ce927-ebce-4824-ae14-36712fe15948\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk" Nov 02 17:54:32.944031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.943576 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvxpw\" (UniqueName: \"kubernetes.io/projected/1bb46ea2-bbe1-43c6-88b8-346276e9b7d0-kube-api-access-zvxpw\") pod \"payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j\" (UID: \"1bb46ea2-bbe1-43c6-88b8-346276e9b7d0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j" Nov 02 17:54:32.944031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.943650 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156ce927-ebce-4824-ae14-36712fe15948-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk\" (UID: \"156ce927-ebce-4824-ae14-36712fe15948\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk" Nov 02 17:54:32.965986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.944736 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156ce927-ebce-4824-ae14-36712fe15948-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk\" (UID: \"156ce927-ebce-4824-ae14-36712fe15948\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk" Nov 02 17:54:32.965986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:32.945702 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb46ea2-bbe1-43c6-88b8-346276e9b7d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j\" (UID: \"1bb46ea2-bbe1-43c6-88b8-346276e9b7d0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j" Nov 02 17:54:33.014390 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:33.014333 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:33.029848 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:33.029704 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvxpw\" (UniqueName: \"kubernetes.io/projected/1bb46ea2-bbe1-43c6-88b8-346276e9b7d0-kube-api-access-zvxpw\") pod \"payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j\" (UID: \"1bb46ea2-bbe1-43c6-88b8-346276e9b7d0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j" Nov 02 17:54:33.030481 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:33.030458 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7wbk\" (UniqueName: \"kubernetes.io/projected/156ce927-ebce-4824-ae14-36712fe15948-kube-api-access-w7wbk\") pod \"payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk\" (UID: \"156ce927-ebce-4824-ae14-36712fe15948\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk" Nov 02 17:54:33.036725 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:33.035224 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk" Nov 02 17:54:33.304558 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:33.304523 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j" Nov 02 17:54:33.420171 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:33.420140 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk"] Nov 02 17:54:33.420465 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:54:33.420444 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156ce927_ebce_4824_ae14_36712fe15948.slice/crio-9c6b810a8aacaae149f88b8809d88b94468501b967d82af5a3ed4dfeef72f809 WatchSource:0}: Error finding container 9c6b810a8aacaae149f88b8809d88b94468501b967d82af5a3ed4dfeef72f809: Status 404 returned error can't find the container with id 9c6b810a8aacaae149f88b8809d88b94468501b967d82af5a3ed4dfeef72f809 Nov 02 17:54:33.586046 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:33.586019 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j"] Nov 02 17:54:33.587355 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:54:33.587328 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb46ea2_bbe1_43c6_88b8_346276e9b7d0.slice/crio-b94aeb10901b6cda1fab462e2af41c4c3fe25d1e389f10d5b9c8433b60e1ee3a WatchSource:0}: Error finding container b94aeb10901b6cda1fab462e2af41c4c3fe25d1e389f10d5b9c8433b60e1ee3a: Status 404 returned error can't find the container with id b94aeb10901b6cda1fab462e2af41c4c3fe25d1e389f10d5b9c8433b60e1ee3a Nov 02 17:54:33.820605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:33.820514 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:34.003158 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:34.003128 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:34.321044 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:34.321009 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d" path="/var/lib/kubelet/pods/2c0ba7ba-b19f-4f6b-9997-1860d1b9ca1d/volumes" Nov 02 17:54:34.400006 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:34.394095 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:34.436129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:34.436063 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j" event={"ID":"1bb46ea2-bbe1-43c6-88b8-346276e9b7d0","Type":"ContainerStarted","Data":"e32d78fb9f22e248563baccca3ee8a9884219a39643f6a95eb0dc5c345a80f14"} Nov 02 17:54:34.436344 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:34.436130 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j" event={"ID":"1bb46ea2-bbe1-43c6-88b8-346276e9b7d0","Type":"ContainerStarted","Data":"b94aeb10901b6cda1fab462e2af41c4c3fe25d1e389f10d5b9c8433b60e1ee3a"} Nov 02 17:54:34.436344 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:34.436198 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j" Nov 02 17:54:34.441579 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:34.440345 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk" event={"ID":"156ce927-ebce-4824-ae14-36712fe15948","Type":"ContainerStarted","Data":"546daec82ef5637cc85a8ccf1c8e51ba95b815a2c771288b54e2e9e53974ef40"} Nov 02 17:54:34.441579 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:34.440403 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk" event={"ID":"156ce927-ebce-4824-ae14-36712fe15948","Type":"ContainerStarted","Data":"9c6b810a8aacaae149f88b8809d88b94468501b967d82af5a3ed4dfeef72f809"} Nov 02 17:54:34.441579 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:34.441407 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk" Nov 02 17:54:34.457848 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:34.457783 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j" podStartSLOduration=2.457762207 podStartE2EDuration="2.457762207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:34.453118077 +0000 UTC m=+1246.241623605" watchObservedRunningTime="2026-11-02 17:54:34.457762207 +0000 UTC m=+1246.246267729" Nov 02 17:54:34.475571 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:34.475519 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk" podStartSLOduration=2.475506749 podStartE2EDuration="2.475506749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:34.471990362 +0000 UTC m=+1246.260495883" watchObservedRunningTime="2026-11-02 17:54:34.475506749 +0000 UTC m=+1246.264012277" Nov 02 17:54:34.748482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:34.748445 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:34.910976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:34.910106 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:35.535933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:35.535899 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:35.758395 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:35.757843 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:36.521008 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:36.520637 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:37.247444 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:37.244003 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:37.920630 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:37.915375 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:38.636000 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:38.631476 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:39.193624 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.193588 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp"] Nov 02 17:54:39.198496 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.198459 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:54:39.206923 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.206898 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-56e6db-serving-cert\"" Nov 02 17:54:39.210724 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.210698 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp"] Nov 02 17:54:39.254118 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.254085 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b078f1c1-1391-4686-96b3-35b15b027285-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp\" (UID: \"b078f1c1-1391-4686-96b3-35b15b027285\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:54:39.254299 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.254149 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbbzx\" (UniqueName: \"kubernetes.io/projected/b078f1c1-1391-4686-96b3-35b15b027285-kube-api-access-tbbzx\") pod \"maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp\" (UID: \"b078f1c1-1391-4686-96b3-35b15b027285\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:54:39.254345 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.254320 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b078f1c1-1391-4686-96b3-35b15b027285-maas-api-tls\") pod \"maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp\" (UID: \"b078f1c1-1391-4686-96b3-35b15b027285\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:54:39.254475 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.254455 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b078f1c1-1391-4686-96b3-35b15b027285-metrics-certs\") pod \"maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp\" (UID: \"b078f1c1-1391-4686-96b3-35b15b027285\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:54:39.298931 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.298895 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:39.362104 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.362063 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b078f1c1-1391-4686-96b3-35b15b027285-metrics-certs\") pod \"maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp\" (UID: \"b078f1c1-1391-4686-96b3-35b15b027285\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:54:39.362418 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.362390 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b078f1c1-1391-4686-96b3-35b15b027285-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp\" (UID: \"b078f1c1-1391-4686-96b3-35b15b027285\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:54:39.365976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.362558 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbbzx\" (UniqueName: \"kubernetes.io/projected/b078f1c1-1391-4686-96b3-35b15b027285-kube-api-access-tbbzx\") pod \"maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp\" (UID: \"b078f1c1-1391-4686-96b3-35b15b027285\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:54:39.366940 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.366910 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b078f1c1-1391-4686-96b3-35b15b027285-maas-api-tls\") pod \"maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp\" (UID: \"b078f1c1-1391-4686-96b3-35b15b027285\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:54:39.377983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.372245 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b078f1c1-1391-4686-96b3-35b15b027285-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp\" (UID: \"b078f1c1-1391-4686-96b3-35b15b027285\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:54:39.379642 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:54:39.379615 5549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-serving-cert: secret "maas-api-e2e-praxis-b-56e6db-serving-cert" not found Nov 02 17:54:39.397142 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:54:39.397115 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b078f1c1-1391-4686-96b3-35b15b027285-maas-api-tls podName:b078f1c1-1391-4686-96b3-35b15b027285 nodeName:}" failed. No retries permitted until 2026-11-02 17:54:39.89708632 +0000 UTC m=+1251.685591831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b078f1c1-1391-4686-96b3-35b15b027285-maas-api-tls") pod "maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" (UID: "b078f1c1-1391-4686-96b3-35b15b027285") : secret "maas-api-e2e-praxis-b-56e6db-serving-cert" not found Nov 02 17:54:39.397626 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.392733 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbbzx\" (UniqueName: \"kubernetes.io/projected/b078f1c1-1391-4686-96b3-35b15b027285-kube-api-access-tbbzx\") pod \"maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp\" (UID: \"b078f1c1-1391-4686-96b3-35b15b027285\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:54:39.410661 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.410628 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b078f1c1-1391-4686-96b3-35b15b027285-metrics-certs\") pod \"maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp\" (UID: \"b078f1c1-1391-4686-96b3-35b15b027285\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:54:39.440780 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.440745 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:39.607921 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.607876 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:39.983423 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.983387 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b078f1c1-1391-4686-96b3-35b15b027285-maas-api-tls\") pod \"maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp\" (UID: \"b078f1c1-1391-4686-96b3-35b15b027285\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:54:39.986016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:39.985991 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b078f1c1-1391-4686-96b3-35b15b027285-maas-api-tls\") pod \"maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp\" (UID: \"b078f1c1-1391-4686-96b3-35b15b027285\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:54:40.121402 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:40.121359 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:54:40.153985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:40.151603 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:40.468225 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:40.468196 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp"] Nov 02 17:54:40.469008 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:54:40.468979 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb078f1c1_1391_4686_96b3_35b15b027285.slice/crio-552e5209548161a94f0cec2da90b5eb6f77a9a1bd141601e8351bbf3282fb97d WatchSource:0}: Error finding container 552e5209548161a94f0cec2da90b5eb6f77a9a1bd141601e8351bbf3282fb97d: Status 404 returned error can't find the container with id 552e5209548161a94f0cec2da90b5eb6f77a9a1bd141601e8351bbf3282fb97d Nov 02 17:54:40.490867 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:40.490796 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" event={"ID":"b078f1c1-1391-4686-96b3-35b15b027285","Type":"ContainerStarted","Data":"552e5209548161a94f0cec2da90b5eb6f77a9a1bd141601e8351bbf3282fb97d"} Nov 02 17:54:40.989991 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:40.989733 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:41.499308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:41.499269 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" event={"ID":"b078f1c1-1391-4686-96b3-35b15b027285","Type":"ContainerStarted","Data":"5a93165894addde4ef36c1460f6819d092dc47cfb17889ceefc7ec77688ccbd1"} Nov 02 17:54:41.499689 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:41.499333 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:54:41.517239 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:41.517189 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" podStartSLOduration=1.939492366 podStartE2EDuration="2.517176009s" totalImagesPullingTime="577.683643ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:41.516051187 +0000 UTC m=+1253.304556730" watchObservedRunningTime="2026-11-02 17:54:41.517176009 +0000 UTC m=+1253.305681580" Nov 02 17:54:41.835036 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:41.834919 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:42.567013 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:42.566024 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:42.658658 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:42.658628 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z"] Nov 02 17:54:42.675703 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:42.675668 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn"] Nov 02 17:54:42.821489 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:42.821393 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:42.971979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:42.971005 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:43.276176 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:43.276132 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:43.474400 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:43.474360 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:44.195622 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:44.195578 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:44.952050 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:44.951886 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:45.785685 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:45.785648 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:46.454830 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:46.454767 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:46.480438 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:46.480405 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j" Nov 02 17:54:46.483213 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:46.483185 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk" Nov 02 17:54:47.131010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.130943 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb"] Nov 02 17:54:47.131462 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.131216 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" podUID="36810b4f-4409-4a63-96fa-8c4565058368" containerName="maas-api" containerID="cri-o://c073892f66176455dfd091074f698eee87dd99233f7d724b0742f22d519af0cf" gracePeriod=30 Nov 02 17:54:47.236252 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.236215 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:47.260892 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.260857 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:54:47.569344 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.569306 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:54:47.608496 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.608398 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:47.614135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.614100 5549 generic.go:356] "Generic (PLEG): container finished" podID="36810b4f-4409-4a63-96fa-8c4565058368" containerID="c073892f66176455dfd091074f698eee87dd99233f7d724b0742f22d519af0cf" exitCode=0 Nov 02 17:54:47.614359 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.614326 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" event={"ID":"36810b4f-4409-4a63-96fa-8c4565058368","Type":"ContainerDied","Data":"c073892f66176455dfd091074f698eee87dd99233f7d724b0742f22d519af0cf"} Nov 02 17:54:47.682155 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:54:47.682114 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:54:47.682512 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.682483 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" podUID="34b73845-9bed-45b5-b698-12759713f944" containerName="payload-processing" containerID="cri-o://f1b20e39185f9185d922b4fb586d649c2fa26e9f593484feaf60339d185e3ba2" gracePeriod=25 Nov 02 17:54:47.684077 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.682741 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn" podUID="f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d" containerName="payload-pre-processing" containerID="cri-o://65f253ae6df5be8111db4829ec9cb3968246f1b48138d74fc32c6e1b8304e087" gracePeriod=25 Nov 02 17:54:47.692826 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.692739 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" Nov 02 17:54:47.797331 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.797293 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36810b4f-4409-4a63-96fa-8c4565058368-kube-api-access-vwsqb\" (UniqueName: \"kubernetes.io/projected/36810b4f-4409-4a63-96fa-8c4565058368-kube-api-access-vwsqb\") pod \"36810b4f-4409-4a63-96fa-8c4565058368\" (UID: \"36810b4f-4409-4a63-96fa-8c4565058368\") " Nov 02 17:54:47.797520 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.797355 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36810b4f-4409-4a63-96fa-8c4565058368-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36810b4f-4409-4a63-96fa-8c4565058368-maas-api-tls\") pod \"36810b4f-4409-4a63-96fa-8c4565058368\" (UID: \"36810b4f-4409-4a63-96fa-8c4565058368\") " Nov 02 17:54:47.797520 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.797469 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36810b4f-4409-4a63-96fa-8c4565058368-metrics-certs\" (UniqueName: \"kubernetes.io/secret/36810b4f-4409-4a63-96fa-8c4565058368-metrics-certs\") pod \"36810b4f-4409-4a63-96fa-8c4565058368\" (UID: \"36810b4f-4409-4a63-96fa-8c4565058368\") " Nov 02 17:54:47.797662 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.797520 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36810b4f-4409-4a63-96fa-8c4565058368-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36810b4f-4409-4a63-96fa-8c4565058368-openshift-service-ca\") pod \"36810b4f-4409-4a63-96fa-8c4565058368\" (UID: \"36810b4f-4409-4a63-96fa-8c4565058368\") " Nov 02 17:54:47.798003 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.797934 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36810b4f-4409-4a63-96fa-8c4565058368-openshift-service-ca" pod "36810b4f-4409-4a63-96fa-8c4565058368" (UID: "36810b4f-4409-4a63-96fa-8c4565058368"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:47.798173 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.798150 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36810b4f-4409-4a63-96fa-8c4565058368-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:47.799551 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.799526 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36810b4f-4409-4a63-96fa-8c4565058368-maas-api-tls" pod "36810b4f-4409-4a63-96fa-8c4565058368" (UID: "36810b4f-4409-4a63-96fa-8c4565058368"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:47.799650 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.799568 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36810b4f-4409-4a63-96fa-8c4565058368-kube-api-access-vwsqb" pod "36810b4f-4409-4a63-96fa-8c4565058368" (UID: "36810b4f-4409-4a63-96fa-8c4565058368"). InnerVolumeSpecName "kube-api-access-vwsqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:47.799785 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.799761 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36810b4f-4409-4a63-96fa-8c4565058368-metrics-certs" pod "36810b4f-4409-4a63-96fa-8c4565058368" (UID: "36810b4f-4409-4a63-96fa-8c4565058368"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:47.912995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.911989 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36810b4f-4409-4a63-96fa-8c4565058368-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:47.912995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.912029 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwsqb\" (UniqueName: \"kubernetes.io/projected/36810b4f-4409-4a63-96fa-8c4565058368-kube-api-access-vwsqb\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:47.912995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:47.912047 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36810b4f-4409-4a63-96fa-8c4565058368-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:48.266619 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.266226 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn" Nov 02 17:54:48.497365 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.497325 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d-kube-api-access-4lcx6\" (UniqueName: \"kubernetes.io/projected/f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d-kube-api-access-4lcx6\") pod \"f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d\" (UID: \"f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d\") " Nov 02 17:54:48.501330 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.501291 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d-plugins-config-volume\") pod \"f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d\" (UID: \"f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d\") " Nov 02 17:54:48.553007 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.552967 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d-plugins-config-volume" pod "f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d" (UID: "f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:48.610877 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.610829 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:48.630563 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.630486 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d-kube-api-access-4lcx6" pod "f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d" (UID: "f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d"). InnerVolumeSpecName "kube-api-access-4lcx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:48.633787 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.632131 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:48.750055 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.715270 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" Nov 02 17:54:48.750055 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.717679 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lcx6\" (UniqueName: \"kubernetes.io/projected/f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d-kube-api-access-4lcx6\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:48.750055 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.733029 5549 generic.go:356] "Generic (PLEG): container finished" podID="34b73845-9bed-45b5-b698-12759713f944" containerID="f1b20e39185f9185d922b4fb586d649c2fa26e9f593484feaf60339d185e3ba2" exitCode=0 Nov 02 17:54:48.750055 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.733454 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" event={"ID":"34b73845-9bed-45b5-b698-12759713f944","Type":"ContainerDied","Data":"f1b20e39185f9185d922b4fb586d649c2fa26e9f593484feaf60339d185e3ba2"} Nov 02 17:54:48.750055 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.733513 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" event={"ID":"34b73845-9bed-45b5-b698-12759713f944","Type":"ContainerDied","Data":"b28f80e140ec2b2382a76b945bcd65e6c291b7116fcd043e42ed605970623114"} Nov 02 17:54:48.750055 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.733557 5549 scope.go:122] "RemoveContainer" containerID="f1b20e39185f9185d922b4fb586d649c2fa26e9f593484feaf60339d185e3ba2" Nov 02 17:54:48.921610 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.872007 5549 scope.go:122] "RemoveContainer" containerID="f1b20e39185f9185d922b4fb586d649c2fa26e9f593484feaf60339d185e3ba2" Nov 02 17:54:48.921610 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:54:48.911350 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b20e39185f9185d922b4fb586d649c2fa26e9f593484feaf60339d185e3ba2\": container with ID starting with f1b20e39185f9185d922b4fb586d649c2fa26e9f593484feaf60339d185e3ba2 not found: ID does not exist" containerID="f1b20e39185f9185d922b4fb586d649c2fa26e9f593484feaf60339d185e3ba2" Nov 02 17:54:48.921610 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.911392 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b20e39185f9185d922b4fb586d649c2fa26e9f593484feaf60339d185e3ba2"} err="failed to get container status \"f1b20e39185f9185d922b4fb586d649c2fa26e9f593484feaf60339d185e3ba2\": rpc error: code = NotFound desc = could not find container \"f1b20e39185f9185d922b4fb586d649c2fa26e9f593484feaf60339d185e3ba2\": container with ID starting with f1b20e39185f9185d922b4fb586d649c2fa26e9f593484feaf60339d185e3ba2 not found: ID does not exist" Nov 02 17:54:48.921610 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.911644 5549 generic.go:356] "Generic (PLEG): container finished" podID="f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d" containerID="65f253ae6df5be8111db4829ec9cb3968246f1b48138d74fc32c6e1b8304e087" exitCode=0 Nov 02 17:54:48.921610 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.911781 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn" event={"ID":"f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d","Type":"ContainerDied","Data":"65f253ae6df5be8111db4829ec9cb3968246f1b48138d74fc32c6e1b8304e087"} Nov 02 17:54:48.921610 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.911831 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn" event={"ID":"f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d","Type":"ContainerDied","Data":"0d68b8545f81a0ca6c9d34f745890db10cbf65efe75b65751b6dcd96845276df"} Nov 02 17:54:48.921610 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.911860 5549 scope.go:122] "RemoveContainer" containerID="65f253ae6df5be8111db4829ec9cb3968246f1b48138d74fc32c6e1b8304e087" Nov 02 17:54:48.935976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:48.925110 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn" Nov 02 17:54:49.030059 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:49.009994 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b73845-9bed-45b5-b698-12759713f944-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b73845-9bed-45b5-b698-12759713f944-plugins-config-volume\") pod \"34b73845-9bed-45b5-b698-12759713f944\" (UID: \"34b73845-9bed-45b5-b698-12759713f944\") " Nov 02 17:54:49.030059 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:49.010214 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b73845-9bed-45b5-b698-12759713f944-kube-api-access-slkj4\" (UniqueName: \"kubernetes.io/projected/34b73845-9bed-45b5-b698-12759713f944-kube-api-access-slkj4\") pod \"34b73845-9bed-45b5-b698-12759713f944\" (UID: \"34b73845-9bed-45b5-b698-12759713f944\") " Nov 02 17:54:49.112209 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:49.112119 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b73845-9bed-45b5-b698-12759713f944-plugins-config-volume" pod "34b73845-9bed-45b5-b698-12759713f944" (UID: "34b73845-9bed-45b5-b698-12759713f944"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:49.143877 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:49.140432 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" event={"ID":"36810b4f-4409-4a63-96fa-8c4565058368","Type":"ContainerDied","Data":"a64d28b229e42a10c36b46e090815cb9c885cf98871f04d352eb7fb8c4a0a9de"} Nov 02 17:54:49.143877 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:49.140561 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb" Nov 02 17:54:49.201138 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:49.201073 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn"] Nov 02 17:54:49.208942 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:49.208512 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e198eb-85dcddcc5f-qnbrn"] Nov 02 17:54:49.254116 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:49.253757 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b73845-9bed-45b5-b698-12759713f944-kube-api-access-slkj4" pod "34b73845-9bed-45b5-b698-12759713f944" (UID: "34b73845-9bed-45b5-b698-12759713f944"). InnerVolumeSpecName "kube-api-access-slkj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:49.265031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:49.264933 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slkj4\" (UniqueName: \"kubernetes.io/projected/34b73845-9bed-45b5-b698-12759713f944-kube-api-access-slkj4\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:49.265031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:49.264999 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b73845-9bed-45b5-b698-12759713f944-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:49.266883 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:49.266456 5549 scope.go:122] "RemoveContainer" containerID="65f253ae6df5be8111db4829ec9cb3968246f1b48138d74fc32c6e1b8304e087" Nov 02 17:54:49.281428 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:54:49.280660 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f253ae6df5be8111db4829ec9cb3968246f1b48138d74fc32c6e1b8304e087\": container with ID starting with 65f253ae6df5be8111db4829ec9cb3968246f1b48138d74fc32c6e1b8304e087 not found: ID does not exist" containerID="65f253ae6df5be8111db4829ec9cb3968246f1b48138d74fc32c6e1b8304e087" Nov 02 17:54:49.281428 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:49.280707 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f253ae6df5be8111db4829ec9cb3968246f1b48138d74fc32c6e1b8304e087"} err="failed to get container status \"65f253ae6df5be8111db4829ec9cb3968246f1b48138d74fc32c6e1b8304e087\": rpc error: code = NotFound desc = could not find container \"65f253ae6df5be8111db4829ec9cb3968246f1b48138d74fc32c6e1b8304e087\": container with ID starting with 65f253ae6df5be8111db4829ec9cb3968246f1b48138d74fc32c6e1b8304e087 not found: ID does not exist" Nov 02 17:54:49.281428 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:49.280735 5549 scope.go:122] "RemoveContainer" containerID="c073892f66176455dfd091074f698eee87dd99233f7d724b0742f22d519af0cf" Nov 02 17:54:49.300804 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:49.300763 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb"] Nov 02 17:54:49.342073 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:49.342040 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e198eb-86679f676c-8nkgb"] Nov 02 17:54:49.551937 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:49.551834 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" podUID="34b73845-9bed-45b5-b698-12759713f944" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.156:50052\" within 1s: context deadline exceeded" Nov 02 17:54:49.567974 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:54:49.552521 5549 logging.go:55] [core] [Channel #3094 SubChannel #3095]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.156:50052", ServerName: "10.133.0.156:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.156:50052: operation was canceled" Nov 02 17:54:50.164551 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:50.164515 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:50.222979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:50.222310 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z" Nov 02 17:54:50.319616 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:50.319567 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z"] Nov 02 17:54:50.325979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:50.325169 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 17:54:50.328000 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:50.327581 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:50.362989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:50.358791 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36810b4f-4409-4a63-96fa-8c4565058368" path="/var/lib/kubelet/pods/36810b4f-4409-4a63-96fa-8c4565058368/volumes" Nov 02 17:54:50.364514 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:50.363881 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d" path="/var/lib/kubelet/pods/f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d/volumes" Nov 02 17:54:50.364514 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:50.364365 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e198eb-7bdccc957-tj97z"] Nov 02 17:54:50.364857 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:50.364828 5549 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" podUID="3827bd23-547c-4ae6-83c5-90c71860f814" containerName="main" probeResult="failure" output="Get \"https://10.133.0.154:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:50.365283 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:50.365256 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" podUID="3827bd23-547c-4ae6-83c5-90c71860f814" containerName="main" probeResult="failure" output="Get \"https://10.133.0.154:8000/ready\": context deadline exceeded" Nov 02 17:54:50.369464 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:50.369439 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:50.686974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:50.683461 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:50.825578 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:50.825536 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:51.104082 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:51.079046 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:51.141989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:51.133269 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 02 17:54:51.172998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:51.164690 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:51.182456 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:51.182093 5549 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:51.182456 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:51.182163 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" podUID="034a5399-ee48-4bfb-885c-0bcfd316985e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:51.350979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:51.329097 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 17:54:51.350979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:51.329460 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 17:54:51.350979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:51.330094 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:54:51.350979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:51.330149 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:54:51.410073 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:51.382034 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" podUID="3827bd23-547c-4ae6-83c5-90c71860f814" containerName="main" probeResult="failure" output="Get \"https://10.133.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:51.410073 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:51.382478 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:54:51.629016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:51.614520 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:51.964088 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:51.964036 5549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:52.101975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:52.089318 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 17:54:52.101975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:52.089754 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:54:52.143823 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:52.141839 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:52.143823 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:52.142574 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:54:52.361703 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:52.361091 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 17:54:52.362225 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:52.361047 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:52.397843 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:52.397117 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" podUID="3827bd23-547c-4ae6-83c5-90c71860f814" containerName="main" probeResult="failure" output="Get \"https://10.133.0.154:8000/ready\": context deadline exceeded" Nov 02 17:54:52.410276 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:52.410240 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b73845-9bed-45b5-b698-12759713f944" path="/var/lib/kubelet/pods/34b73845-9bed-45b5-b698-12759713f944/volumes" Nov 02 17:54:52.412591 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:52.412567 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:54:52.412811 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:52.411737 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:53.009977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:52.996482 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 17:54:53.060602 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:53.060556 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 17:54:53.094657 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:54:53.092006 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36810b4f_4409_4a63_96fa_8c4565058368.slice/crio-a64d28b229e42a10c36b46e090815cb9c885cf98871f04d352eb7fb8c4a0a9de\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf586b4c3_d237_4d83_a2bb_5fbc85c3bc6d.slice/crio-0d68b8545f81a0ca6c9d34f745890db10cbf65efe75b65751b6dcd96845276df\": RecentStats: unable to find data in memory cache]" Nov 02 17:54:53.094657 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:53.092838 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:54:53.129307 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:53.129282 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:54:53.280591 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:53.280496 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:53.441049 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:53.441013 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:53.805032 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:54:53.794598 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf586b4c3_d237_4d83_a2bb_5fbc85c3bc6d.slice/crio-0d68b8545f81a0ca6c9d34f745890db10cbf65efe75b65751b6dcd96845276df\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36810b4f_4409_4a63_96fa_8c4565058368.slice/crio-a64d28b229e42a10c36b46e090815cb9c885cf98871f04d352eb7fb8c4a0a9de\": RecentStats: unable to find data in memory cache]" Nov 02 17:54:53.984970 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:53.984912 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:54.739184 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:54.739136 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:55.528930 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:55.528866 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:56.393593 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:56.393553 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:57.255283 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:57.255242 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:58.081077 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:58.075502 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:58.738173 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:58.738129 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:58.942922 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:58.942886 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:59.807036 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:54:59.792606 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:00.561522 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:00.561302 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:00.799963 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:00.799924 5549 generic.go:356] "Generic (PLEG): container finished" podID="c8d28e37-d453-4fb6-878e-68b551f73dde" containerID="42805b3aacbf92b7dd51e641348c5c1609f271d8f994d885c42f1c10843542fb" exitCode=0 Nov 02 17:55:00.800167 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:00.800000 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" event={"ID":"c8d28e37-d453-4fb6-878e-68b551f73dde","Type":"ContainerDied","Data":"42805b3aacbf92b7dd51e641348c5c1609f271d8f994d885c42f1c10843542fb"} Nov 02 17:55:00.800502 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:00.800484 5549 scope.go:122] "RemoveContainer" containerID="42805b3aacbf92b7dd51e641348c5c1609f271d8f994d885c42f1c10843542fb" Nov 02 17:55:00.802101 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:00.802070 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:55:00.802187 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:00.802143 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:55:00.802187 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:00.802180 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:55:01.190969 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:01.190829 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:01.396746 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:01.396705 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:01.633020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:01.632581 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:01.814409 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:01.814376 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" event={"ID":"c8d28e37-d453-4fb6-878e-68b551f73dde","Type":"ContainerStarted","Data":"259b3b3edf67dbade9f08d1f48a8c2fd5cb4938b902ef7ec74b8679eb236a859"} Nov 02 17:55:01.916685 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:01.916591 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:02.305279 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:02.305237 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:55:02.306712 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:02.306683 5549 patch_prober.go:29] interesting pod/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:55:02.306865 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:02.306738 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" podUID="c8d28e37-d453-4fb6-878e-68b551f73dde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 17:55:02.566713 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:02.566619 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:03.063878 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:03.063836 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf586b4c3_d237_4d83_a2bb_5fbc85c3bc6d.slice/crio-0d68b8545f81a0ca6c9d34f745890db10cbf65efe75b65751b6dcd96845276df\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36810b4f_4409_4a63_96fa_8c4565058368.slice/crio-a64d28b229e42a10c36b46e090815cb9c885cf98871f04d352eb7fb8c4a0a9de\": RecentStats: unable to find data in memory cache]" Nov 02 17:55:03.064121 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:03.064011 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36810b4f_4409_4a63_96fa_8c4565058368.slice/crio-a64d28b229e42a10c36b46e090815cb9c885cf98871f04d352eb7fb8c4a0a9de\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf586b4c3_d237_4d83_a2bb_5fbc85c3bc6d.slice/crio-0d68b8545f81a0ca6c9d34f745890db10cbf65efe75b65751b6dcd96845276df\": RecentStats: unable to find data in memory cache]" Nov 02 17:55:03.134058 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:03.134018 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36810b4f_4409_4a63_96fa_8c4565058368.slice/crio-a64d28b229e42a10c36b46e090815cb9c885cf98871f04d352eb7fb8c4a0a9de\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf586b4c3_d237_4d83_a2bb_5fbc85c3bc6d.slice/crio-0d68b8545f81a0ca6c9d34f745890db10cbf65efe75b65751b6dcd96845276df\": RecentStats: unable to find data in memory cache]" Nov 02 17:55:03.305668 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:03.305626 5549 patch_prober.go:29] interesting pod/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:55:03.306094 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:03.305696 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" podUID="c8d28e37-d453-4fb6-878e-68b551f73dde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 17:55:03.563977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:03.560159 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:04.247469 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:04.247292 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:04.309393 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:04.309337 5549 patch_prober.go:29] interesting pod/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:55:04.310035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:04.309997 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" podUID="c8d28e37-d453-4fb6-878e-68b551f73dde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 17:55:04.617542 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:04.617456 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:55:05.170947 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:05.170909 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:05.305484 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:05.305448 5549 patch_prober.go:29] interesting pod/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:55:05.305663 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:05.305503 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" podUID="c8d28e37-d453-4fb6-878e-68b551f73dde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 17:55:05.927830 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:05.927794 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:06.305521 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:06.305480 5549 patch_prober.go:29] interesting pod/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:55:06.305704 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:06.305547 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" podUID="c8d28e37-d453-4fb6-878e-68b551f73dde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 17:55:06.729669 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:06.729559 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:07.196178 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.196147 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml"] Nov 02 17:55:07.196916 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.196885 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" podUID="d26e1b21-d8a8-4bd6-bc52-0052582bfb9d" containerName="maas-api" containerID="cri-o://c691557de8307107de58aabc62c063ceda0562ca4f8c429f6ddd4b9270beb4dd" gracePeriod=30 Nov 02 17:55:07.303440 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.303405 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:07.326336 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.326303 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:55:07.328594 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.328564 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:55:07.596356 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.595864 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" Nov 02 17:55:07.645321 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.635508 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:07.684997 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.647255 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-kube-api-access-wvrm5\" (UniqueName: \"kubernetes.io/projected/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-kube-api-access-wvrm5\") pod \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\" (UID: \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\") " Nov 02 17:55:07.684997 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.666157 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-maas-api-tls\") pod \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\" (UID: \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\") " Nov 02 17:55:07.684997 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.666347 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-openshift-service-ca\") pod \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\" (UID: \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\") " Nov 02 17:55:07.684997 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.666468 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-metrics-certs\") pod \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\" (UID: \"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d\") " Nov 02 17:55:07.684997 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.683719 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-openshift-service-ca" pod "d26e1b21-d8a8-4bd6-bc52-0052582bfb9d" (UID: "d26e1b21-d8a8-4bd6-bc52-0052582bfb9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:07.689615 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.689106 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-metrics-certs" pod "d26e1b21-d8a8-4bd6-bc52-0052582bfb9d" (UID: "d26e1b21-d8a8-4bd6-bc52-0052582bfb9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:07.712005 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.710325 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-maas-api-tls" pod "d26e1b21-d8a8-4bd6-bc52-0052582bfb9d" (UID: "d26e1b21-d8a8-4bd6-bc52-0052582bfb9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:07.716222 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.716116 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-kube-api-access-wvrm5" pod "d26e1b21-d8a8-4bd6-bc52-0052582bfb9d" (UID: "d26e1b21-d8a8-4bd6-bc52-0052582bfb9d"). InnerVolumeSpecName "kube-api-access-wvrm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:07.782895 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.769082 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvrm5\" (UniqueName: \"kubernetes.io/projected/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-kube-api-access-wvrm5\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:07.782895 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.769120 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:07.782895 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.769137 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:07.782895 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.769151 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:07.855061 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.855029 5549 generic.go:356] "Generic (PLEG): container finished" podID="d26e1b21-d8a8-4bd6-bc52-0052582bfb9d" containerID="c691557de8307107de58aabc62c063ceda0562ca4f8c429f6ddd4b9270beb4dd" exitCode=0 Nov 02 17:55:07.855237 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.855080 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" Nov 02 17:55:07.855237 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.855107 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" event={"ID":"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d","Type":"ContainerDied","Data":"c691557de8307107de58aabc62c063ceda0562ca4f8c429f6ddd4b9270beb4dd"} Nov 02 17:55:07.855237 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.855188 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml" event={"ID":"d26e1b21-d8a8-4bd6-bc52-0052582bfb9d","Type":"ContainerDied","Data":"b8135b0ce1ac68003a7b522aa506202ac5ce01eda557d39bf3fae3e94402b895"} Nov 02 17:55:07.855237 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.855211 5549 scope.go:122] "RemoveContainer" containerID="c691557de8307107de58aabc62c063ceda0562ca4f8c429f6ddd4b9270beb4dd" Nov 02 17:55:07.866288 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.866267 5549 scope.go:122] "RemoveContainer" containerID="c691557de8307107de58aabc62c063ceda0562ca4f8c429f6ddd4b9270beb4dd" Nov 02 17:55:07.866567 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:07.866549 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c691557de8307107de58aabc62c063ceda0562ca4f8c429f6ddd4b9270beb4dd\": container with ID starting with c691557de8307107de58aabc62c063ceda0562ca4f8c429f6ddd4b9270beb4dd not found: ID does not exist" containerID="c691557de8307107de58aabc62c063ceda0562ca4f8c429f6ddd4b9270beb4dd" Nov 02 17:55:07.866619 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.866574 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c691557de8307107de58aabc62c063ceda0562ca4f8c429f6ddd4b9270beb4dd"} err="failed to get container status \"c691557de8307107de58aabc62c063ceda0562ca4f8c429f6ddd4b9270beb4dd\": rpc error: code = NotFound desc = could not find container \"c691557de8307107de58aabc62c063ceda0562ca4f8c429f6ddd4b9270beb4dd\": container with ID starting with c691557de8307107de58aabc62c063ceda0562ca4f8c429f6ddd4b9270beb4dd not found: ID does not exist" Nov 02 17:55:07.880194 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.880169 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml"] Nov 02 17:55:07.884285 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:07.884261 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfd356-7cc9cbfb56-vk5ml"] Nov 02 17:55:08.256169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:08.256137 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:08.320684 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:08.320651 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26e1b21-d8a8-4bd6-bc52-0052582bfb9d" path="/var/lib/kubelet/pods/d26e1b21-d8a8-4bd6-bc52-0052582bfb9d/volumes" Nov 02 17:55:08.410542 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:08.410504 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:08.838412 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:08.838372 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf586b4c3_d237_4d83_a2bb_5fbc85c3bc6d.slice/crio-0d68b8545f81a0ca6c9d34f745890db10cbf65efe75b65751b6dcd96845276df\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36810b4f_4409_4a63_96fa_8c4565058368.slice/crio-a64d28b229e42a10c36b46e090815cb9c885cf98871f04d352eb7fb8c4a0a9de\": RecentStats: unable to find data in memory cache]" Nov 02 17:55:09.021234 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:09.021200 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:09.899333 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:09.899299 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:10.829022 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:10.828905 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:12.299979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:12.299055 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:12.410976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:12.407616 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:12.976986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:12.976883 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 17:55:13.037573 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:13.036090 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:13.093986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:13.077561 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:13.093986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:13.086133 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:13.204978 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:13.204897 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36810b4f_4409_4a63_96fa_8c4565058368.slice/crio-a64d28b229e42a10c36b46e090815cb9c885cf98871f04d352eb7fb8c4a0a9de\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf586b4c3_d237_4d83_a2bb_5fbc85c3bc6d.slice/crio-0d68b8545f81a0ca6c9d34f745890db10cbf65efe75b65751b6dcd96845276df\": RecentStats: unable to find data in memory cache]" Nov 02 17:55:13.780045 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:13.780005 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:14.300909 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:14.300874 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:14.649326 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:14.649239 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:15.610197 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:15.610157 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:15.821064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:15.821031 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:16.225876 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:16.225845 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:16.454743 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:16.454704 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878"] Nov 02 17:55:16.455274 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:16.455236 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" podUID="c8d28e37-d453-4fb6-878e-68b551f73dde" containerName="istio-proxy" containerID="cri-o://259b3b3edf67dbade9f08d1f48a8c2fd5cb4938b902ef7ec74b8679eb236a859" gracePeriod=30 Nov 02 17:55:16.603051 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:16.600482 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:17.011425 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:17.011354 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:17.395850 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:17.395768 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:17.586947 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:17.586905 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:55:18.073659 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:18.073608 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:18.856979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:18.855885 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:19.181896 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:19.181816 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh"] Nov 02 17:55:19.311722 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:19.311679 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:19.773770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:19.773729 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:20.467370 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:20.467322 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:21.358631 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.358396 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:21.765235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.765207 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:55:21.921322 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.921237 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8d28e37-d453-4fb6-878e-68b551f73dde-kube-api-access-mltdc\" (UniqueName: \"kubernetes.io/projected/c8d28e37-d453-4fb6-878e-68b551f73dde-kube-api-access-mltdc\") pod \"c8d28e37-d453-4fb6-878e-68b551f73dde\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " Nov 02 17:55:21.921322 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.921307 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-envoy\") pod \"c8d28e37-d453-4fb6-878e-68b551f73dde\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " Nov 02 17:55:21.921571 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.921339 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-workload-socket\") pod \"c8d28e37-d453-4fb6-878e-68b551f73dde\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " Nov 02 17:55:21.921571 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.921373 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8d28e37-d453-4fb6-878e-68b551f73dde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d28e37-d453-4fb6-878e-68b551f73dde-istiod-ca-cert\") pod \"c8d28e37-d453-4fb6-878e-68b551f73dde\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " Nov 02 17:55:21.921571 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.921403 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-podinfo\") pod \"c8d28e37-d453-4fb6-878e-68b551f73dde\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " Nov 02 17:55:21.921571 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.921508 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-data\") pod \"c8d28e37-d453-4fb6-878e-68b551f73dde\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " Nov 02 17:55:21.921759 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.921608 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-token\" (UniqueName: \"kubernetes.io/projected/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-token\") pod \"c8d28e37-d453-4fb6-878e-68b551f73dde\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " Nov 02 17:55:21.921759 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.921640 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-workload-certs\") pod \"c8d28e37-d453-4fb6-878e-68b551f73dde\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " Nov 02 17:55:21.921759 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.921713 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-credential-socket\") pod \"c8d28e37-d453-4fb6-878e-68b551f73dde\" (UID: \"c8d28e37-d453-4fb6-878e-68b551f73dde\") " Nov 02 17:55:21.921941 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.921903 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-workload-socket" pod "c8d28e37-d453-4fb6-878e-68b551f73dde" (UID: "c8d28e37-d453-4fb6-878e-68b551f73dde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:21.922580 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.922138 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-data" pod "c8d28e37-d453-4fb6-878e-68b551f73dde" (UID: "c8d28e37-d453-4fb6-878e-68b551f73dde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:21.922580 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.922159 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d28e37-d453-4fb6-878e-68b551f73dde-istiod-ca-cert" pod "c8d28e37-d453-4fb6-878e-68b551f73dde" (UID: "c8d28e37-d453-4fb6-878e-68b551f73dde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:21.922580 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.922367 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-workload-certs" pod "c8d28e37-d453-4fb6-878e-68b551f73dde" (UID: "c8d28e37-d453-4fb6-878e-68b551f73dde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:21.922580 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.922400 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-credential-socket" pod "c8d28e37-d453-4fb6-878e-68b551f73dde" (UID: "c8d28e37-d453-4fb6-878e-68b551f73dde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:21.922580 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.922504 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:21.922580 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.922527 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:21.922580 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.922544 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:21.922580 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.922558 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d28e37-d453-4fb6-878e-68b551f73dde-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:21.924005 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.923979 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-envoy" pod "c8d28e37-d453-4fb6-878e-68b551f73dde" (UID: "c8d28e37-d453-4fb6-878e-68b551f73dde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:21.924156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.924022 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d28e37-d453-4fb6-878e-68b551f73dde-kube-api-access-mltdc" pod "c8d28e37-d453-4fb6-878e-68b551f73dde" (UID: "c8d28e37-d453-4fb6-878e-68b551f73dde"). InnerVolumeSpecName "kube-api-access-mltdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:21.924645 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.924619 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-token" pod "c8d28e37-d453-4fb6-878e-68b551f73dde" (UID: "c8d28e37-d453-4fb6-878e-68b551f73dde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:21.924829 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:21.924815 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-podinfo" pod "c8d28e37-d453-4fb6-878e-68b551f73dde" (UID: "c8d28e37-d453-4fb6-878e-68b551f73dde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:55:22.023890 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.023840 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mltdc\" (UniqueName: \"kubernetes.io/projected/c8d28e37-d453-4fb6-878e-68b551f73dde-kube-api-access-mltdc\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:22.023890 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.023884 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:22.023890 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.023899 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:22.024180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.023912 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d28e37-d453-4fb6-878e-68b551f73dde-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:22.024180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.023926 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d28e37-d453-4fb6-878e-68b551f73dde-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:22.184705 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.184616 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:22.250245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.250212 5549 generic.go:356] "Generic (PLEG): container finished" podID="c8d28e37-d453-4fb6-878e-68b551f73dde" containerID="259b3b3edf67dbade9f08d1f48a8c2fd5cb4938b902ef7ec74b8679eb236a859" exitCode=0 Nov 02 17:55:22.250418 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.250265 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" Nov 02 17:55:22.250418 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.250291 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" event={"ID":"c8d28e37-d453-4fb6-878e-68b551f73dde","Type":"ContainerDied","Data":"259b3b3edf67dbade9f08d1f48a8c2fd5cb4938b902ef7ec74b8679eb236a859"} Nov 02 17:55:22.250418 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.250349 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878" event={"ID":"c8d28e37-d453-4fb6-878e-68b551f73dde","Type":"ContainerDied","Data":"f04a60a933036a571a38800ac648993b9c25c216e8635a7514a8cbce4c42cbb8"} Nov 02 17:55:22.250418 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.250373 5549 scope.go:122] "RemoveContainer" containerID="259b3b3edf67dbade9f08d1f48a8c2fd5cb4938b902ef7ec74b8679eb236a859" Nov 02 17:55:22.261292 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.261273 5549 scope.go:122] "RemoveContainer" containerID="42805b3aacbf92b7dd51e641348c5c1609f271d8f994d885c42f1c10843542fb" Nov 02 17:55:22.270436 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.270418 5549 scope.go:122] "RemoveContainer" containerID="259b3b3edf67dbade9f08d1f48a8c2fd5cb4938b902ef7ec74b8679eb236a859" Nov 02 17:55:22.270721 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:22.270701 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"259b3b3edf67dbade9f08d1f48a8c2fd5cb4938b902ef7ec74b8679eb236a859\": container with ID starting with 259b3b3edf67dbade9f08d1f48a8c2fd5cb4938b902ef7ec74b8679eb236a859 not found: ID does not exist" containerID="259b3b3edf67dbade9f08d1f48a8c2fd5cb4938b902ef7ec74b8679eb236a859" Nov 02 17:55:22.270770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.270729 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259b3b3edf67dbade9f08d1f48a8c2fd5cb4938b902ef7ec74b8679eb236a859"} err="failed to get container status \"259b3b3edf67dbade9f08d1f48a8c2fd5cb4938b902ef7ec74b8679eb236a859\": rpc error: code = NotFound desc = could not find container \"259b3b3edf67dbade9f08d1f48a8c2fd5cb4938b902ef7ec74b8679eb236a859\": container with ID starting with 259b3b3edf67dbade9f08d1f48a8c2fd5cb4938b902ef7ec74b8679eb236a859 not found: ID does not exist" Nov 02 17:55:22.270770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.270748 5549 scope.go:122] "RemoveContainer" containerID="42805b3aacbf92b7dd51e641348c5c1609f271d8f994d885c42f1c10843542fb" Nov 02 17:55:22.271034 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:22.271016 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42805b3aacbf92b7dd51e641348c5c1609f271d8f994d885c42f1c10843542fb\": container with ID starting with 42805b3aacbf92b7dd51e641348c5c1609f271d8f994d885c42f1c10843542fb not found: ID does not exist" containerID="42805b3aacbf92b7dd51e641348c5c1609f271d8f994d885c42f1c10843542fb" Nov 02 17:55:22.271097 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.271041 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42805b3aacbf92b7dd51e641348c5c1609f271d8f994d885c42f1c10843542fb"} err="failed to get container status \"42805b3aacbf92b7dd51e641348c5c1609f271d8f994d885c42f1c10843542fb\": rpc error: code = NotFound desc = could not find container \"42805b3aacbf92b7dd51e641348c5c1609f271d8f994d885c42f1c10843542fb\": container with ID starting with 42805b3aacbf92b7dd51e641348c5c1609f271d8f994d885c42f1c10843542fb not found: ID does not exist" Nov 02 17:55:22.277377 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.277355 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878"] Nov 02 17:55:22.283119 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.283096 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-dfd356-openshift-default-5cd7cbfc49-kh878"] Nov 02 17:55:22.320605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.320577 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8d28e37-d453-4fb6-878e-68b551f73dde" path="/var/lib/kubelet/pods/c8d28e37-d453-4fb6-878e-68b551f73dde/volumes" Nov 02 17:55:22.949055 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:22.946592 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:23.299822 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:23.299736 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36810b4f_4409_4a63_96fa_8c4565058368.slice/crio-a64d28b229e42a10c36b46e090815cb9c885cf98871f04d352eb7fb8c4a0a9de\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf586b4c3_d237_4d83_a2bb_5fbc85c3bc6d.slice/crio-0d68b8545f81a0ca6c9d34f745890db10cbf65efe75b65751b6dcd96845276df\": RecentStats: unable to find data in memory cache]" Nov 02 17:55:23.744452 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:23.744399 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:23.794977 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:23.794052 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36810b4f_4409_4a63_96fa_8c4565058368.slice/crio-a64d28b229e42a10c36b46e090815cb9c885cf98871f04d352eb7fb8c4a0a9de\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf586b4c3_d237_4d83_a2bb_5fbc85c3bc6d.slice/crio-0d68b8545f81a0ca6c9d34f745890db10cbf65efe75b65751b6dcd96845276df\": RecentStats: unable to find data in memory cache]" Nov 02 17:55:24.686941 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:24.686903 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:55:24.830188 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:24.830144 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:25.831977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:25.831117 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk"] Nov 02 17:55:25.857495 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:25.857452 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j"] Nov 02 17:55:26.142038 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:26.117530 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:26.524036 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:26.506307 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:27.025093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:27.025042 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:27.030160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:27.030122 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 17:55:27.103978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:27.095053 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" podUID="3827bd23-547c-4ae6-83c5-90c71860f814" containerName="main" probeResult="failure" output="Get \"https://10.133.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:27.270109 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:27.270059 5549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:27.270797 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:27.270390 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:27.466988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:27.464159 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 17:55:28.080979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:28.057236 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 17:55:28.107982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:28.082342 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:55:28.168984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:28.110008 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 17:55:28.168984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:28.110008 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" podUID="3827bd23-547c-4ae6-83c5-90c71860f814" containerName="main" probeResult="failure" output="Get \"https://10.133.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:28.402044 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:28.392432 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:28.496021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:28.495897 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:28.532742 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:28.532699 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:55:28.535945 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:28.535918 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:55:28.559289 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:28.559259 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:55:28.574242 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:28.573775 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 17:55:28.574708 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:28.574691 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 17:55:28.578680 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:28.578658 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 17:55:28.578936 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:28.578921 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:55:28.670167 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:28.670062 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:29.841912 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:29.841863 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:30.832210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:30.832150 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk" podUID="156ce927-ebce-4824-ae14-36712fe15948" containerName="payload-processing" containerID="cri-o://546daec82ef5637cc85a8ccf1c8e51ba95b815a2c771288b54e2e9e53974ef40" gracePeriod=25 Nov 02 17:55:30.858646 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:30.858608 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j" podUID="1bb46ea2-bbe1-43c6-88b8-346276e9b7d0" containerName="payload-pre-processing" containerID="cri-o://e32d78fb9f22e248563baccca3ee8a9884219a39643f6a95eb0dc5c345a80f14" gracePeriod=25 Nov 02 17:55:31.065551 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.065488 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:31.178833 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.178799 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk" Nov 02 17:55:31.202383 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.202346 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156ce927-ebce-4824-ae14-36712fe15948-kube-api-access-w7wbk\" (UniqueName: \"kubernetes.io/projected/156ce927-ebce-4824-ae14-36712fe15948-kube-api-access-w7wbk\") pod \"156ce927-ebce-4824-ae14-36712fe15948\" (UID: \"156ce927-ebce-4824-ae14-36712fe15948\") " Nov 02 17:55:31.202919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.202889 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/156ce927-ebce-4824-ae14-36712fe15948-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156ce927-ebce-4824-ae14-36712fe15948-plugins-config-volume\") pod \"156ce927-ebce-4824-ae14-36712fe15948\" (UID: \"156ce927-ebce-4824-ae14-36712fe15948\") " Nov 02 17:55:31.203460 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.203326 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156ce927-ebce-4824-ae14-36712fe15948-plugins-config-volume" pod "156ce927-ebce-4824-ae14-36712fe15948" (UID: "156ce927-ebce-4824-ae14-36712fe15948"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:31.203607 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.203589 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156ce927-ebce-4824-ae14-36712fe15948-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:31.204792 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.204769 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156ce927-ebce-4824-ae14-36712fe15948-kube-api-access-w7wbk" pod "156ce927-ebce-4824-ae14-36712fe15948" (UID: "156ce927-ebce-4824-ae14-36712fe15948"). InnerVolumeSpecName "kube-api-access-w7wbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:31.228188 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.228158 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j" Nov 02 17:55:31.304900 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.304862 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bb46ea2-bbe1-43c6-88b8-346276e9b7d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb46ea2-bbe1-43c6-88b8-346276e9b7d0-plugins-config-volume\") pod \"1bb46ea2-bbe1-43c6-88b8-346276e9b7d0\" (UID: \"1bb46ea2-bbe1-43c6-88b8-346276e9b7d0\") " Nov 02 17:55:31.305127 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.305113 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb46ea2-bbe1-43c6-88b8-346276e9b7d0-kube-api-access-zvxpw\" (UniqueName: \"kubernetes.io/projected/1bb46ea2-bbe1-43c6-88b8-346276e9b7d0-kube-api-access-zvxpw\") pod \"1bb46ea2-bbe1-43c6-88b8-346276e9b7d0\" (UID: \"1bb46ea2-bbe1-43c6-88b8-346276e9b7d0\") " Nov 02 17:55:31.305343 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.305315 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb46ea2-bbe1-43c6-88b8-346276e9b7d0-plugins-config-volume" pod "1bb46ea2-bbe1-43c6-88b8-346276e9b7d0" (UID: "1bb46ea2-bbe1-43c6-88b8-346276e9b7d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:31.305681 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.305649 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7wbk\" (UniqueName: \"kubernetes.io/projected/156ce927-ebce-4824-ae14-36712fe15948-kube-api-access-w7wbk\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:31.305681 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.305674 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb46ea2-bbe1-43c6-88b8-346276e9b7d0-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:31.307198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.307175 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb46ea2-bbe1-43c6-88b8-346276e9b7d0-kube-api-access-zvxpw" pod "1bb46ea2-bbe1-43c6-88b8-346276e9b7d0" (UID: "1bb46ea2-bbe1-43c6-88b8-346276e9b7d0"). InnerVolumeSpecName "kube-api-access-zvxpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:31.407597 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.407517 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvxpw\" (UniqueName: \"kubernetes.io/projected/1bb46ea2-bbe1-43c6-88b8-346276e9b7d0-kube-api-access-zvxpw\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:31.591376 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.591340 5549 generic.go:356] "Generic (PLEG): container finished" podID="1bb46ea2-bbe1-43c6-88b8-346276e9b7d0" containerID="e32d78fb9f22e248563baccca3ee8a9884219a39643f6a95eb0dc5c345a80f14" exitCode=0 Nov 02 17:55:31.591580 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.591396 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j" Nov 02 17:55:31.591580 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.591424 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j" event={"ID":"1bb46ea2-bbe1-43c6-88b8-346276e9b7d0","Type":"ContainerDied","Data":"e32d78fb9f22e248563baccca3ee8a9884219a39643f6a95eb0dc5c345a80f14"} Nov 02 17:55:31.591580 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.591489 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j" event={"ID":"1bb46ea2-bbe1-43c6-88b8-346276e9b7d0","Type":"ContainerDied","Data":"b94aeb10901b6cda1fab462e2af41c4c3fe25d1e389f10d5b9c8433b60e1ee3a"} Nov 02 17:55:31.591580 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.591514 5549 scope.go:122] "RemoveContainer" containerID="e32d78fb9f22e248563baccca3ee8a9884219a39643f6a95eb0dc5c345a80f14" Nov 02 17:55:31.593035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.593012 5549 generic.go:356] "Generic (PLEG): container finished" podID="156ce927-ebce-4824-ae14-36712fe15948" containerID="546daec82ef5637cc85a8ccf1c8e51ba95b815a2c771288b54e2e9e53974ef40" exitCode=0 Nov 02 17:55:31.593130 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.593065 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk" Nov 02 17:55:31.593181 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.593067 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk" event={"ID":"156ce927-ebce-4824-ae14-36712fe15948","Type":"ContainerDied","Data":"546daec82ef5637cc85a8ccf1c8e51ba95b815a2c771288b54e2e9e53974ef40"} Nov 02 17:55:31.593229 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.593210 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk" event={"ID":"156ce927-ebce-4824-ae14-36712fe15948","Type":"ContainerDied","Data":"9c6b810a8aacaae149f88b8809d88b94468501b967d82af5a3ed4dfeef72f809"} Nov 02 17:55:31.602067 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.602046 5549 scope.go:122] "RemoveContainer" containerID="e32d78fb9f22e248563baccca3ee8a9884219a39643f6a95eb0dc5c345a80f14" Nov 02 17:55:31.602524 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:31.602387 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e32d78fb9f22e248563baccca3ee8a9884219a39643f6a95eb0dc5c345a80f14\": container with ID starting with e32d78fb9f22e248563baccca3ee8a9884219a39643f6a95eb0dc5c345a80f14 not found: ID does not exist" containerID="e32d78fb9f22e248563baccca3ee8a9884219a39643f6a95eb0dc5c345a80f14" Nov 02 17:55:31.602524 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.602417 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e32d78fb9f22e248563baccca3ee8a9884219a39643f6a95eb0dc5c345a80f14"} err="failed to get container status \"e32d78fb9f22e248563baccca3ee8a9884219a39643f6a95eb0dc5c345a80f14\": rpc error: code = NotFound desc = could not find container \"e32d78fb9f22e248563baccca3ee8a9884219a39643f6a95eb0dc5c345a80f14\": container with ID starting with e32d78fb9f22e248563baccca3ee8a9884219a39643f6a95eb0dc5c345a80f14 not found: ID does not exist" Nov 02 17:55:31.602524 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.602431 5549 scope.go:122] "RemoveContainer" containerID="546daec82ef5637cc85a8ccf1c8e51ba95b815a2c771288b54e2e9e53974ef40" Nov 02 17:55:31.611635 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.611606 5549 scope.go:122] "RemoveContainer" containerID="546daec82ef5637cc85a8ccf1c8e51ba95b815a2c771288b54e2e9e53974ef40" Nov 02 17:55:31.611902 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:31.611882 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546daec82ef5637cc85a8ccf1c8e51ba95b815a2c771288b54e2e9e53974ef40\": container with ID starting with 546daec82ef5637cc85a8ccf1c8e51ba95b815a2c771288b54e2e9e53974ef40 not found: ID does not exist" containerID="546daec82ef5637cc85a8ccf1c8e51ba95b815a2c771288b54e2e9e53974ef40" Nov 02 17:55:31.611944 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.611907 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546daec82ef5637cc85a8ccf1c8e51ba95b815a2c771288b54e2e9e53974ef40"} err="failed to get container status \"546daec82ef5637cc85a8ccf1c8e51ba95b815a2c771288b54e2e9e53974ef40\": rpc error: code = NotFound desc = could not find container \"546daec82ef5637cc85a8ccf1c8e51ba95b815a2c771288b54e2e9e53974ef40\": container with ID starting with 546daec82ef5637cc85a8ccf1c8e51ba95b815a2c771288b54e2e9e53974ef40 not found: ID does not exist" Nov 02 17:55:31.626605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.626577 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j"] Nov 02 17:55:31.629287 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.629266 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-56e6db-7c5f5cdb79-dzq7j"] Nov 02 17:55:31.651806 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.651779 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk"] Nov 02 17:55:31.663902 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:31.663834 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-56e6db-5b5876fcc-6pjkk"] Nov 02 17:55:32.147413 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:32.147382 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:32.322878 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:32.322827 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="156ce927-ebce-4824-ae14-36712fe15948" path="/var/lib/kubelet/pods/156ce927-ebce-4824-ae14-36712fe15948/volumes" Nov 02 17:55:32.323597 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:32.323569 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bb46ea2-bbe1-43c6-88b8-346276e9b7d0" path="/var/lib/kubelet/pods/1bb46ea2-bbe1-43c6-88b8-346276e9b7d0/volumes" Nov 02 17:55:32.474653 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:32.474601 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:32.821028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:32.820027 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:33.123052 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:33.118288 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:33.314599 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:33.314557 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36810b4f_4409_4a63_96fa_8c4565058368.slice/crio-a64d28b229e42a10c36b46e090815cb9c885cf98871f04d352eb7fb8c4a0a9de\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf586b4c3_d237_4d83_a2bb_5fbc85c3bc6d.slice/crio-0d68b8545f81a0ca6c9d34f745890db10cbf65efe75b65751b6dcd96845276df\": RecentStats: unable to find data in memory cache]" Nov 02 17:55:33.345068 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:33.345036 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:34.166643 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.166611 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:34.218806 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.215401 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" podUID="3827bd23-547c-4ae6-83c5-90c71860f814" containerName="main" containerID="cri-o://7740a846d6bd96f7ed598d166db9313b29b8e68997874ed5b7e690059de16267" gracePeriod=45 Nov 02 17:55:34.514426 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.514400 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:55:34.617533 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.617499 5549 generic.go:356] "Generic (PLEG): container finished" podID="3827bd23-547c-4ae6-83c5-90c71860f814" containerID="7740a846d6bd96f7ed598d166db9313b29b8e68997874ed5b7e690059de16267" exitCode=0 Nov 02 17:55:34.617730 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.617540 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" Nov 02 17:55:34.617730 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.617582 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" event={"ID":"3827bd23-547c-4ae6-83c5-90c71860f814","Type":"ContainerDied","Data":"7740a846d6bd96f7ed598d166db9313b29b8e68997874ed5b7e690059de16267"} Nov 02 17:55:34.617730 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.617628 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh" event={"ID":"3827bd23-547c-4ae6-83c5-90c71860f814","Type":"ContainerDied","Data":"15325dbb4691fe82f7f937660926eb388b2e4c7581b7aad44a76ba75d3ebc5f7"} Nov 02 17:55:34.617730 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.617650 5549 scope.go:122] "RemoveContainer" containerID="7740a846d6bd96f7ed598d166db9313b29b8e68997874ed5b7e690059de16267" Nov 02 17:55:34.627446 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.627425 5549 scope.go:122] "RemoveContainer" containerID="7740a846d6bd96f7ed598d166db9313b29b8e68997874ed5b7e690059de16267" Nov 02 17:55:34.627761 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:34.627734 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7740a846d6bd96f7ed598d166db9313b29b8e68997874ed5b7e690059de16267\": container with ID starting with 7740a846d6bd96f7ed598d166db9313b29b8e68997874ed5b7e690059de16267 not found: ID does not exist" containerID="7740a846d6bd96f7ed598d166db9313b29b8e68997874ed5b7e690059de16267" Nov 02 17:55:34.627819 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.627764 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7740a846d6bd96f7ed598d166db9313b29b8e68997874ed5b7e690059de16267"} err="failed to get container status \"7740a846d6bd96f7ed598d166db9313b29b8e68997874ed5b7e690059de16267\": rpc error: code = NotFound desc = could not find container \"7740a846d6bd96f7ed598d166db9313b29b8e68997874ed5b7e690059de16267\": container with ID starting with 7740a846d6bd96f7ed598d166db9313b29b8e68997874ed5b7e690059de16267 not found: ID does not exist" Nov 02 17:55:34.683648 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.683604 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-tmp-dir\") pod \"3827bd23-547c-4ae6-83c5-90c71860f814\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " Nov 02 17:55:34.683825 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.683775 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3827bd23-547c-4ae6-83c5-90c71860f814-tls-certs\" (UniqueName: \"kubernetes.io/secret/3827bd23-547c-4ae6-83c5-90c71860f814-tls-certs\") pod \"3827bd23-547c-4ae6-83c5-90c71860f814\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " Nov 02 17:55:34.683825 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.683807 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-home\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-home\") pod \"3827bd23-547c-4ae6-83c5-90c71860f814\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " Nov 02 17:55:34.683919 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.683881 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-tmp-dir" pod "3827bd23-547c-4ae6-83c5-90c71860f814" (UID: "3827bd23-547c-4ae6-83c5-90c71860f814"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:34.683986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.683939 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-dshm\") pod \"3827bd23-547c-4ae6-83c5-90c71860f814\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " Nov 02 17:55:34.684053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.684036 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3827bd23-547c-4ae6-83c5-90c71860f814-kube-api-access-shqsf\" (UniqueName: \"kubernetes.io/projected/3827bd23-547c-4ae6-83c5-90c71860f814-kube-api-access-shqsf\") pod \"3827bd23-547c-4ae6-83c5-90c71860f814\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " Nov 02 17:55:34.684104 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.684083 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-home" pod "3827bd23-547c-4ae6-83c5-90c71860f814" (UID: "3827bd23-547c-4ae6-83c5-90c71860f814"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:34.684169 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.684150 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-model-cache\") pod \"3827bd23-547c-4ae6-83c5-90c71860f814\" (UID: \"3827bd23-547c-4ae6-83c5-90c71860f814\") " Nov 02 17:55:34.684861 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.684834 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:34.684861 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.684863 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:34.685185 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.685159 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-model-cache" pod "3827bd23-547c-4ae6-83c5-90c71860f814" (UID: "3827bd23-547c-4ae6-83c5-90c71860f814"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:34.686500 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.686475 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3827bd23-547c-4ae6-83c5-90c71860f814-tls-certs" pod "3827bd23-547c-4ae6-83c5-90c71860f814" (UID: "3827bd23-547c-4ae6-83c5-90c71860f814"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:34.686582 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.686547 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3827bd23-547c-4ae6-83c5-90c71860f814-kube-api-access-shqsf" pod "3827bd23-547c-4ae6-83c5-90c71860f814" (UID: "3827bd23-547c-4ae6-83c5-90c71860f814"). InnerVolumeSpecName "kube-api-access-shqsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:34.686747 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.686725 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-dshm" pod "3827bd23-547c-4ae6-83c5-90c71860f814" (UID: "3827bd23-547c-4ae6-83c5-90c71860f814"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:34.688897 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.688872 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp"] Nov 02 17:55:34.689143 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.689119 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" podUID="b078f1c1-1391-4686-96b3-35b15b027285" containerName="maas-api" containerID="cri-o://5a93165894addde4ef36c1460f6819d092dc47cfb17889ceefc7ec77688ccbd1" gracePeriod=30 Nov 02 17:55:34.786041 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.785919 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3827bd23-547c-4ae6-83c5-90c71860f814-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:34.786041 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.785973 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:34.786041 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.785989 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shqsf\" (UniqueName: \"kubernetes.io/projected/3827bd23-547c-4ae6-83c5-90c71860f814-kube-api-access-shqsf\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:34.786041 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.786004 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3827bd23-547c-4ae6-83c5-90c71860f814-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:34.895001 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.894931 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:34.942763 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:34.942737 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:55:35.044035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.041013 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh"] Nov 02 17:55:35.080728 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.076068 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-d8e217/praxis-route-d8e217-kserve-754885c495-rgfjh"] Nov 02 17:55:35.124668 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.118505 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b078f1c1-1391-4686-96b3-35b15b027285-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b078f1c1-1391-4686-96b3-35b15b027285-maas-api-tls\") pod \"b078f1c1-1391-4686-96b3-35b15b027285\" (UID: \"b078f1c1-1391-4686-96b3-35b15b027285\") " Nov 02 17:55:35.124668 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.118620 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b078f1c1-1391-4686-96b3-35b15b027285-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b078f1c1-1391-4686-96b3-35b15b027285-openshift-service-ca\") pod \"b078f1c1-1391-4686-96b3-35b15b027285\" (UID: \"b078f1c1-1391-4686-96b3-35b15b027285\") " Nov 02 17:55:35.124668 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.118753 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b078f1c1-1391-4686-96b3-35b15b027285-kube-api-access-tbbzx\" (UniqueName: \"kubernetes.io/projected/b078f1c1-1391-4686-96b3-35b15b027285-kube-api-access-tbbzx\") pod \"b078f1c1-1391-4686-96b3-35b15b027285\" (UID: \"b078f1c1-1391-4686-96b3-35b15b027285\") " Nov 02 17:55:35.124668 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.118794 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b078f1c1-1391-4686-96b3-35b15b027285-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b078f1c1-1391-4686-96b3-35b15b027285-metrics-certs\") pod \"b078f1c1-1391-4686-96b3-35b15b027285\" (UID: \"b078f1c1-1391-4686-96b3-35b15b027285\") " Nov 02 17:55:35.125797 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.125757 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b078f1c1-1391-4686-96b3-35b15b027285-metrics-certs" pod "b078f1c1-1391-4686-96b3-35b15b027285" (UID: "b078f1c1-1391-4686-96b3-35b15b027285"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:35.129025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.128993 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b078f1c1-1391-4686-96b3-35b15b027285-maas-api-tls" pod "b078f1c1-1391-4686-96b3-35b15b027285" (UID: "b078f1c1-1391-4686-96b3-35b15b027285"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:35.129485 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.129464 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b078f1c1-1391-4686-96b3-35b15b027285-openshift-service-ca" pod "b078f1c1-1391-4686-96b3-35b15b027285" (UID: "b078f1c1-1391-4686-96b3-35b15b027285"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:35.132465 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.132229 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b078f1c1-1391-4686-96b3-35b15b027285-kube-api-access-tbbzx" pod "b078f1c1-1391-4686-96b3-35b15b027285" (UID: "b078f1c1-1391-4686-96b3-35b15b027285"). InnerVolumeSpecName "kube-api-access-tbbzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:35.220798 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.220760 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b078f1c1-1391-4686-96b3-35b15b027285-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:35.220798 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.220793 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b078f1c1-1391-4686-96b3-35b15b027285-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:35.220798 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.220804 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b078f1c1-1391-4686-96b3-35b15b027285-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:35.221085 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.220813 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbbzx\" (UniqueName: \"kubernetes.io/projected/b078f1c1-1391-4686-96b3-35b15b027285-kube-api-access-tbbzx\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:35.411321 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.411287 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:35.626173 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.626053 5549 generic.go:356] "Generic (PLEG): container finished" podID="b078f1c1-1391-4686-96b3-35b15b027285" containerID="5a93165894addde4ef36c1460f6819d092dc47cfb17889ceefc7ec77688ccbd1" exitCode=0 Nov 02 17:55:35.626173 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.626094 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" Nov 02 17:55:35.626173 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.626129 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" event={"ID":"b078f1c1-1391-4686-96b3-35b15b027285","Type":"ContainerDied","Data":"5a93165894addde4ef36c1460f6819d092dc47cfb17889ceefc7ec77688ccbd1"} Nov 02 17:55:35.626777 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.626186 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp" event={"ID":"b078f1c1-1391-4686-96b3-35b15b027285","Type":"ContainerDied","Data":"552e5209548161a94f0cec2da90b5eb6f77a9a1bd141601e8351bbf3282fb97d"} Nov 02 17:55:35.626777 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.626209 5549 scope.go:122] "RemoveContainer" containerID="5a93165894addde4ef36c1460f6819d092dc47cfb17889ceefc7ec77688ccbd1" Nov 02 17:55:35.641352 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.641333 5549 scope.go:122] "RemoveContainer" containerID="5a93165894addde4ef36c1460f6819d092dc47cfb17889ceefc7ec77688ccbd1" Nov 02 17:55:35.641651 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:35.641632 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a93165894addde4ef36c1460f6819d092dc47cfb17889ceefc7ec77688ccbd1\": container with ID starting with 5a93165894addde4ef36c1460f6819d092dc47cfb17889ceefc7ec77688ccbd1 not found: ID does not exist" containerID="5a93165894addde4ef36c1460f6819d092dc47cfb17889ceefc7ec77688ccbd1" Nov 02 17:55:35.641699 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.641660 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a93165894addde4ef36c1460f6819d092dc47cfb17889ceefc7ec77688ccbd1"} err="failed to get container status \"5a93165894addde4ef36c1460f6819d092dc47cfb17889ceefc7ec77688ccbd1\": rpc error: code = NotFound desc = could not find container \"5a93165894addde4ef36c1460f6819d092dc47cfb17889ceefc7ec77688ccbd1\": container with ID starting with 5a93165894addde4ef36c1460f6819d092dc47cfb17889ceefc7ec77688ccbd1 not found: ID does not exist" Nov 02 17:55:35.656621 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.656560 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp"] Nov 02 17:55:35.659816 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:35.659787 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-56e6db-cb4b7fd6f-k9wgp"] Nov 02 17:55:36.231756 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:36.231725 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:36.320219 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:36.320184 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3827bd23-547c-4ae6-83c5-90c71860f814" path="/var/lib/kubelet/pods/3827bd23-547c-4ae6-83c5-90c71860f814/volumes" Nov 02 17:55:36.320716 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:36.320701 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b078f1c1-1391-4686-96b3-35b15b027285" path="/var/lib/kubelet/pods/b078f1c1-1391-4686-96b3-35b15b027285/volumes" Nov 02 17:55:36.562456 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:36.562372 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:36.809605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:36.809544 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:37.192768 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.192731 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:37.480822 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.480791 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:55:37.499113 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.499079 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:37.936331 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.936199 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:37.984612 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.984575 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt"] Nov 02 17:55:37.985227 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985212 5549 cpu_manager.go:414] "containerMap: removing container" podUID="f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d" containerName="payload-pre-processing" Nov 02 17:55:37.985306 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985227 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d" containerName="payload-pre-processing" Nov 02 17:55:37.985306 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985250 5549 cpu_manager.go:414] "containerMap: removing container" podUID="3827bd23-547c-4ae6-83c5-90c71860f814" containerName="main" Nov 02 17:55:37.985306 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985256 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3827bd23-547c-4ae6-83c5-90c71860f814" containerName="main" Nov 02 17:55:37.985306 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985271 5549 cpu_manager.go:414] "containerMap: removing container" podUID="b078f1c1-1391-4686-96b3-35b15b027285" containerName="maas-api" Nov 02 17:55:37.985306 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985276 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b078f1c1-1391-4686-96b3-35b15b027285" containerName="maas-api" Nov 02 17:55:37.985306 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985285 5549 cpu_manager.go:414] "containerMap: removing container" podUID="c8d28e37-d453-4fb6-878e-68b551f73dde" containerName="istio-proxy" Nov 02 17:55:37.985306 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985290 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d28e37-d453-4fb6-878e-68b551f73dde" containerName="istio-proxy" Nov 02 17:55:37.985306 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985302 5549 cpu_manager.go:414] "containerMap: removing container" podUID="36810b4f-4409-4a63-96fa-8c4565058368" containerName="maas-api" Nov 02 17:55:37.985306 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985307 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36810b4f-4409-4a63-96fa-8c4565058368" containerName="maas-api" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985315 5549 cpu_manager.go:414] "containerMap: removing container" podUID="1bb46ea2-bbe1-43c6-88b8-346276e9b7d0" containerName="payload-pre-processing" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985320 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb46ea2-bbe1-43c6-88b8-346276e9b7d0" containerName="payload-pre-processing" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985330 5549 cpu_manager.go:414] "containerMap: removing container" podUID="d26e1b21-d8a8-4bd6-bc52-0052582bfb9d" containerName="maas-api" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985336 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26e1b21-d8a8-4bd6-bc52-0052582bfb9d" containerName="maas-api" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985346 5549 cpu_manager.go:414] "containerMap: removing container" podUID="156ce927-ebce-4824-ae14-36712fe15948" containerName="payload-processing" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985351 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156ce927-ebce-4824-ae14-36712fe15948" containerName="payload-processing" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985357 5549 cpu_manager.go:414] "containerMap: removing container" podUID="34b73845-9bed-45b5-b698-12759713f944" containerName="payload-processing" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985361 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b73845-9bed-45b5-b698-12759713f944" containerName="payload-processing" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985372 5549 cpu_manager.go:414] "containerMap: removing container" podUID="c8d28e37-d453-4fb6-878e-68b551f73dde" containerName="istio-proxy" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985377 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d28e37-d453-4fb6-878e-68b551f73dde" containerName="istio-proxy" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985455 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="156ce927-ebce-4824-ae14-36712fe15948" containerName="payload-processing" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985465 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b73845-9bed-45b5-b698-12759713f944" containerName="payload-processing" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985473 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d28e37-d453-4fb6-878e-68b551f73dde" containerName="istio-proxy" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985479 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="f586b4c3-d237-4d83-a2bb-5fbc85c3bc6d" containerName="payload-pre-processing" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985487 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3827bd23-547c-4ae6-83c5-90c71860f814" containerName="main" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985494 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="b078f1c1-1391-4686-96b3-35b15b027285" containerName="maas-api" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985500 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="36810b4f-4409-4a63-96fa-8c4565058368" containerName="maas-api" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985505 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb46ea2-bbe1-43c6-88b8-346276e9b7d0" containerName="payload-pre-processing" Nov 02 17:55:37.985613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.985512 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26e1b21-d8a8-4bd6-bc52-0052582bfb9d" containerName="maas-api" Nov 02 17:55:37.988724 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.988708 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv"] Nov 02 17:55:37.988855 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.988834 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" Nov 02 17:55:37.989647 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.989624 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d28e37-d453-4fb6-878e-68b551f73dde" containerName="istio-proxy" Nov 02 17:55:37.992865 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.992844 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" Nov 02 17:55:37.999160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.999138 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-47e1c2-dockercfg-hp2sr\"" Nov 02 17:55:37.999777 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:37.999749 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-47e1c2\"" Nov 02 17:55:38.011069 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.011042 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv"] Nov 02 17:55:38.014181 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.014159 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt"] Nov 02 17:55:38.076257 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.076223 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt\" (UID: \"fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" Nov 02 17:55:38.076451 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.076277 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3785d4-1670-4f59-8f60-20e66fc3210b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv\" (UID: \"ed3785d4-1670-4f59-8f60-20e66fc3210b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" Nov 02 17:55:38.076451 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.076437 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbs4t\" (UniqueName: \"kubernetes.io/projected/fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9-kube-api-access-fbs4t\") pod \"payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt\" (UID: \"fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" Nov 02 17:55:38.076577 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.076472 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6t5p\" (UniqueName: \"kubernetes.io/projected/ed3785d4-1670-4f59-8f60-20e66fc3210b-kube-api-access-b6t5p\") pod \"payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv\" (UID: \"ed3785d4-1670-4f59-8f60-20e66fc3210b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" Nov 02 17:55:38.193028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.181084 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbs4t\" (UniqueName: \"kubernetes.io/projected/fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9-kube-api-access-fbs4t\") pod \"payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt\" (UID: \"fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" Nov 02 17:55:38.193028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.181168 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6t5p\" (UniqueName: \"kubernetes.io/projected/ed3785d4-1670-4f59-8f60-20e66fc3210b-kube-api-access-b6t5p\") pod \"payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv\" (UID: \"ed3785d4-1670-4f59-8f60-20e66fc3210b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" Nov 02 17:55:38.193028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.181345 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt\" (UID: \"fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" Nov 02 17:55:38.193028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.181399 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3785d4-1670-4f59-8f60-20e66fc3210b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv\" (UID: \"ed3785d4-1670-4f59-8f60-20e66fc3210b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" Nov 02 17:55:38.193028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.182354 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3785d4-1670-4f59-8f60-20e66fc3210b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv\" (UID: \"ed3785d4-1670-4f59-8f60-20e66fc3210b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" Nov 02 17:55:38.193028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.183626 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt\" (UID: \"fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" Nov 02 17:55:38.216008 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.211455 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:38.240759 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.240721 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbs4t\" (UniqueName: \"kubernetes.io/projected/fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9-kube-api-access-fbs4t\") pod \"payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt\" (UID: \"fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" Nov 02 17:55:38.265700 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.251835 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6t5p\" (UniqueName: \"kubernetes.io/projected/ed3785d4-1670-4f59-8f60-20e66fc3210b-kube-api-access-b6t5p\") pod \"payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv\" (UID: \"ed3785d4-1670-4f59-8f60-20e66fc3210b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" Nov 02 17:55:38.305931 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.305655 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" Nov 02 17:55:38.317289 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.317251 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" Nov 02 17:55:38.417193 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.414647 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t"] Nov 02 17:55:38.475010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.473181 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t"] Nov 02 17:55:38.475010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.473314 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.509993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.506109 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.509993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.506224 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.509993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.506287 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbtst\" (UniqueName: \"kubernetes.io/projected/a543a25d-8467-4bac-98c4-e0cf38dd2c76-kube-api-access-lbtst\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.509993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.506343 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.509993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.506413 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.509993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.506500 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.509993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.506524 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.509993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.506556 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.509993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.506614 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.601328 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.599277 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:38.608869 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.607893 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.608869 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.608003 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbtst\" (UniqueName: \"kubernetes.io/projected/a543a25d-8467-4bac-98c4-e0cf38dd2c76-kube-api-access-lbtst\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.608869 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.608068 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.608869 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.608136 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.608869 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.608191 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.608869 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.608216 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.608869 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.608246 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.608869 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.608300 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.608869 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.608359 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.649294 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.638281 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.649294 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.638562 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.649294 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.638786 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.653129 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.652543 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.654292 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.653734 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.707984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.689891 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.756658 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.714160 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.756658 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.756533 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.782135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.782098 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbtst\" (UniqueName: \"kubernetes.io/projected/a543a25d-8467-4bac-98c4-e0cf38dd2c76-kube-api-access-lbtst\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:38.832615 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:38.832555 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:39.010448 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:39.010342 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36810b4f_4409_4a63_96fa_8c4565058368.slice/crio-a64d28b229e42a10c36b46e090815cb9c885cf98871f04d352eb7fb8c4a0a9de\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf586b4c3_d237_4d83_a2bb_5fbc85c3bc6d.slice/crio-0d68b8545f81a0ca6c9d34f745890db10cbf65efe75b65751b6dcd96845276df\": RecentStats: unable to find data in memory cache]" Nov 02 17:55:39.091587 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:39.074632 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv"] Nov 02 17:55:39.109045 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:39.109010 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt"] Nov 02 17:55:39.364060 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:39.363991 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t"] Nov 02 17:55:39.372459 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:39.371893 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:55:39.372459 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:39.372009 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:55:39.372459 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:39.372057 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:55:39.694978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:39.691053 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:39.889040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:39.878353 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" event={"ID":"a543a25d-8467-4bac-98c4-e0cf38dd2c76","Type":"ContainerStarted","Data":"d045a6238809325661f83110e0d0c36c21a94caa5e351759728f082824c2837d"} Nov 02 17:55:39.889040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:39.878425 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" event={"ID":"a543a25d-8467-4bac-98c4-e0cf38dd2c76","Type":"ContainerStarted","Data":"bd27c916de1fd3b0aaab405ba5b41cd6a85820a409c473c1516b3fd4b1580142"} Nov 02 17:55:39.942217 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:39.942062 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" event={"ID":"fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9","Type":"ContainerStarted","Data":"f4d8caeaeb2667a596582fa2681860eeb9e7eee4a01ec78f420454507d432df8"} Nov 02 17:55:39.942217 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:39.942141 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" event={"ID":"fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9","Type":"ContainerStarted","Data":"f55b41a5fca7939b593718bd26a7c00809a252f43c359538bbfa55b70c5f7784"} Nov 02 17:55:39.942217 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:39.942174 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" Nov 02 17:55:39.944563 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:39.944507 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" podStartSLOduration=1.9444889220000001 podStartE2EDuration="1.944488922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:39.930363198 +0000 UTC m=+1311.718868751" watchObservedRunningTime="2026-11-02 17:55:39.944488922 +0000 UTC m=+1311.732994450" Nov 02 17:55:39.982909 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:39.982857 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" event={"ID":"ed3785d4-1670-4f59-8f60-20e66fc3210b","Type":"ContainerStarted","Data":"435a9a363139b9ee473b5c44650a05c964a40c050f7063185e5f8cdcf884b374"} Nov 02 17:55:39.984124 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:39.984097 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" event={"ID":"ed3785d4-1670-4f59-8f60-20e66fc3210b","Type":"ContainerStarted","Data":"c63c5a2643d7222a16f07f18e9fb24cf14c8ca56a0ca942ac1e078d05bade0fd"} Nov 02 17:55:39.984316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:39.984301 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" Nov 02 17:55:40.033978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:40.033025 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" podStartSLOduration=3.033005053 podStartE2EDuration="3.033005053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:39.979976958 +0000 UTC m=+1311.768482486" watchObservedRunningTime="2026-11-02 17:55:40.033005053 +0000 UTC m=+1311.821510585" Nov 02 17:55:40.649988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:40.646904 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:40.843142 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:40.843105 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:40.845136 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:40.845104 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 17:55:40.845270 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:40.845153 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" podUID="a543a25d-8467-4bac-98c4-e0cf38dd2c76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 17:55:41.552729 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:41.552679 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:41.842922 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:41.842832 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 17:55:41.842922 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:41.842888 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" podUID="a543a25d-8467-4bac-98c4-e0cf38dd2c76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 17:55:42.580979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:42.579015 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:42.842789 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:42.842706 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 17:55:42.842789 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:42.842762 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" podUID="a543a25d-8467-4bac-98c4-e0cf38dd2c76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 17:55:43.335740 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:43.335691 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf586b4c3_d237_4d83_a2bb_5fbc85c3bc6d.slice/crio-0d68b8545f81a0ca6c9d34f745890db10cbf65efe75b65751b6dcd96845276df\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36810b4f_4409_4a63_96fa_8c4565058368.slice/crio-a64d28b229e42a10c36b46e090815cb9c885cf98871f04d352eb7fb8c4a0a9de\": RecentStats: unable to find data in memory cache]" Nov 02 17:55:43.402983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:43.380145 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:43.843303 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:43.843270 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 17:55:43.843700 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:43.843326 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" podUID="a543a25d-8467-4bac-98c4-e0cf38dd2c76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 17:55:44.031724 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.031668 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" podStartSLOduration=7.031653894 podStartE2EDuration="7.031653894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:40.023572139 +0000 UTC m=+1311.812077668" watchObservedRunningTime="2026-11-02 17:55:44.031653894 +0000 UTC m=+1315.820159426" Nov 02 17:55:44.035170 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.035135 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8"] Nov 02 17:55:44.039648 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.039623 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 17:55:44.046632 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.046603 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-47e1c2-serving-cert\"" Nov 02 17:55:44.057186 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.057156 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8"] Nov 02 17:55:44.063672 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.063639 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8\" (UID: \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 17:55:44.063842 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.063727 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qdw7\" (UniqueName: \"kubernetes.io/projected/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-kube-api-access-9qdw7\") pod \"maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8\" (UID: \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 17:55:44.063931 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.063887 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8\" (UID: \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 17:55:44.064164 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.064137 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8\" (UID: \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 17:55:44.185671 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.185578 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:44.206272 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.186911 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8\" (UID: \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 17:55:44.214281 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.214243 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8\" (UID: \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 17:55:44.220981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.214610 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8\" (UID: \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 17:55:44.221382 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.221342 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qdw7\" (UniqueName: \"kubernetes.io/projected/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-kube-api-access-9qdw7\") pod \"maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8\" (UID: \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 17:55:44.224020 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:44.187099 5549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-serving-cert: secret "maas-api-e2e-shared-a-w4-47e1c2-serving-cert" not found Nov 02 17:55:44.237974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.234976 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8\" (UID: \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 17:55:44.259612 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:44.259583 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-maas-api-tls podName:7fe6f607-ad08-4d0c-b92f-1618ce0aa86b nodeName:}" failed. No retries permitted until 2026-11-02 17:55:44.759544773 +0000 UTC m=+1316.548050282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-maas-api-tls") pod "maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" (UID: "7fe6f607-ad08-4d0c-b92f-1618ce0aa86b") : secret "maas-api-e2e-shared-a-w4-47e1c2-serving-cert" not found Nov 02 17:55:44.279881 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.279843 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8\" (UID: \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 17:55:44.315728 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.315686 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qdw7\" (UniqueName: \"kubernetes.io/projected/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-kube-api-access-9qdw7\") pod \"maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8\" (UID: \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 17:55:44.400977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.400847 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:44.842977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.842338 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 17:55:44.842977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.842419 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" podUID="a543a25d-8467-4bac-98c4-e0cf38dd2c76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 17:55:44.861993 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.850391 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8\" (UID: \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 17:55:44.875975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.870087 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8\" (UID: \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 17:55:44.963140 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:44.963104 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 17:55:45.120618 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:45.120531 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:45.344389 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:45.344351 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8"] Nov 02 17:55:45.349772 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:55:45.349735 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe6f607_ad08_4d0c_b92f_1618ce0aa86b.slice/crio-0f38aaa2542ca023ef5a204c01bfdef0ceb9f63a16bab128fe9428f983f7e281 WatchSource:0}: Error finding container 0f38aaa2542ca023ef5a204c01bfdef0ceb9f63a16bab128fe9428f983f7e281: Status 404 returned error can't find the container with id 0f38aaa2542ca023ef5a204c01bfdef0ceb9f63a16bab128fe9428f983f7e281 Nov 02 17:55:45.498976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:45.498511 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:45.715997 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:45.715946 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:45.854027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:45.845337 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 17:55:45.854027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:45.845400 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" podUID="a543a25d-8467-4bac-98c4-e0cf38dd2c76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 17:55:45.890975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:45.890683 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:46.071767 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:46.071725 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" event={"ID":"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b","Type":"ContainerStarted","Data":"0f38aaa2542ca023ef5a204c01bfdef0ceb9f63a16bab128fe9428f983f7e281"} Nov 02 17:55:46.260013 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:46.259940 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:46.478352 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:46.478318 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:46.815651 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:46.815551 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:46.842284 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:46.842248 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 17:55:46.842455 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:46.842299 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" podUID="a543a25d-8467-4bac-98c4-e0cf38dd2c76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 17:55:47.085244 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:47.085165 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" event={"ID":"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b","Type":"ContainerStarted","Data":"379cb1f28659ec324cbb076644db1bf413b472d72cc6fe075e8acb5508353286"} Nov 02 17:55:47.086113 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:47.086063 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 17:55:47.135152 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:47.135082 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" podStartSLOduration=2.491799486 podStartE2EDuration="3.135062629s" totalImagesPullingTime="643.263143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:47.122308935 +0000 UTC m=+1318.910814462" watchObservedRunningTime="2026-11-02 17:55:47.135062629 +0000 UTC m=+1318.923568157" Nov 02 17:55:47.176460 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:47.176405 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:47.515974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:47.515217 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:47.658562 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:47.658515 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:55:48.026216 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:48.026175 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:48.026728 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:48.026701 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:48.027894 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:48.027873 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 17:55:48.477062 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:48.477024 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:49.505769 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:49.505680 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:50.077113 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:50.077078 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:50.276215 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:50.276161 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:50.695443 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:50.695388 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:51.001710 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.001623 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" Nov 02 17:55:51.353240 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.353164 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:51.382823 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.382785 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf"] Nov 02 17:55:51.389698 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.389660 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf" Nov 02 17:55:51.403496 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.403459 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-86f5f0\"" Nov 02 17:55:51.405401 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.405359 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0ffd57-6326-4c50-aea7-36e52d0e40d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf\" (UID: \"6d0ffd57-6326-4c50-aea7-36e52d0e40d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf" Nov 02 17:55:51.405549 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.405458 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktkr7\" (UniqueName: \"kubernetes.io/projected/6d0ffd57-6326-4c50-aea7-36e52d0e40d0-kube-api-access-ktkr7\") pod \"payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf\" (UID: \"6d0ffd57-6326-4c50-aea7-36e52d0e40d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf" Nov 02 17:55:51.420255 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.420217 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf"] Nov 02 17:55:51.437349 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.437307 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd"] Nov 02 17:55:51.443200 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.443171 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd" Nov 02 17:55:51.454003 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.453004 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-86f5f0-dockercfg-7trqd\"" Nov 02 17:55:51.454197 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.454168 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd"] Nov 02 17:55:51.541030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.536383 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0ffd57-6326-4c50-aea7-36e52d0e40d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf\" (UID: \"6d0ffd57-6326-4c50-aea7-36e52d0e40d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf" Nov 02 17:55:51.541030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.536460 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9grlw\" (UniqueName: \"kubernetes.io/projected/5897373f-0b89-4603-9b04-a99f0f23582b-kube-api-access-9grlw\") pod \"payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd\" (UID: \"5897373f-0b89-4603-9b04-a99f0f23582b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd" Nov 02 17:55:51.541030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.536495 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5897373f-0b89-4603-9b04-a99f0f23582b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd\" (UID: \"5897373f-0b89-4603-9b04-a99f0f23582b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd" Nov 02 17:55:51.541030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.536527 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktkr7\" (UniqueName: \"kubernetes.io/projected/6d0ffd57-6326-4c50-aea7-36e52d0e40d0-kube-api-access-ktkr7\") pod \"payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf\" (UID: \"6d0ffd57-6326-4c50-aea7-36e52d0e40d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf" Nov 02 17:55:51.541030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.538140 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0ffd57-6326-4c50-aea7-36e52d0e40d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf\" (UID: \"6d0ffd57-6326-4c50-aea7-36e52d0e40d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf" Nov 02 17:55:51.680240 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.679218 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:51.711973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.709673 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktkr7\" (UniqueName: \"kubernetes.io/projected/6d0ffd57-6326-4c50-aea7-36e52d0e40d0-kube-api-access-ktkr7\") pod \"payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf\" (UID: \"6d0ffd57-6326-4c50-aea7-36e52d0e40d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf" Nov 02 17:55:51.728716 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.714556 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf" Nov 02 17:55:51.728716 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.715419 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9grlw\" (UniqueName: \"kubernetes.io/projected/5897373f-0b89-4603-9b04-a99f0f23582b-kube-api-access-9grlw\") pod \"payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd\" (UID: \"5897373f-0b89-4603-9b04-a99f0f23582b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd" Nov 02 17:55:51.728716 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.715468 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5897373f-0b89-4603-9b04-a99f0f23582b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd\" (UID: \"5897373f-0b89-4603-9b04-a99f0f23582b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd" Nov 02 17:55:51.728716 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.722123 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5897373f-0b89-4603-9b04-a99f0f23582b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd\" (UID: \"5897373f-0b89-4603-9b04-a99f0f23582b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd" Nov 02 17:55:51.748605 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.748576 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9grlw\" (UniqueName: \"kubernetes.io/projected/5897373f-0b89-4603-9b04-a99f0f23582b-kube-api-access-9grlw\") pod \"payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd\" (UID: \"5897373f-0b89-4603-9b04-a99f0f23582b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd" Nov 02 17:55:51.765796 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.765761 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd" Nov 02 17:55:51.970730 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:51.970679 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:52.013650 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:52.013613 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" Nov 02 17:55:52.035317 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:52.035287 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf"] Nov 02 17:55:52.036321 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:55:52.036301 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0ffd57_6326_4c50_aea7_36e52d0e40d0.slice/crio-a9b022d50d33545e72ba5472512a4bdbd5e0d7f51fffd772f07ef89540b2a7b9 WatchSource:0}: Error finding container a9b022d50d33545e72ba5472512a4bdbd5e0d7f51fffd772f07ef89540b2a7b9: Status 404 returned error can't find the container with id a9b022d50d33545e72ba5472512a4bdbd5e0d7f51fffd772f07ef89540b2a7b9 Nov 02 17:55:52.076734 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:52.076633 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd"] Nov 02 17:55:52.080542 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:55:52.080513 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5897373f_0b89_4603_9b04_a99f0f23582b.slice/crio-32437ebd1b82c290a133e34817883c50d1bc8a0181f3d203ca8345b5d0f740d2 WatchSource:0}: Error finding container 32437ebd1b82c290a133e34817883c50d1bc8a0181f3d203ca8345b5d0f740d2: Status 404 returned error can't find the container with id 32437ebd1b82c290a133e34817883c50d1bc8a0181f3d203ca8345b5d0f740d2 Nov 02 17:55:52.123086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:52.123055 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd" event={"ID":"5897373f-0b89-4603-9b04-a99f0f23582b","Type":"ContainerStarted","Data":"32437ebd1b82c290a133e34817883c50d1bc8a0181f3d203ca8345b5d0f740d2"} Nov 02 17:55:52.127334 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:52.127308 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf" event={"ID":"6d0ffd57-6326-4c50-aea7-36e52d0e40d0","Type":"ContainerStarted","Data":"a9b022d50d33545e72ba5472512a4bdbd5e0d7f51fffd772f07ef89540b2a7b9"} Nov 02 17:55:52.750584 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:52.750546 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:53.136158 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:53.136067 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd" event={"ID":"5897373f-0b89-4603-9b04-a99f0f23582b","Type":"ContainerStarted","Data":"2b703500ea3d4d25171ca465015fa8dfd6980276469f2b39047fccb83843785f"} Nov 02 17:55:53.136362 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:53.136165 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd" Nov 02 17:55:53.137688 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:53.137665 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf" event={"ID":"6d0ffd57-6326-4c50-aea7-36e52d0e40d0","Type":"ContainerStarted","Data":"40fd4a906d1bc9b435dfb6ea17240115f5834ff2eb4d9ad09ac52533d077f430"} Nov 02 17:55:53.137788 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:53.137781 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf" Nov 02 17:55:53.162731 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:53.162684 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd" podStartSLOduration=2.162671256 podStartE2EDuration="2.162671256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:53.161032255 +0000 UTC m=+1324.949537782" watchObservedRunningTime="2026-11-02 17:55:53.162671256 +0000 UTC m=+1324.951176784" Nov 02 17:55:53.187906 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:53.187855 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf" podStartSLOduration=2.187843117 podStartE2EDuration="2.187843117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:53.185174486 +0000 UTC m=+1324.973680015" watchObservedRunningTime="2026-11-02 17:55:53.187843117 +0000 UTC m=+1324.976348644" Nov 02 17:55:53.200019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:53.199992 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:53.544479 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:53.544440 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:53.796036 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:53.795284 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:53.980977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:53.971923 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:54.106408 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:54.106324 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 17:55:54.486979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:54.483571 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:55.239978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:55.239321 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:56.335030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:56.334980 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:56.715903 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:56.715866 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:57.029027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:57.028899 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:57.262245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:57.262202 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:57.535241 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:57.535202 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:57.799761 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:57.799677 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:55:58.377202 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.377162 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:58.427103 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.427070 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk"] Nov 02 17:55:58.431818 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.431796 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 17:55:58.454525 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.454488 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk"] Nov 02 17:55:58.458366 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.458338 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7"] Nov 02 17:55:58.463436 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.463409 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7" Nov 02 17:55:58.474840 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.474812 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-86f5f0-serving-cert\"" Nov 02 17:55:58.475092 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.474836 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-b6b6f4-dockercfg-p76pr\"" Nov 02 17:55:58.475227 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.474905 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-b6b6f4\"" Nov 02 17:55:58.480402 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.480375 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7"] Nov 02 17:55:58.517044 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.517003 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt5t2\" (UniqueName: \"kubernetes.io/projected/8f88db9e-ffae-47f6-ad0e-4d2c834af420-kube-api-access-pt5t2\") pod \"payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7\" (UID: \"8f88db9e-ffae-47f6-ad0e-4d2c834af420\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7" Nov 02 17:55:58.517197 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.517062 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fb1f539-186f-4f47-9ca1-abf514ef710e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk\" (UID: \"6fb1f539-186f-4f47-9ca1-abf514ef710e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 17:55:58.517197 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.517142 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fb1f539-186f-4f47-9ca1-abf514ef710e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk\" (UID: \"6fb1f539-186f-4f47-9ca1-abf514ef710e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 17:55:58.517307 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.517251 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fb1f539-186f-4f47-9ca1-abf514ef710e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk\" (UID: \"6fb1f539-186f-4f47-9ca1-abf514ef710e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 17:55:58.517416 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.517401 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2pz4\" (UniqueName: \"kubernetes.io/projected/6fb1f539-186f-4f47-9ca1-abf514ef710e-kube-api-access-k2pz4\") pod \"maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk\" (UID: \"6fb1f539-186f-4f47-9ca1-abf514ef710e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 17:55:58.517501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.517486 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f88db9e-ffae-47f6-ad0e-4d2c834af420-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7\" (UID: \"8f88db9e-ffae-47f6-ad0e-4d2c834af420\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7" Nov 02 17:55:58.535252 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.535200 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn"] Nov 02 17:55:58.540864 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.540835 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn"] Nov 02 17:55:58.541060 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.540924 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn" Nov 02 17:55:58.618775 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.618733 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fb1f539-186f-4f47-9ca1-abf514ef710e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk\" (UID: \"6fb1f539-186f-4f47-9ca1-abf514ef710e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 17:55:58.619842 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.619788 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn\" (UID: \"bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn" Nov 02 17:55:58.620683 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.619968 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2pz4\" (UniqueName: \"kubernetes.io/projected/6fb1f539-186f-4f47-9ca1-abf514ef710e-kube-api-access-k2pz4\") pod \"maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk\" (UID: \"6fb1f539-186f-4f47-9ca1-abf514ef710e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 17:55:58.620683 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.620047 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4slq8\" (UniqueName: \"kubernetes.io/projected/bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5-kube-api-access-4slq8\") pod \"payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn\" (UID: \"bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn" Nov 02 17:55:58.620683 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.620085 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f88db9e-ffae-47f6-ad0e-4d2c834af420-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7\" (UID: \"8f88db9e-ffae-47f6-ad0e-4d2c834af420\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7" Nov 02 17:55:58.620683 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.620129 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt5t2\" (UniqueName: \"kubernetes.io/projected/8f88db9e-ffae-47f6-ad0e-4d2c834af420-kube-api-access-pt5t2\") pod \"payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7\" (UID: \"8f88db9e-ffae-47f6-ad0e-4d2c834af420\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7" Nov 02 17:55:58.620683 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.620167 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fb1f539-186f-4f47-9ca1-abf514ef710e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk\" (UID: \"6fb1f539-186f-4f47-9ca1-abf514ef710e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 17:55:58.620683 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.620253 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fb1f539-186f-4f47-9ca1-abf514ef710e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk\" (UID: \"6fb1f539-186f-4f47-9ca1-abf514ef710e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 17:55:58.620683 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.619703 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fb1f539-186f-4f47-9ca1-abf514ef710e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk\" (UID: \"6fb1f539-186f-4f47-9ca1-abf514ef710e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 17:55:58.620683 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:58.620447 5549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-serving-cert: secret "maas-api-e2e-shared-a-w2-86f5f0-serving-cert" not found Nov 02 17:55:58.620683 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:55:58.620537 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb1f539-186f-4f47-9ca1-abf514ef710e-maas-api-tls podName:6fb1f539-186f-4f47-9ca1-abf514ef710e nodeName:}" failed. No retries permitted until 2026-11-02 17:55:59.120514553 +0000 UTC m=+1330.909020062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6fb1f539-186f-4f47-9ca1-abf514ef710e-maas-api-tls") pod "maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" (UID: "6fb1f539-186f-4f47-9ca1-abf514ef710e") : secret "maas-api-e2e-shared-a-w2-86f5f0-serving-cert" not found Nov 02 17:55:58.621632 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.621608 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f88db9e-ffae-47f6-ad0e-4d2c834af420-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7\" (UID: \"8f88db9e-ffae-47f6-ad0e-4d2c834af420\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7" Nov 02 17:55:58.632536 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.632452 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fb1f539-186f-4f47-9ca1-abf514ef710e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk\" (UID: \"6fb1f539-186f-4f47-9ca1-abf514ef710e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 17:55:58.690051 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.681526 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:58.694026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.691047 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2pz4\" (UniqueName: \"kubernetes.io/projected/6fb1f539-186f-4f47-9ca1-abf514ef710e-kube-api-access-k2pz4\") pod \"maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk\" (UID: \"6fb1f539-186f-4f47-9ca1-abf514ef710e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 17:55:58.708977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.694720 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt5t2\" (UniqueName: \"kubernetes.io/projected/8f88db9e-ffae-47f6-ad0e-4d2c834af420-kube-api-access-pt5t2\") pod \"payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7\" (UID: \"8f88db9e-ffae-47f6-ad0e-4d2c834af420\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7" Nov 02 17:55:58.752045 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.735790 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4slq8\" (UniqueName: \"kubernetes.io/projected/bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5-kube-api-access-4slq8\") pod \"payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn\" (UID: \"bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn" Nov 02 17:55:58.756298 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.754251 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn\" (UID: \"bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn" Nov 02 17:55:58.756555 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.753363 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn\" (UID: \"bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn" Nov 02 17:55:58.794866 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.773848 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4slq8\" (UniqueName: \"kubernetes.io/projected/bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5-kube-api-access-4slq8\") pod \"payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn\" (UID: \"bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn" Nov 02 17:55:58.794866 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.787217 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7" Nov 02 17:55:58.882001 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.861560 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn" Nov 02 17:55:58.926990 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:58.926212 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:59.166086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:59.152538 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:59.273987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:59.246023 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fb1f539-186f-4f47-9ca1-abf514ef710e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk\" (UID: \"6fb1f539-186f-4f47-9ca1-abf514ef710e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 17:55:59.291656 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:59.291615 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fb1f539-186f-4f47-9ca1-abf514ef710e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk\" (UID: \"6fb1f539-186f-4f47-9ca1-abf514ef710e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 17:55:59.349029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:59.348252 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 17:55:59.424233 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:55:59.424164 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf572a0_b48e_4b0b_adf5_e1e8b8a70bb5.slice/crio-597051a6ab881520e3aefe089930df41e3cf9dc2ee16c6c12dc0f8d7c095fd12 WatchSource:0}: Error finding container 597051a6ab881520e3aefe089930df41e3cf9dc2ee16c6c12dc0f8d7c095fd12: Status 404 returned error can't find the container with id 597051a6ab881520e3aefe089930df41e3cf9dc2ee16c6c12dc0f8d7c095fd12 Nov 02 17:55:59.430900 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:59.430392 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn"] Nov 02 17:55:59.583351 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:59.575227 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7"] Nov 02 17:55:59.848294 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:59.846579 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:59.934029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:55:59.932497 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk"] Nov 02 17:55:59.989004 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:55:59.983118 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb1f539_186f_4f47_9ca1_abf514ef710e.slice/crio-8f75a05ebc20a76d93dd46a7dba967d95c57085261b6f9df5c1dba07bc1c6e37 WatchSource:0}: Error finding container 8f75a05ebc20a76d93dd46a7dba967d95c57085261b6f9df5c1dba07bc1c6e37: Status 404 returned error can't find the container with id 8f75a05ebc20a76d93dd46a7dba967d95c57085261b6f9df5c1dba07bc1c6e37 Nov 02 17:56:00.406975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:00.406591 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:00.482619 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:00.482549 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7" podStartSLOduration=2.482528546 podStartE2EDuration="2.482528546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:00.456322113 +0000 UTC m=+1332.244827641" watchObservedRunningTime="2026-11-02 17:56:00.482528546 +0000 UTC m=+1332.271034076" Nov 02 17:56:00.507005 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:00.487049 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7" Nov 02 17:56:00.507005 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:00.487140 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7" event={"ID":"8f88db9e-ffae-47f6-ad0e-4d2c834af420","Type":"ContainerStarted","Data":"3368036274b55e6cc59be424b8924f5fa956befa214198d391e0fe41909971f2"} Nov 02 17:56:00.507005 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:00.487172 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7" event={"ID":"8f88db9e-ffae-47f6-ad0e-4d2c834af420","Type":"ContainerStarted","Data":"5f46d4991502e8034ba1ee59f3ace518eeb74a85ce8cbbbc98eba5c1cf6c8ed6"} Nov 02 17:56:00.720036 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:00.681505 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" event={"ID":"6fb1f539-186f-4f47-9ca1-abf514ef710e","Type":"ContainerStarted","Data":"8f75a05ebc20a76d93dd46a7dba967d95c57085261b6f9df5c1dba07bc1c6e37"} Nov 02 17:56:00.724303 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:00.724210 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:00.779986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:00.778669 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn" event={"ID":"bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5","Type":"ContainerStarted","Data":"a61c948d81a6741ec472c067da6a27d2abbb00a6b1eebeb08b5b2555a86eddce"} Nov 02 17:56:00.779986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:00.778743 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn" event={"ID":"bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5","Type":"ContainerStarted","Data":"597051a6ab881520e3aefe089930df41e3cf9dc2ee16c6c12dc0f8d7c095fd12"} Nov 02 17:56:00.779986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:00.779890 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn" Nov 02 17:56:00.845981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:00.831661 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c"] Nov 02 17:56:00.856264 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:00.854566 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn" podStartSLOduration=2.854545669 podStartE2EDuration="2.854545669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:00.852614989 +0000 UTC m=+1332.641120520" watchObservedRunningTime="2026-11-02 17:56:00.854545669 +0000 UTC m=+1332.643051200" Nov 02 17:56:00.995198 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:00.994690 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:01.362978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:01.362579 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:01.381668 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:01.381624 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:56:01.381984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:01.381932 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" podUID="75712abb-4a4d-4b51-9b9d-620589c1ffb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:56:01.559246 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:01.549935 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:01.787640 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:01.787594 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" event={"ID":"6fb1f539-186f-4f47-9ca1-abf514ef710e","Type":"ContainerStarted","Data":"e56edcd13dd770576c4aceea8fc7be6e025aeb20b9781c991deb07809997ebf5"} Nov 02 17:56:01.788108 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:01.787871 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 17:56:01.811948 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:01.811832 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" podStartSLOduration=2.632490824 podStartE2EDuration="3.811814333s" totalImagesPullingTime="1.179323509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:01.809300025 +0000 UTC m=+1333.597805555" watchObservedRunningTime="2026-11-02 17:56:01.811814333 +0000 UTC m=+1333.600319890" Nov 02 17:56:02.295977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:02.288107 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:02.550030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:02.543620 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:03.578983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:03.575760 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:04.160014 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:04.159985 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd" Nov 02 17:56:04.160210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:04.160058 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf" Nov 02 17:56:04.501352 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:04.501312 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:05.515117 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:05.515084 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:05.709991 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:05.709818 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:05.857102 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:05.857010 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" podUID="c3689510-357c-4d9c-93d5-8971c2bddea9" containerName="payload-processing" containerID="cri-o://e85125f3b9ae5e88653cb38356feece6727a994f42a9423e47a00b3bd7011151" gracePeriod=25 Nov 02 17:56:06.515237 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:06.515166 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:06.752242 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:06.752207 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" podUID="c3689510-357c-4d9c-93d5-8971c2bddea9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:56:06.890014 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:06.889902 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc"] Nov 02 17:56:06.892923 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:06.892879 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:06.894904 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:06.894870 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" Nov 02 17:56:06.905556 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:06.905537 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-1e2281\"" Nov 02 17:56:06.908922 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:06.908904 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-1e2281-dockercfg-htwrp\"" Nov 02 17:56:06.920935 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:06.920907 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc"] Nov 02 17:56:06.938342 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:06.938313 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg"] Nov 02 17:56:06.944003 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:06.943978 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" Nov 02 17:56:06.967486 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:06.967450 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg"] Nov 02 17:56:06.976620 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:06.976581 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea5ef1e-e3a6-47cf-a908-843442f86e83-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc\" (UID: \"3ea5ef1e-e3a6-47cf-a908-843442f86e83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" Nov 02 17:56:06.976780 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:06.976711 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dsfl\" (UniqueName: \"kubernetes.io/projected/3ea5ef1e-e3a6-47cf-a908-843442f86e83-kube-api-access-2dsfl\") pod \"payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc\" (UID: \"3ea5ef1e-e3a6-47cf-a908-843442f86e83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" Nov 02 17:56:07.087019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.086976 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea5ef1e-e3a6-47cf-a908-843442f86e83-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc\" (UID: \"3ea5ef1e-e3a6-47cf-a908-843442f86e83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" Nov 02 17:56:07.100983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.088203 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea5ef1e-e3a6-47cf-a908-843442f86e83-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc\" (UID: \"3ea5ef1e-e3a6-47cf-a908-843442f86e83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" Nov 02 17:56:07.100983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.093145 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dsfl\" (UniqueName: \"kubernetes.io/projected/3ea5ef1e-e3a6-47cf-a908-843442f86e83-kube-api-access-2dsfl\") pod \"payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc\" (UID: \"3ea5ef1e-e3a6-47cf-a908-843442f86e83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" Nov 02 17:56:07.100983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.093235 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29kw7\" (UniqueName: \"kubernetes.io/projected/bc67d335-3268-420d-ab7b-e63af8d1165f-kube-api-access-29kw7\") pod \"payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg\" (UID: \"bc67d335-3268-420d-ab7b-e63af8d1165f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" Nov 02 17:56:07.100983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.093342 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc67d335-3268-420d-ab7b-e63af8d1165f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg\" (UID: \"bc67d335-3268-420d-ab7b-e63af8d1165f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" Nov 02 17:56:07.123977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.121822 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dsfl\" (UniqueName: \"kubernetes.io/projected/3ea5ef1e-e3a6-47cf-a908-843442f86e83-kube-api-access-2dsfl\") pod \"payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc\" (UID: \"3ea5ef1e-e3a6-47cf-a908-843442f86e83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" Nov 02 17:56:07.139986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.137722 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:07.208034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.203497 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29kw7\" (UniqueName: \"kubernetes.io/projected/bc67d335-3268-420d-ab7b-e63af8d1165f-kube-api-access-29kw7\") pod \"payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg\" (UID: \"bc67d335-3268-420d-ab7b-e63af8d1165f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" Nov 02 17:56:07.208034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.203624 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc67d335-3268-420d-ab7b-e63af8d1165f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg\" (UID: \"bc67d335-3268-420d-ab7b-e63af8d1165f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" Nov 02 17:56:07.208034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.204655 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc67d335-3268-420d-ab7b-e63af8d1165f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg\" (UID: \"bc67d335-3268-420d-ab7b-e63af8d1165f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" Nov 02 17:56:07.216985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.216202 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" Nov 02 17:56:07.243633 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.243558 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29kw7\" (UniqueName: \"kubernetes.io/projected/bc67d335-3268-420d-ab7b-e63af8d1165f-kube-api-access-29kw7\") pod \"payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg\" (UID: \"bc67d335-3268-420d-ab7b-e63af8d1165f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" Nov 02 17:56:07.270800 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.270769 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" Nov 02 17:56:07.385399 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.385359 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:07.544850 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:56:07.544817 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea5ef1e_e3a6_47cf_a908_843442f86e83.slice/crio-639682cfb220ff06518e4db3c4f6813d81d9351d267f6ea7d9acc569c62fdf18 WatchSource:0}: Error finding container 639682cfb220ff06518e4db3c4f6813d81d9351d267f6ea7d9acc569c62fdf18: Status 404 returned error can't find the container with id 639682cfb220ff06518e4db3c4f6813d81d9351d267f6ea7d9acc569c62fdf18 Nov 02 17:56:07.546530 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.546401 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc"] Nov 02 17:56:07.577153 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.577124 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg"] Nov 02 17:56:07.580300 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:56:07.580272 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc67d335_3268_420d_ab7b_e63af8d1165f.slice/crio-8fcc164951fdb38da76729981610418b0bbc81960419616410ff454146ae571e WatchSource:0}: Error finding container 8fcc164951fdb38da76729981610418b0bbc81960419616410ff454146ae571e: Status 404 returned error can't find the container with id 8fcc164951fdb38da76729981610418b0bbc81960419616410ff454146ae571e Nov 02 17:56:07.800821 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.799460 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 17:56:07.836158 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.836123 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" event={"ID":"bc67d335-3268-420d-ab7b-e63af8d1165f","Type":"ContainerStarted","Data":"75c4d40d27053a96893d5f4307e89aa8399e470b3303c8ff5df71052c2bce0d0"} Nov 02 17:56:07.836363 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.836188 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" Nov 02 17:56:07.836363 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.836211 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" event={"ID":"bc67d335-3268-420d-ab7b-e63af8d1165f","Type":"ContainerStarted","Data":"8fcc164951fdb38da76729981610418b0bbc81960419616410ff454146ae571e"} Nov 02 17:56:07.837912 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.837879 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" event={"ID":"3ea5ef1e-e3a6-47cf-a908-843442f86e83","Type":"ContainerStarted","Data":"6587f53e125cdb5a9a5a2b0f01c716e03455e6bbf61d979306a352a616e064ab"} Nov 02 17:56:07.838076 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.837934 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" event={"ID":"3ea5ef1e-e3a6-47cf-a908-843442f86e83","Type":"ContainerStarted","Data":"639682cfb220ff06518e4db3c4f6813d81d9351d267f6ea7d9acc569c62fdf18"} Nov 02 17:56:07.838076 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.837978 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" Nov 02 17:56:07.898566 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.898503 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" podStartSLOduration=1.898483299 podStartE2EDuration="1.898483299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:07.879389228 +0000 UTC m=+1339.667894781" watchObservedRunningTime="2026-11-02 17:56:07.898483299 +0000 UTC m=+1339.686988829" Nov 02 17:56:07.908139 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.908100 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:07.930607 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:07.930544 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" podStartSLOduration=1.9305254889999999 podStartE2EDuration="1.930525489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:07.926641915 +0000 UTC m=+1339.715147466" watchObservedRunningTime="2026-11-02 17:56:07.930525489 +0000 UTC m=+1339.719031018" Nov 02 17:56:08.134026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:08.133549 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:09.065536 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:09.062925 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:09.906981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:09.892474 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:10.126979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:10.116390 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:11.179979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:11.174383 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:11.563530 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:11.563492 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:11.753894 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:11.753858 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:11.791350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:11.791320 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7" Nov 02 17:56:12.508977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:12.508650 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:12.797382 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:12.797294 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn" Nov 02 17:56:13.085874 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.085792 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2"] Nov 02 17:56:13.091350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.091332 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" Nov 02 17:56:13.117612 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.117580 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-9165c3\"" Nov 02 17:56:13.120892 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.120858 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2"] Nov 02 17:56:13.124028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.123999 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:13.158874 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.158825 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgm5g\" (UniqueName: \"kubernetes.io/projected/3190ca5d-41b1-474c-aaf6-8859ef3d2dfd-kube-api-access-dgm5g\") pod \"payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2\" (UID: \"3190ca5d-41b1-474c-aaf6-8859ef3d2dfd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" Nov 02 17:56:13.159350 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.159318 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3190ca5d-41b1-474c-aaf6-8859ef3d2dfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2\" (UID: \"3190ca5d-41b1-474c-aaf6-8859ef3d2dfd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" Nov 02 17:56:13.287975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.285662 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3190ca5d-41b1-474c-aaf6-8859ef3d2dfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2\" (UID: \"3190ca5d-41b1-474c-aaf6-8859ef3d2dfd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" Nov 02 17:56:13.287975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.285771 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgm5g\" (UniqueName: \"kubernetes.io/projected/3190ca5d-41b1-474c-aaf6-8859ef3d2dfd-kube-api-access-dgm5g\") pod \"payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2\" (UID: \"3190ca5d-41b1-474c-aaf6-8859ef3d2dfd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" Nov 02 17:56:13.287975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.287178 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3190ca5d-41b1-474c-aaf6-8859ef3d2dfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2\" (UID: \"3190ca5d-41b1-474c-aaf6-8859ef3d2dfd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" Nov 02 17:56:13.328977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.314710 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgm5g\" (UniqueName: \"kubernetes.io/projected/3190ca5d-41b1-474c-aaf6-8859ef3d2dfd-kube-api-access-dgm5g\") pod \"payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2\" (UID: \"3190ca5d-41b1-474c-aaf6-8859ef3d2dfd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" Nov 02 17:56:13.346626 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.346539 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:13.408979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.408561 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" Nov 02 17:56:13.787203 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.787177 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2"] Nov 02 17:56:13.788499 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:56:13.788467 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3190ca5d_41b1_474c_aaf6_8859ef3d2dfd.slice/crio-5ea23bd2eab0a08386d6af535739a88a287059141c9234c10bf4e35b8f1c3d0d WatchSource:0}: Error finding container 5ea23bd2eab0a08386d6af535739a88a287059141c9234c10bf4e35b8f1c3d0d: Status 404 returned error can't find the container with id 5ea23bd2eab0a08386d6af535739a88a287059141c9234c10bf4e35b8f1c3d0d Nov 02 17:56:13.957979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.956673 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:13.962304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.962265 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" event={"ID":"3190ca5d-41b1-474c-aaf6-8859ef3d2dfd","Type":"ContainerStarted","Data":"d932f1c08ebec84f639291c68c29d0c03c6bb76d69884e7cdc7f8de93e794bac"} Nov 02 17:56:13.962489 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.962334 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" event={"ID":"3190ca5d-41b1-474c-aaf6-8859ef3d2dfd","Type":"ContainerStarted","Data":"5ea23bd2eab0a08386d6af535739a88a287059141c9234c10bf4e35b8f1c3d0d"} Nov 02 17:56:13.962556 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.962512 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" Nov 02 17:56:13.987689 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:13.987640 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" podStartSLOduration=0.987625958 podStartE2EDuration="987.625958ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:13.985120649 +0000 UTC m=+1345.773626177" watchObservedRunningTime="2026-11-02 17:56:13.987625958 +0000 UTC m=+1345.776131526" Nov 02 17:56:14.779414 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:14.779368 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:15.267207 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.267166 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:15.518826 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.518735 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859"] Nov 02 17:56:15.523686 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.523661 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 17:56:15.534401 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.534370 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-1e2281-serving-cert\"" Nov 02 17:56:15.536315 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.536260 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859"] Nov 02 17:56:15.688016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.653946 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:15.688016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.667234 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/189c356c-d893-4df2-b800-97fd592fbe62-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859\" (UID: \"189c356c-d893-4df2-b800-97fd592fbe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 17:56:15.688016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.667450 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/189c356c-d893-4df2-b800-97fd592fbe62-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859\" (UID: \"189c356c-d893-4df2-b800-97fd592fbe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 17:56:15.688016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.667487 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/189c356c-d893-4df2-b800-97fd592fbe62-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859\" (UID: \"189c356c-d893-4df2-b800-97fd592fbe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 17:56:15.688016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.667612 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnq46\" (UniqueName: \"kubernetes.io/projected/189c356c-d893-4df2-b800-97fd592fbe62-kube-api-access-rnq46\") pod \"maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859\" (UID: \"189c356c-d893-4df2-b800-97fd592fbe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 17:56:15.769405 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.769309 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/189c356c-d893-4df2-b800-97fd592fbe62-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859\" (UID: \"189c356c-d893-4df2-b800-97fd592fbe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 17:56:15.769405 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.769366 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/189c356c-d893-4df2-b800-97fd592fbe62-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859\" (UID: \"189c356c-d893-4df2-b800-97fd592fbe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 17:56:15.769703 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.769507 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnq46\" (UniqueName: \"kubernetes.io/projected/189c356c-d893-4df2-b800-97fd592fbe62-kube-api-access-rnq46\") pod \"maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859\" (UID: \"189c356c-d893-4df2-b800-97fd592fbe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 17:56:15.769703 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.769623 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/189c356c-d893-4df2-b800-97fd592fbe62-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859\" (UID: \"189c356c-d893-4df2-b800-97fd592fbe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 17:56:15.769902 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:56:15.769881 5549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-serving-cert: secret "maas-api-e2e-shared-a-w0-1e2281-serving-cert" not found Nov 02 17:56:15.769991 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:56:15.769966 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/189c356c-d893-4df2-b800-97fd592fbe62-maas-api-tls podName:189c356c-d893-4df2-b800-97fd592fbe62 nodeName:}" failed. No retries permitted until 2026-11-02 17:56:16.269930028 +0000 UTC m=+1348.058435557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/189c356c-d893-4df2-b800-97fd592fbe62-maas-api-tls") pod "maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" (UID: "189c356c-d893-4df2-b800-97fd592fbe62") : secret "maas-api-e2e-shared-a-w0-1e2281-serving-cert" not found Nov 02 17:56:15.770214 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.770192 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/189c356c-d893-4df2-b800-97fd592fbe62-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859\" (UID: \"189c356c-d893-4df2-b800-97fd592fbe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 17:56:15.772464 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.772440 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/189c356c-d893-4df2-b800-97fd592fbe62-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859\" (UID: \"189c356c-d893-4df2-b800-97fd592fbe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 17:56:15.781737 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.781709 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnq46\" (UniqueName: \"kubernetes.io/projected/189c356c-d893-4df2-b800-97fd592fbe62-kube-api-access-rnq46\") pod \"maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859\" (UID: \"189c356c-d893-4df2-b800-97fd592fbe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 17:56:15.814820 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.814789 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:15.986312 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:15.986274 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:16.235884 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:16.235843 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:16.281174 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:16.281131 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/189c356c-d893-4df2-b800-97fd592fbe62-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859\" (UID: \"189c356c-d893-4df2-b800-97fd592fbe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 17:56:16.337075 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:16.337033 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/189c356c-d893-4df2-b800-97fd592fbe62-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859\" (UID: \"189c356c-d893-4df2-b800-97fd592fbe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 17:56:16.445331 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:16.445278 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 17:56:16.751687 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:16.751578 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" podUID="c3689510-357c-4d9c-93d5-8971c2bddea9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:56:16.877031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:16.876988 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859"] Nov 02 17:56:16.878346 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:56:16.878321 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189c356c_d893_4df2_b800_97fd592fbe62.slice/crio-4fd5c32ac8d9262f5e73f96d3328d8ae2b3353dead41be038acc356061011501 WatchSource:0}: Error finding container 4fd5c32ac8d9262f5e73f96d3328d8ae2b3353dead41be038acc356061011501: Status 404 returned error can't find the container with id 4fd5c32ac8d9262f5e73f96d3328d8ae2b3353dead41be038acc356061011501 Nov 02 17:56:16.912812 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:16.909826 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:17.030270 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:17.029282 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" event={"ID":"189c356c-d893-4df2-b800-97fd592fbe62","Type":"ContainerStarted","Data":"4fd5c32ac8d9262f5e73f96d3328d8ae2b3353dead41be038acc356061011501"} Nov 02 17:56:17.071625 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:17.071585 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:17.554205 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:17.554167 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:17.589211 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:56:17.589106 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:56:17.938756 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:17.938671 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:18.099975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:18.099375 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" event={"ID":"189c356c-d893-4df2-b800-97fd592fbe62","Type":"ContainerStarted","Data":"4e7787213d4e4c25e6d6f32aba8ea6f6e2c8e45a77007b8672ad9622c0dc1b82"} Nov 02 17:56:18.111290 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:18.100636 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 17:56:18.144975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:18.144645 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" podStartSLOduration=2.693867942 podStartE2EDuration="3.144622857s" totalImagesPullingTime="450.754915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:18.130430531 +0000 UTC m=+1349.918936212" watchObservedRunningTime="2026-11-02 17:56:18.144622857 +0000 UTC m=+1349.933128385" Nov 02 17:56:18.776855 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:18.776821 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:18.894976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:18.890192 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" Nov 02 17:56:18.894976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:18.891718 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" Nov 02 17:56:20.298603 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:20.298543 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:20.482920 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:20.482849 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:21.552024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:21.549714 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:22.100045 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.100003 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:22.132108 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.132071 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27"] Nov 02 17:56:22.136861 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.136842 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" Nov 02 17:56:22.148877 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.148849 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-b1a788\"" Nov 02 17:56:22.149042 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.148859 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-b1a788-dockercfg-c6bxr\"" Nov 02 17:56:22.163083 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.163053 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27"] Nov 02 17:56:22.202830 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.202797 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v"] Nov 02 17:56:22.203503 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.203479 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3d4c2f-d60c-41dd-af49-4923ee32729e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27\" (UID: \"5a3d4c2f-d60c-41dd-af49-4923ee32729e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" Nov 02 17:56:22.203709 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.203691 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkvth\" (UniqueName: \"kubernetes.io/projected/5a3d4c2f-d60c-41dd-af49-4923ee32729e-kube-api-access-pkvth\") pod \"payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27\" (UID: \"5a3d4c2f-d60c-41dd-af49-4923ee32729e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" Nov 02 17:56:22.207858 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.207838 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" Nov 02 17:56:22.230863 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.230832 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v"] Nov 02 17:56:22.289474 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.289426 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:22.305234 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.305199 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkvth\" (UniqueName: \"kubernetes.io/projected/5a3d4c2f-d60c-41dd-af49-4923ee32729e-kube-api-access-pkvth\") pod \"payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27\" (UID: \"5a3d4c2f-d60c-41dd-af49-4923ee32729e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" Nov 02 17:56:22.305400 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.305372 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3d4c2f-d60c-41dd-af49-4923ee32729e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27\" (UID: \"5a3d4c2f-d60c-41dd-af49-4923ee32729e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" Nov 02 17:56:22.305495 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.305479 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924d0e07-2010-4007-8148-5b7623447adb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v\" (UID: \"924d0e07-2010-4007-8148-5b7623447adb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" Nov 02 17:56:22.305617 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.305602 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrhvb\" (UniqueName: \"kubernetes.io/projected/924d0e07-2010-4007-8148-5b7623447adb-kube-api-access-lrhvb\") pod \"payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v\" (UID: \"924d0e07-2010-4007-8148-5b7623447adb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" Nov 02 17:56:22.306467 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.306439 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3d4c2f-d60c-41dd-af49-4923ee32729e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27\" (UID: \"5a3d4c2f-d60c-41dd-af49-4923ee32729e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" Nov 02 17:56:22.315768 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.315740 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkvth\" (UniqueName: \"kubernetes.io/projected/5a3d4c2f-d60c-41dd-af49-4923ee32729e-kube-api-access-pkvth\") pod \"payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27\" (UID: \"5a3d4c2f-d60c-41dd-af49-4923ee32729e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" Nov 02 17:56:22.406988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.406841 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924d0e07-2010-4007-8148-5b7623447adb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v\" (UID: \"924d0e07-2010-4007-8148-5b7623447adb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" Nov 02 17:56:22.407172 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.407046 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrhvb\" (UniqueName: \"kubernetes.io/projected/924d0e07-2010-4007-8148-5b7623447adb-kube-api-access-lrhvb\") pod \"payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v\" (UID: \"924d0e07-2010-4007-8148-5b7623447adb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" Nov 02 17:56:22.407616 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.407592 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924d0e07-2010-4007-8148-5b7623447adb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v\" (UID: \"924d0e07-2010-4007-8148-5b7623447adb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" Nov 02 17:56:22.416460 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.416425 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrhvb\" (UniqueName: \"kubernetes.io/projected/924d0e07-2010-4007-8148-5b7623447adb-kube-api-access-lrhvb\") pod \"payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v\" (UID: \"924d0e07-2010-4007-8148-5b7623447adb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" Nov 02 17:56:22.452251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.452223 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" Nov 02 17:56:22.487439 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.487393 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:22.586978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.534390 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" Nov 02 17:56:22.797063 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.797015 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:22.946576 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:22.946552 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27"] Nov 02 17:56:22.947287 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:56:22.947260 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3d4c2f_d60c_41dd_af49_4923ee32729e.slice/crio-838ff6a1d7f4e2a4f1919a49c7dbf3492fc87b86af6c4d94fe38190a82e8bdd7 WatchSource:0}: Error finding container 838ff6a1d7f4e2a4f1919a49c7dbf3492fc87b86af6c4d94fe38190a82e8bdd7: Status 404 returned error can't find the container with id 838ff6a1d7f4e2a4f1919a49c7dbf3492fc87b86af6c4d94fe38190a82e8bdd7 Nov 02 17:56:23.013978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:23.000532 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:23.084122 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:23.084088 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v"] Nov 02 17:56:23.084349 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:56:23.084329 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924d0e07_2010_4007_8148_5b7623447adb.slice/crio-fd9a647bb2d0981a4a8bf3328ff61ea632b77076efab1de4da7fef0d8e6f00b0 WatchSource:0}: Error finding container fd9a647bb2d0981a4a8bf3328ff61ea632b77076efab1de4da7fef0d8e6f00b0: Status 404 returned error can't find the container with id fd9a647bb2d0981a4a8bf3328ff61ea632b77076efab1de4da7fef0d8e6f00b0 Nov 02 17:56:23.289187 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:23.281082 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:23.416978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:23.413659 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" event={"ID":"924d0e07-2010-4007-8148-5b7623447adb","Type":"ContainerStarted","Data":"bc7bb080824974272a3098bff74707eeb6ae69fdcbe0b7386e7acbbb66edce87"} Nov 02 17:56:23.416978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:23.413752 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" event={"ID":"924d0e07-2010-4007-8148-5b7623447adb","Type":"ContainerStarted","Data":"fd9a647bb2d0981a4a8bf3328ff61ea632b77076efab1de4da7fef0d8e6f00b0"} Nov 02 17:56:23.416978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:23.415836 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" Nov 02 17:56:23.424596 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:23.423068 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" event={"ID":"5a3d4c2f-d60c-41dd-af49-4923ee32729e","Type":"ContainerStarted","Data":"b09c2bce08988ad036a89217b2bed2580aee6394c46b8ea48f9d40e3714be92f"} Nov 02 17:56:23.424596 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:23.424193 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" Nov 02 17:56:23.424596 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:23.424270 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" event={"ID":"5a3d4c2f-d60c-41dd-af49-4923ee32729e","Type":"ContainerStarted","Data":"838ff6a1d7f4e2a4f1919a49c7dbf3492fc87b86af6c4d94fe38190a82e8bdd7"} Nov 02 17:56:23.460669 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:23.443761 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" podStartSLOduration=1.443739008 podStartE2EDuration="1.443739008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:23.435544349 +0000 UTC m=+1355.224049879" watchObservedRunningTime="2026-11-02 17:56:23.443739008 +0000 UTC m=+1355.232244536" Nov 02 17:56:23.460669 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:23.459690 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:23.476982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:23.468137 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" podStartSLOduration=1.468116106 podStartE2EDuration="1.468116106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:23.462347579 +0000 UTC m=+1355.250853108" watchObservedRunningTime="2026-11-02 17:56:23.468116106 +0000 UTC m=+1355.256621635" Nov 02 17:56:24.372696 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:24.372651 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:24.530975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:24.528676 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:24.971671 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:24.971644 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" Nov 02 17:56:25.299062 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:25.298977 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 17:56:25.678028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:25.677705 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:26.191681 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.191655 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" Nov 02 17:56:26.256097 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.255939 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:26.340849 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.340809 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3689510-357c-4d9c-93d5-8971c2bddea9-kube-api-access-hbrwp\" (UniqueName: \"kubernetes.io/projected/c3689510-357c-4d9c-93d5-8971c2bddea9-kube-api-access-hbrwp\") pod \"c3689510-357c-4d9c-93d5-8971c2bddea9\" (UID: \"c3689510-357c-4d9c-93d5-8971c2bddea9\") " Nov 02 17:56:26.341017 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.340936 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3689510-357c-4d9c-93d5-8971c2bddea9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3689510-357c-4d9c-93d5-8971c2bddea9-plugins-config-volume\") pod \"c3689510-357c-4d9c-93d5-8971c2bddea9\" (UID: \"c3689510-357c-4d9c-93d5-8971c2bddea9\") " Nov 02 17:56:26.341827 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.341801 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3689510-357c-4d9c-93d5-8971c2bddea9-plugins-config-volume" pod "c3689510-357c-4d9c-93d5-8971c2bddea9" (UID: "c3689510-357c-4d9c-93d5-8971c2bddea9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:26.344490 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.344465 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3689510-357c-4d9c-93d5-8971c2bddea9-kube-api-access-hbrwp" pod "c3689510-357c-4d9c-93d5-8971c2bddea9" (UID: "c3689510-357c-4d9c-93d5-8971c2bddea9"). InnerVolumeSpecName "kube-api-access-hbrwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:26.440642 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.440597 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:26.442448 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.442416 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3689510-357c-4d9c-93d5-8971c2bddea9-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:26.442448 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.442447 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbrwp\" (UniqueName: \"kubernetes.io/projected/c3689510-357c-4d9c-93d5-8971c2bddea9-kube-api-access-hbrwp\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:26.445261 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.445235 5549 generic.go:356] "Generic (PLEG): container finished" podID="c3689510-357c-4d9c-93d5-8971c2bddea9" containerID="e85125f3b9ae5e88653cb38356feece6727a994f42a9423e47a00b3bd7011151" exitCode=0 Nov 02 17:56:26.445388 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.445273 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" event={"ID":"c3689510-357c-4d9c-93d5-8971c2bddea9","Type":"ContainerDied","Data":"e85125f3b9ae5e88653cb38356feece6727a994f42a9423e47a00b3bd7011151"} Nov 02 17:56:26.445388 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.445280 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" Nov 02 17:56:26.445388 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.445317 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c" event={"ID":"c3689510-357c-4d9c-93d5-8971c2bddea9","Type":"ContainerDied","Data":"7602139d500aa7946f082972dea79ddb59150faa8a872f2e5c91afa3047391b4"} Nov 02 17:56:26.445388 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.445340 5549 scope.go:122] "RemoveContainer" containerID="e85125f3b9ae5e88653cb38356feece6727a994f42a9423e47a00b3bd7011151" Nov 02 17:56:26.456490 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.456463 5549 scope.go:122] "RemoveContainer" containerID="e85125f3b9ae5e88653cb38356feece6727a994f42a9423e47a00b3bd7011151" Nov 02 17:56:26.456779 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:56:26.456757 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e85125f3b9ae5e88653cb38356feece6727a994f42a9423e47a00b3bd7011151\": container with ID starting with e85125f3b9ae5e88653cb38356feece6727a994f42a9423e47a00b3bd7011151 not found: ID does not exist" containerID="e85125f3b9ae5e88653cb38356feece6727a994f42a9423e47a00b3bd7011151" Nov 02 17:56:26.456842 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.456788 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e85125f3b9ae5e88653cb38356feece6727a994f42a9423e47a00b3bd7011151"} err="failed to get container status \"e85125f3b9ae5e88653cb38356feece6727a994f42a9423e47a00b3bd7011151\": rpc error: code = NotFound desc = could not find container \"e85125f3b9ae5e88653cb38356feece6727a994f42a9423e47a00b3bd7011151\": container with ID starting with e85125f3b9ae5e88653cb38356feece6727a994f42a9423e47a00b3bd7011151 not found: ID does not exist" Nov 02 17:56:26.484880 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.484845 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c"] Nov 02 17:56:26.490939 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.490908 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d8e217-7d5dc8cdd9-xb99c"] Nov 02 17:56:26.629407 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.629313 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:26.785351 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.785301 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:26.966982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:26.963823 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:27.290447 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:27.290328 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:27.455284 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:27.455231 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:27.944846 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:27.943836 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:28.116750 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.116710 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:28.320846 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.320810 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3689510-357c-4d9c-93d5-8971c2bddea9" path="/var/lib/kubelet/pods/c3689510-357c-4d9c-93d5-8971c2bddea9/volumes" Nov 02 17:56:28.691769 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.691688 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:28.709918 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.709880 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4"] Nov 02 17:56:28.710655 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.710636 5549 cpu_manager.go:414] "containerMap: removing container" podUID="c3689510-357c-4d9c-93d5-8971c2bddea9" containerName="payload-processing" Nov 02 17:56:28.710655 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.710654 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3689510-357c-4d9c-93d5-8971c2bddea9" containerName="payload-processing" Nov 02 17:56:28.710804 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.710795 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3689510-357c-4d9c-93d5-8971c2bddea9" containerName="payload-processing" Nov 02 17:56:28.715683 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.715663 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" Nov 02 17:56:28.720362 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.720333 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-2961d5\"" Nov 02 17:56:28.723539 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.723513 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4"] Nov 02 17:56:28.759393 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.759354 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-2961d5-dockercfg-vpz2g\"" Nov 02 17:56:28.764064 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.764031 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm"] Nov 02 17:56:28.769733 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.769708 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" Nov 02 17:56:28.778262 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.778234 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm"] Nov 02 17:56:28.831978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.828912 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-729sg\" (UniqueName: \"kubernetes.io/projected/fd061465-fa5b-4565-a280-f546c45b9f0e-kube-api-access-729sg\") pod \"payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm\" (UID: \"fd061465-fa5b-4565-a280-f546c45b9f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" Nov 02 17:56:28.831978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.829045 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw5n5\" (UniqueName: \"kubernetes.io/projected/e14cd218-72ea-4e71-bd6d-daad87430119-kube-api-access-jw5n5\") pod \"payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4\" (UID: \"e14cd218-72ea-4e71-bd6d-daad87430119\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" Nov 02 17:56:28.831978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.829268 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14cd218-72ea-4e71-bd6d-daad87430119-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4\" (UID: \"e14cd218-72ea-4e71-bd6d-daad87430119\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" Nov 02 17:56:28.831978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.829316 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd061465-fa5b-4565-a280-f546c45b9f0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm\" (UID: \"fd061465-fa5b-4565-a280-f546c45b9f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" Nov 02 17:56:28.928627 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.926676 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:28.932741 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.930435 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw5n5\" (UniqueName: \"kubernetes.io/projected/e14cd218-72ea-4e71-bd6d-daad87430119-kube-api-access-jw5n5\") pod \"payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4\" (UID: \"e14cd218-72ea-4e71-bd6d-daad87430119\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" Nov 02 17:56:28.932741 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.930608 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14cd218-72ea-4e71-bd6d-daad87430119-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4\" (UID: \"e14cd218-72ea-4e71-bd6d-daad87430119\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" Nov 02 17:56:28.932741 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.930652 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd061465-fa5b-4565-a280-f546c45b9f0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm\" (UID: \"fd061465-fa5b-4565-a280-f546c45b9f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" Nov 02 17:56:28.932741 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.930729 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-729sg\" (UniqueName: \"kubernetes.io/projected/fd061465-fa5b-4565-a280-f546c45b9f0e-kube-api-access-729sg\") pod \"payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm\" (UID: \"fd061465-fa5b-4565-a280-f546c45b9f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" Nov 02 17:56:28.971972 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.953522 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd061465-fa5b-4565-a280-f546c45b9f0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm\" (UID: \"fd061465-fa5b-4565-a280-f546c45b9f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" Nov 02 17:56:29.010973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:28.985249 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14cd218-72ea-4e71-bd6d-daad87430119-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4\" (UID: \"e14cd218-72ea-4e71-bd6d-daad87430119\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" Nov 02 17:56:29.093989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:29.083448 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:29.168662 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:29.167821 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-729sg\" (UniqueName: \"kubernetes.io/projected/fd061465-fa5b-4565-a280-f546c45b9f0e-kube-api-access-729sg\") pod \"payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm\" (UID: \"fd061465-fa5b-4565-a280-f546c45b9f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" Nov 02 17:56:29.189421 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:29.189382 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw5n5\" (UniqueName: \"kubernetes.io/projected/e14cd218-72ea-4e71-bd6d-daad87430119-kube-api-access-jw5n5\") pod \"payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4\" (UID: \"e14cd218-72ea-4e71-bd6d-daad87430119\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" Nov 02 17:56:29.332160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:29.332070 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" Nov 02 17:56:29.387051 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:29.386999 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" Nov 02 17:56:29.631087 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:29.631055 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4"] Nov 02 17:56:29.632702 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:56:29.632676 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14cd218_72ea_4e71_bd6d_daad87430119.slice/crio-5f92dd46d2cd1d0629f6213f12b39ad320602916f2c1a99fbec91ed6d8011f96 WatchSource:0}: Error finding container 5f92dd46d2cd1d0629f6213f12b39ad320602916f2c1a99fbec91ed6d8011f96: Status 404 returned error can't find the container with id 5f92dd46d2cd1d0629f6213f12b39ad320602916f2c1a99fbec91ed6d8011f96 Nov 02 17:56:29.718016 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:29.717992 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm"] Nov 02 17:56:29.720302 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:56:29.720278 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd061465_fa5b_4565_a280_f546c45b9f0e.slice/crio-2df4f47e31b13aefd9e946f285f38f8341a33e2d190e54be0be6d5e1ceba0061 WatchSource:0}: Error finding container 2df4f47e31b13aefd9e946f285f38f8341a33e2d190e54be0be6d5e1ceba0061: Status 404 returned error can't find the container with id 2df4f47e31b13aefd9e946f285f38f8341a33e2d190e54be0be6d5e1ceba0061 Nov 02 17:56:30.076973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:30.074528 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:30.518586 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:30.518550 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" event={"ID":"e14cd218-72ea-4e71-bd6d-daad87430119","Type":"ContainerStarted","Data":"cc6f8a40384ca3fd6280c093d95e645b13dd2f74a1315606da71eb3d81808398"} Nov 02 17:56:30.518771 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:30.518611 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" event={"ID":"e14cd218-72ea-4e71-bd6d-daad87430119","Type":"ContainerStarted","Data":"5f92dd46d2cd1d0629f6213f12b39ad320602916f2c1a99fbec91ed6d8011f96"} Nov 02 17:56:30.518771 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:30.518647 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" Nov 02 17:56:30.520415 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:30.520391 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" event={"ID":"fd061465-fa5b-4565-a280-f546c45b9f0e","Type":"ContainerStarted","Data":"01c128338df9ff573678190f5bda667e1e8df6353d8588f5f8d3a793e2c6016c"} Nov 02 17:56:30.520517 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:30.520428 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" event={"ID":"fd061465-fa5b-4565-a280-f546c45b9f0e","Type":"ContainerStarted","Data":"2df4f47e31b13aefd9e946f285f38f8341a33e2d190e54be0be6d5e1ceba0061"} Nov 02 17:56:30.520517 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:30.520456 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" Nov 02 17:56:30.545453 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:30.545391 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" podStartSLOduration=2.545377772 podStartE2EDuration="2.545377772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:30.54399909 +0000 UTC m=+1362.332504615" watchObservedRunningTime="2026-11-02 17:56:30.545377772 +0000 UTC m=+1362.333883294" Nov 02 17:56:30.567048 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:30.566992 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" podStartSLOduration=2.5669782740000002 podStartE2EDuration="2.566978274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:30.565943021 +0000 UTC m=+1362.354448548" watchObservedRunningTime="2026-11-02 17:56:30.566978274 +0000 UTC m=+1362.355483803" Nov 02 17:56:31.160978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:31.148321 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:31.928176 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:31.920709 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:32.766989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:32.763619 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:33.043030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:33.042928 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn"] Nov 02 17:56:33.063602 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:33.063568 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc"] Nov 02 17:56:33.174572 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:33.174534 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:33.335727 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:33.335596 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:33.937489 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:33.937429 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:34.120913 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:34.120876 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:34.433530 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:34.433502 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" Nov 02 17:56:34.433702 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:34.433575 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" Nov 02 17:56:35.046175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:35.046134 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:35.675320 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:35.675283 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:35.703869 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:35.703831 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf"] Nov 02 17:56:35.704187 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:35.704154 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" podUID="dd40957c-0e44-445b-8274-afa6ba043fc5" containerName="maas-api" containerID="cri-o://904d9019cfb836a3cc1505dd8769b8cea124b90207ab934f445206c8ffd35a11" gracePeriod=30 Nov 02 17:56:35.871214 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:35.871174 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:36.086581 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.085032 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:36.208976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.207177 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:56:36.252026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.250401 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:36.274355 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.273366 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd40957c-0e44-445b-8274-afa6ba043fc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd40957c-0e44-445b-8274-afa6ba043fc5-metrics-certs\") pod \"dd40957c-0e44-445b-8274-afa6ba043fc5\" (UID: \"dd40957c-0e44-445b-8274-afa6ba043fc5\") " Nov 02 17:56:36.274355 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.273502 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd40957c-0e44-445b-8274-afa6ba043fc5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd40957c-0e44-445b-8274-afa6ba043fc5-openshift-service-ca\") pod \"dd40957c-0e44-445b-8274-afa6ba043fc5\" (UID: \"dd40957c-0e44-445b-8274-afa6ba043fc5\") " Nov 02 17:56:36.274355 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.273598 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd40957c-0e44-445b-8274-afa6ba043fc5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd40957c-0e44-445b-8274-afa6ba043fc5-maas-api-tls\") pod \"dd40957c-0e44-445b-8274-afa6ba043fc5\" (UID: \"dd40957c-0e44-445b-8274-afa6ba043fc5\") " Nov 02 17:56:36.274355 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.273700 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd40957c-0e44-445b-8274-afa6ba043fc5-kube-api-access-2blgp\" (UniqueName: \"kubernetes.io/projected/dd40957c-0e44-445b-8274-afa6ba043fc5-kube-api-access-2blgp\") pod \"dd40957c-0e44-445b-8274-afa6ba043fc5\" (UID: \"dd40957c-0e44-445b-8274-afa6ba043fc5\") " Nov 02 17:56:36.278375 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.278342 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd40957c-0e44-445b-8274-afa6ba043fc5-metrics-certs" pod "dd40957c-0e44-445b-8274-afa6ba043fc5" (UID: "dd40957c-0e44-445b-8274-afa6ba043fc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:36.278697 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.278672 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd40957c-0e44-445b-8274-afa6ba043fc5-openshift-service-ca" pod "dd40957c-0e44-445b-8274-afa6ba043fc5" (UID: "dd40957c-0e44-445b-8274-afa6ba043fc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:36.281221 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.281195 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd40957c-0e44-445b-8274-afa6ba043fc5-kube-api-access-2blgp" pod "dd40957c-0e44-445b-8274-afa6ba043fc5" (UID: "dd40957c-0e44-445b-8274-afa6ba043fc5"). InnerVolumeSpecName "kube-api-access-2blgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:36.281343 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.281308 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd40957c-0e44-445b-8274-afa6ba043fc5-maas-api-tls" pod "dd40957c-0e44-445b-8274-afa6ba043fc5" (UID: "dd40957c-0e44-445b-8274-afa6ba043fc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:36.376030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.375916 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd40957c-0e44-445b-8274-afa6ba043fc5-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:36.376030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.375940 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd40957c-0e44-445b-8274-afa6ba043fc5-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:36.376030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.375985 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd40957c-0e44-445b-8274-afa6ba043fc5-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:36.376030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.375999 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2blgp\" (UniqueName: \"kubernetes.io/projected/dd40957c-0e44-445b-8274-afa6ba043fc5-kube-api-access-2blgp\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:36.417397 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.417362 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:36.584212 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.584177 5549 generic.go:356] "Generic (PLEG): container finished" podID="dd40957c-0e44-445b-8274-afa6ba043fc5" containerID="904d9019cfb836a3cc1505dd8769b8cea124b90207ab934f445206c8ffd35a11" exitCode=0 Nov 02 17:56:36.584415 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.584221 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" Nov 02 17:56:36.584415 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.584261 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" event={"ID":"dd40957c-0e44-445b-8274-afa6ba043fc5","Type":"ContainerDied","Data":"904d9019cfb836a3cc1505dd8769b8cea124b90207ab934f445206c8ffd35a11"} Nov 02 17:56:36.584415 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.584324 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf" event={"ID":"dd40957c-0e44-445b-8274-afa6ba043fc5","Type":"ContainerDied","Data":"b3e404084617bd4a74983c7780b244063c5ddc07257ef87cd58a0764adb15448"} Nov 02 17:56:36.584415 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.584346 5549 scope.go:122] "RemoveContainer" containerID="904d9019cfb836a3cc1505dd8769b8cea124b90207ab934f445206c8ffd35a11" Nov 02 17:56:36.594622 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.594595 5549 scope.go:122] "RemoveContainer" containerID="904d9019cfb836a3cc1505dd8769b8cea124b90207ab934f445206c8ffd35a11" Nov 02 17:56:36.594914 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:56:36.594887 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"904d9019cfb836a3cc1505dd8769b8cea124b90207ab934f445206c8ffd35a11\": container with ID starting with 904d9019cfb836a3cc1505dd8769b8cea124b90207ab934f445206c8ffd35a11 not found: ID does not exist" containerID="904d9019cfb836a3cc1505dd8769b8cea124b90207ab934f445206c8ffd35a11" Nov 02 17:56:36.594977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.594923 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"904d9019cfb836a3cc1505dd8769b8cea124b90207ab934f445206c8ffd35a11"} err="failed to get container status \"904d9019cfb836a3cc1505dd8769b8cea124b90207ab934f445206c8ffd35a11\": rpc error: code = NotFound desc = could not find container \"904d9019cfb836a3cc1505dd8769b8cea124b90207ab934f445206c8ffd35a11\": container with ID starting with 904d9019cfb836a3cc1505dd8769b8cea124b90207ab934f445206c8ffd35a11 not found: ID does not exist" Nov 02 17:56:36.604383 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.604355 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf"] Nov 02 17:56:36.610070 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.610046 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f78de-d77d899f5-wpbsf"] Nov 02 17:56:36.977976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:36.977846 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:37.138666 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:37.138569 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:37.661972 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:37.661511 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:38.002678 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.002473 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:38.053973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.050770 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn" podUID="da93fd4f-c062-43b0-add5-167738bd7489" containerName="payload-processing" containerID="cri-o://2f3d129e8faf68e3e280507b71bd2ed741352e0bba1a43d69bf866ce5afa6b56" gracePeriod=25 Nov 02 17:56:38.069613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.069421 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc" podUID="6f268007-ed65-4c71-a091-9a8fa16f4e43" containerName="payload-pre-processing" containerID="cri-o://57d70d71a596b3f3f18654bf0b2d482370045920ca9caac15a408fb0719f257e" gracePeriod=25 Nov 02 17:56:38.327286 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.326430 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd40957c-0e44-445b-8274-afa6ba043fc5" path="/var/lib/kubelet/pods/dd40957c-0e44-445b-8274-afa6ba043fc5/volumes" Nov 02 17:56:38.364036 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.364006 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn" Nov 02 17:56:38.405208 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.405177 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc" Nov 02 17:56:38.440709 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.440676 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f268007-ed65-4c71-a091-9a8fa16f4e43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f268007-ed65-4c71-a091-9a8fa16f4e43-plugins-config-volume\") pod \"6f268007-ed65-4c71-a091-9a8fa16f4e43\" (UID: \"6f268007-ed65-4c71-a091-9a8fa16f4e43\") " Nov 02 17:56:38.440917 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.440799 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da93fd4f-c062-43b0-add5-167738bd7489-kube-api-access-84xhx\" (UniqueName: \"kubernetes.io/projected/da93fd4f-c062-43b0-add5-167738bd7489-kube-api-access-84xhx\") pod \"da93fd4f-c062-43b0-add5-167738bd7489\" (UID: \"da93fd4f-c062-43b0-add5-167738bd7489\") " Nov 02 17:56:38.441050 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.440892 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da93fd4f-c062-43b0-add5-167738bd7489-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da93fd4f-c062-43b0-add5-167738bd7489-plugins-config-volume\") pod \"da93fd4f-c062-43b0-add5-167738bd7489\" (UID: \"da93fd4f-c062-43b0-add5-167738bd7489\") " Nov 02 17:56:38.441470 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.441193 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f268007-ed65-4c71-a091-9a8fa16f4e43-kube-api-access-75nmx\" (UniqueName: \"kubernetes.io/projected/6f268007-ed65-4c71-a091-9a8fa16f4e43-kube-api-access-75nmx\") pod \"6f268007-ed65-4c71-a091-9a8fa16f4e43\" (UID: \"6f268007-ed65-4c71-a091-9a8fa16f4e43\") " Nov 02 17:56:38.441470 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.441276 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da93fd4f-c062-43b0-add5-167738bd7489-plugins-config-volume" pod "da93fd4f-c062-43b0-add5-167738bd7489" (UID: "da93fd4f-c062-43b0-add5-167738bd7489"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:38.441938 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.441892 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da93fd4f-c062-43b0-add5-167738bd7489-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:38.442136 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.442116 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f268007-ed65-4c71-a091-9a8fa16f4e43-plugins-config-volume" pod "6f268007-ed65-4c71-a091-9a8fa16f4e43" (UID: "6f268007-ed65-4c71-a091-9a8fa16f4e43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:38.443372 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.443338 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da93fd4f-c062-43b0-add5-167738bd7489-kube-api-access-84xhx" pod "da93fd4f-c062-43b0-add5-167738bd7489" (UID: "da93fd4f-c062-43b0-add5-167738bd7489"). InnerVolumeSpecName "kube-api-access-84xhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:38.444195 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.444169 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f268007-ed65-4c71-a091-9a8fa16f4e43-kube-api-access-75nmx" pod "6f268007-ed65-4c71-a091-9a8fa16f4e43" (UID: "6f268007-ed65-4c71-a091-9a8fa16f4e43"). InnerVolumeSpecName "kube-api-access-75nmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:38.562465 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.558717 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84xhx\" (UniqueName: \"kubernetes.io/projected/da93fd4f-c062-43b0-add5-167738bd7489-kube-api-access-84xhx\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:38.562465 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.558754 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75nmx\" (UniqueName: \"kubernetes.io/projected/6f268007-ed65-4c71-a091-9a8fa16f4e43-kube-api-access-75nmx\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:38.562465 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.558767 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f268007-ed65-4c71-a091-9a8fa16f4e43-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:38.609592 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.607184 5549 generic.go:356] "Generic (PLEG): container finished" podID="6f268007-ed65-4c71-a091-9a8fa16f4e43" containerID="57d70d71a596b3f3f18654bf0b2d482370045920ca9caac15a408fb0719f257e" exitCode=0 Nov 02 17:56:38.609592 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.607252 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc" Nov 02 17:56:38.609592 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.607275 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc" event={"ID":"6f268007-ed65-4c71-a091-9a8fa16f4e43","Type":"ContainerDied","Data":"57d70d71a596b3f3f18654bf0b2d482370045920ca9caac15a408fb0719f257e"} Nov 02 17:56:38.609592 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.607339 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc" event={"ID":"6f268007-ed65-4c71-a091-9a8fa16f4e43","Type":"ContainerDied","Data":"be4bea94eaddc3c977e7866e341536d73de5d182370dea065a309e6431d17f95"} Nov 02 17:56:38.609592 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.607365 5549 scope.go:122] "RemoveContainer" containerID="57d70d71a596b3f3f18654bf0b2d482370045920ca9caac15a408fb0719f257e" Nov 02 17:56:38.616200 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.616027 5549 generic.go:356] "Generic (PLEG): container finished" podID="da93fd4f-c062-43b0-add5-167738bd7489" containerID="2f3d129e8faf68e3e280507b71bd2ed741352e0bba1a43d69bf866ce5afa6b56" exitCode=0 Nov 02 17:56:38.616309 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.616268 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn" event={"ID":"da93fd4f-c062-43b0-add5-167738bd7489","Type":"ContainerDied","Data":"2f3d129e8faf68e3e280507b71bd2ed741352e0bba1a43d69bf866ce5afa6b56"} Nov 02 17:56:38.616476 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.616428 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn" event={"ID":"da93fd4f-c062-43b0-add5-167738bd7489","Type":"ContainerDied","Data":"09d7fb3979c2b99c07e863492446813a9ff4ae7a97bbac118841b83a477ae79d"} Nov 02 17:56:38.616595 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.616562 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn" Nov 02 17:56:38.637229 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.637193 5549 scope.go:122] "RemoveContainer" containerID="57d70d71a596b3f3f18654bf0b2d482370045920ca9caac15a408fb0719f257e" Nov 02 17:56:38.637830 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:56:38.637804 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57d70d71a596b3f3f18654bf0b2d482370045920ca9caac15a408fb0719f257e\": container with ID starting with 57d70d71a596b3f3f18654bf0b2d482370045920ca9caac15a408fb0719f257e not found: ID does not exist" containerID="57d70d71a596b3f3f18654bf0b2d482370045920ca9caac15a408fb0719f257e" Nov 02 17:56:38.637920 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.637834 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d70d71a596b3f3f18654bf0b2d482370045920ca9caac15a408fb0719f257e"} err="failed to get container status \"57d70d71a596b3f3f18654bf0b2d482370045920ca9caac15a408fb0719f257e\": rpc error: code = NotFound desc = could not find container \"57d70d71a596b3f3f18654bf0b2d482370045920ca9caac15a408fb0719f257e\": container with ID starting with 57d70d71a596b3f3f18654bf0b2d482370045920ca9caac15a408fb0719f257e not found: ID does not exist" Nov 02 17:56:38.637920 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.637852 5549 scope.go:122] "RemoveContainer" containerID="2f3d129e8faf68e3e280507b71bd2ed741352e0bba1a43d69bf866ce5afa6b56" Nov 02 17:56:38.643415 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.643395 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc"] Nov 02 17:56:38.648247 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.648227 5549 scope.go:122] "RemoveContainer" containerID="2f3d129e8faf68e3e280507b71bd2ed741352e0bba1a43d69bf866ce5afa6b56" Nov 02 17:56:38.648567 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:56:38.648542 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3d129e8faf68e3e280507b71bd2ed741352e0bba1a43d69bf866ce5afa6b56\": container with ID starting with 2f3d129e8faf68e3e280507b71bd2ed741352e0bba1a43d69bf866ce5afa6b56 not found: ID does not exist" containerID="2f3d129e8faf68e3e280507b71bd2ed741352e0bba1a43d69bf866ce5afa6b56" Nov 02 17:56:38.648646 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.648575 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3d129e8faf68e3e280507b71bd2ed741352e0bba1a43d69bf866ce5afa6b56"} err="failed to get container status \"2f3d129e8faf68e3e280507b71bd2ed741352e0bba1a43d69bf866ce5afa6b56\": rpc error: code = NotFound desc = could not find container \"2f3d129e8faf68e3e280507b71bd2ed741352e0bba1a43d69bf866ce5afa6b56\": container with ID starting with 2f3d129e8faf68e3e280507b71bd2ed741352e0bba1a43d69bf866ce5afa6b56 not found: ID does not exist" Nov 02 17:56:38.654088 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.654068 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6f78de-bf5596d8c-qmxmc"] Nov 02 17:56:38.667337 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.667311 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn"] Nov 02 17:56:38.670643 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.670622 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6f78de-86c7bb469-8wkcn"] Nov 02 17:56:38.874750 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.874669 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:38.922590 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.922550 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx"] Nov 02 17:56:38.923704 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.923606 5549 cpu_manager.go:414] "containerMap: removing container" podUID="6f268007-ed65-4c71-a091-9a8fa16f4e43" containerName="payload-pre-processing" Nov 02 17:56:38.923704 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.923628 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f268007-ed65-4c71-a091-9a8fa16f4e43" containerName="payload-pre-processing" Nov 02 17:56:38.923704 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.923662 5549 cpu_manager.go:414] "containerMap: removing container" podUID="dd40957c-0e44-445b-8274-afa6ba043fc5" containerName="maas-api" Nov 02 17:56:38.923704 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.923671 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd40957c-0e44-445b-8274-afa6ba043fc5" containerName="maas-api" Nov 02 17:56:38.923704 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.923693 5549 cpu_manager.go:414] "containerMap: removing container" podUID="da93fd4f-c062-43b0-add5-167738bd7489" containerName="payload-processing" Nov 02 17:56:38.923704 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.923701 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da93fd4f-c062-43b0-add5-167738bd7489" containerName="payload-processing" Nov 02 17:56:38.924561 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.923861 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f268007-ed65-4c71-a091-9a8fa16f4e43" containerName="payload-pre-processing" Nov 02 17:56:38.924561 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.923878 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd40957c-0e44-445b-8274-afa6ba043fc5" containerName="maas-api" Nov 02 17:56:38.924561 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.923891 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="da93fd4f-c062-43b0-add5-167738bd7489" containerName="payload-processing" Nov 02 17:56:38.929408 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.929382 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 17:56:38.945711 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.945682 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-2961d5-serving-cert\"" Nov 02 17:56:38.951375 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.951349 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx"] Nov 02 17:56:38.964080 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.964050 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6p6n\" (UniqueName: \"kubernetes.io/projected/a8a52d50-ddee-466d-9630-50d84b3eac4a-kube-api-access-x6p6n\") pod \"maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx\" (UID: \"a8a52d50-ddee-466d-9630-50d84b3eac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 17:56:38.964231 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.964121 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a52d50-ddee-466d-9630-50d84b3eac4a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx\" (UID: \"a8a52d50-ddee-466d-9630-50d84b3eac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 17:56:38.964310 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.964235 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a52d50-ddee-466d-9630-50d84b3eac4a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx\" (UID: \"a8a52d50-ddee-466d-9630-50d84b3eac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 17:56:38.964310 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:38.964274 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a52d50-ddee-466d-9630-50d84b3eac4a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx\" (UID: \"a8a52d50-ddee-466d-9630-50d84b3eac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 17:56:39.058978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:39.058215 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:39.097380 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:39.067142 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6p6n\" (UniqueName: \"kubernetes.io/projected/a8a52d50-ddee-466d-9630-50d84b3eac4a-kube-api-access-x6p6n\") pod \"maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx\" (UID: \"a8a52d50-ddee-466d-9630-50d84b3eac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 17:56:39.097380 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:39.067309 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a52d50-ddee-466d-9630-50d84b3eac4a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx\" (UID: \"a8a52d50-ddee-466d-9630-50d84b3eac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 17:56:39.097380 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:39.067396 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a52d50-ddee-466d-9630-50d84b3eac4a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx\" (UID: \"a8a52d50-ddee-466d-9630-50d84b3eac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 17:56:39.097380 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:39.067459 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a52d50-ddee-466d-9630-50d84b3eac4a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx\" (UID: \"a8a52d50-ddee-466d-9630-50d84b3eac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 17:56:39.097380 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:56:39.072372 5549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-serving-cert: secret "maas-api-e2e-shared-b-w1-2961d5-serving-cert" not found Nov 02 17:56:39.097380 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:56:39.072443 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8a52d50-ddee-466d-9630-50d84b3eac4a-maas-api-tls podName:a8a52d50-ddee-466d-9630-50d84b3eac4a nodeName:}" failed. No retries permitted until 2026-11-02 17:56:39.572421961 +0000 UTC m=+1371.360927471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8a52d50-ddee-466d-9630-50d84b3eac4a-maas-api-tls") pod "maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" (UID: "a8a52d50-ddee-466d-9630-50d84b3eac4a") : secret "maas-api-e2e-shared-b-w1-2961d5-serving-cert" not found Nov 02 17:56:39.097380 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:39.082988 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a52d50-ddee-466d-9630-50d84b3eac4a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx\" (UID: \"a8a52d50-ddee-466d-9630-50d84b3eac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 17:56:39.128287 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:39.098725 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6p6n\" (UniqueName: \"kubernetes.io/projected/a8a52d50-ddee-466d-9630-50d84b3eac4a-kube-api-access-x6p6n\") pod \"maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx\" (UID: \"a8a52d50-ddee-466d-9630-50d84b3eac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 17:56:39.128287 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:39.100009 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a52d50-ddee-466d-9630-50d84b3eac4a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx\" (UID: \"a8a52d50-ddee-466d-9630-50d84b3eac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 17:56:39.210476 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:39.210431 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:39.491316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:39.491205 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:39.577069 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:39.577035 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a52d50-ddee-466d-9630-50d84b3eac4a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx\" (UID: \"a8a52d50-ddee-466d-9630-50d84b3eac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 17:56:39.579590 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:39.579559 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a52d50-ddee-466d-9630-50d84b3eac4a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx\" (UID: \"a8a52d50-ddee-466d-9630-50d84b3eac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 17:56:39.859714 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:39.859625 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 17:56:40.138846 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:40.134668 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:40.166881 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:40.166857 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx"] Nov 02 17:56:40.169091 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:56:40.169063 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a52d50_ddee_466d_9630_50d84b3eac4a.slice/crio-fbb9bb9642429f5c959b608a9ea07757e8744c98ba9e42f9b78080dfe0014f51 WatchSource:0}: Error finding container fbb9bb9642429f5c959b608a9ea07757e8744c98ba9e42f9b78080dfe0014f51: Status 404 returned error can't find the container with id fbb9bb9642429f5c959b608a9ea07757e8744c98ba9e42f9b78080dfe0014f51 Nov 02 17:56:40.320920 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:40.320886 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f268007-ed65-4c71-a091-9a8fa16f4e43" path="/var/lib/kubelet/pods/6f268007-ed65-4c71-a091-9a8fa16f4e43/volumes" Nov 02 17:56:40.321352 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:40.321337 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da93fd4f-c062-43b0-add5-167738bd7489" path="/var/lib/kubelet/pods/da93fd4f-c062-43b0-add5-167738bd7489/volumes" Nov 02 17:56:40.718102 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:40.718062 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" event={"ID":"a8a52d50-ddee-466d-9630-50d84b3eac4a","Type":"ContainerStarted","Data":"0788742d42feff2df6fdbcb778274787c326e49c0d2aef6e5ca3fb32e26120e2"} Nov 02 17:56:40.718599 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:40.718129 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" event={"ID":"a8a52d50-ddee-466d-9630-50d84b3eac4a","Type":"ContainerStarted","Data":"fbb9bb9642429f5c959b608a9ea07757e8744c98ba9e42f9b78080dfe0014f51"} Nov 02 17:56:40.718599 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:40.718283 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 17:56:40.751820 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:40.751758 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" podStartSLOduration=2.3824755570000002 podStartE2EDuration="2.751743411s" totalImagesPullingTime="369.267854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:40.750355763 +0000 UTC m=+1372.538861305" watchObservedRunningTime="2026-11-02 17:56:40.751743411 +0000 UTC m=+1372.540248939" Nov 02 17:56:40.842230 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:40.842148 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:41.007758 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:41.003595 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:41.274976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:41.271420 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:41.531501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:41.531419 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" Nov 02 17:56:41.531679 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:41.531515 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" Nov 02 17:56:41.977308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:41.977261 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:42.080644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.080610 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55"] Nov 02 17:56:42.085307 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.085282 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" Nov 02 17:56:42.092979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.092935 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55"] Nov 02 17:56:42.095076 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.095054 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-7771bf-dockercfg-x4mgk\"" Nov 02 17:56:42.095221 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.095059 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-7771bf\"" Nov 02 17:56:42.119210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.119160 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq"] Nov 02 17:56:42.125594 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.125560 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" Nov 02 17:56:42.130175 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.130141 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq"] Nov 02 17:56:42.155734 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.155698 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:42.164736 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.160886 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb222\" (UniqueName: \"kubernetes.io/projected/f2ab9120-6ff3-489c-8dd2-bfb0f1201172-kube-api-access-vb222\") pod \"payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55\" (UID: \"f2ab9120-6ff3-489c-8dd2-bfb0f1201172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" Nov 02 17:56:42.164736 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.160978 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ab9120-6ff3-489c-8dd2-bfb0f1201172-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55\" (UID: \"f2ab9120-6ff3-489c-8dd2-bfb0f1201172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" Nov 02 17:56:42.347033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.344860 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a011b9f5-8629-4514-b5db-52b58316d975-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq\" (UID: \"a011b9f5-8629-4514-b5db-52b58316d975\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" Nov 02 17:56:42.347033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.344965 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h72qd\" (UniqueName: \"kubernetes.io/projected/a011b9f5-8629-4514-b5db-52b58316d975-kube-api-access-h72qd\") pod \"payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq\" (UID: \"a011b9f5-8629-4514-b5db-52b58316d975\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" Nov 02 17:56:42.347033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.345013 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb222\" (UniqueName: \"kubernetes.io/projected/f2ab9120-6ff3-489c-8dd2-bfb0f1201172-kube-api-access-vb222\") pod \"payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55\" (UID: \"f2ab9120-6ff3-489c-8dd2-bfb0f1201172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" Nov 02 17:56:42.347033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.345083 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ab9120-6ff3-489c-8dd2-bfb0f1201172-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55\" (UID: \"f2ab9120-6ff3-489c-8dd2-bfb0f1201172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" Nov 02 17:56:42.357194 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.354419 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:42.361820 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.361782 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ab9120-6ff3-489c-8dd2-bfb0f1201172-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55\" (UID: \"f2ab9120-6ff3-489c-8dd2-bfb0f1201172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" Nov 02 17:56:42.436696 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.436632 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb222\" (UniqueName: \"kubernetes.io/projected/f2ab9120-6ff3-489c-8dd2-bfb0f1201172-kube-api-access-vb222\") pod \"payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55\" (UID: \"f2ab9120-6ff3-489c-8dd2-bfb0f1201172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" Nov 02 17:56:42.446022 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.445977 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a011b9f5-8629-4514-b5db-52b58316d975-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq\" (UID: \"a011b9f5-8629-4514-b5db-52b58316d975\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" Nov 02 17:56:42.446219 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.446068 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h72qd\" (UniqueName: \"kubernetes.io/projected/a011b9f5-8629-4514-b5db-52b58316d975-kube-api-access-h72qd\") pod \"payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq\" (UID: \"a011b9f5-8629-4514-b5db-52b58316d975\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" Nov 02 17:56:42.446708 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.446684 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a011b9f5-8629-4514-b5db-52b58316d975-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq\" (UID: \"a011b9f5-8629-4514-b5db-52b58316d975\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" Nov 02 17:56:42.457541 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.457505 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h72qd\" (UniqueName: \"kubernetes.io/projected/a011b9f5-8629-4514-b5db-52b58316d975-kube-api-access-h72qd\") pod \"payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq\" (UID: \"a011b9f5-8629-4514-b5db-52b58316d975\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" Nov 02 17:56:42.702940 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.702900 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" Nov 02 17:56:42.741408 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.741371 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" Nov 02 17:56:42.993250 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:42.993223 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55"] Nov 02 17:56:43.079222 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:43.071668 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq"] Nov 02 17:56:43.154015 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:43.143688 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:43.460744 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:43.459786 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:43.655238 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:43.655153 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:43.743651 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:43.743606 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" event={"ID":"f2ab9120-6ff3-489c-8dd2-bfb0f1201172","Type":"ContainerStarted","Data":"b0356e704211f1e024e9c0b35f922726e65dd2668566094309212fafdf7d74b3"} Nov 02 17:56:43.743857 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:43.743679 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" event={"ID":"f2ab9120-6ff3-489c-8dd2-bfb0f1201172","Type":"ContainerStarted","Data":"8c6ee07738772d489247de8a60a6dc40c2ae669a3148d343612745e2db04bca9"} Nov 02 17:56:43.743857 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:43.743712 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" Nov 02 17:56:43.745479 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:43.745450 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" event={"ID":"a011b9f5-8629-4514-b5db-52b58316d975","Type":"ContainerStarted","Data":"f6efcbcf63da2784a53bf62ee323c90942de6d75cdf08738cc06b45efef86d97"} Nov 02 17:56:43.745597 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:43.745499 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" Nov 02 17:56:43.745597 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:43.745517 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" event={"ID":"a011b9f5-8629-4514-b5db-52b58316d975","Type":"ContainerStarted","Data":"954a720d5fd9b41da32f7444e8212b19b8c6eb3e477da2aa8b53a68b00fbc301"} Nov 02 17:56:43.769411 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:43.769346 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" podStartSLOduration=1.769333305 podStartE2EDuration="1.769333305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:43.768190472 +0000 UTC m=+1375.556696000" watchObservedRunningTime="2026-11-02 17:56:43.769333305 +0000 UTC m=+1375.557838874" Nov 02 17:56:43.797281 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:43.797215 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" podStartSLOduration=1.7971981129999999 podStartE2EDuration="1.797198113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:43.795003842 +0000 UTC m=+1375.583509371" watchObservedRunningTime="2026-11-02 17:56:43.797198113 +0000 UTC m=+1375.585703642" Nov 02 17:56:43.990205 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:43.988255 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:44.203779 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:44.203740 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:44.384050 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:44.374653 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:45.012127 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.012081 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz"] Nov 02 17:56:45.018300 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.018271 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.026085 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.026055 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-47e1c2\"/\"kube-root-ca.crt\"" Nov 02 17:56:45.026283 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.026259 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-47e1c2\"/\"openshift-service-ca.crt\"" Nov 02 17:56:45.026376 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.026182 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-47e1c2\"/\"auth-test-model-47e1c2-kserve-self-signed-certs\"" Nov 02 17:56:45.026376 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.026080 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-47e1c2\"/\"default-dockercfg-84qnw\"" Nov 02 17:56:45.030850 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.030825 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz"] Nov 02 17:56:45.110498 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.110464 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:45.147482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.147444 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-home\") pod \"auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.147482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.147485 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-tmp-dir\") pod \"auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.147823 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.147642 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-dshm\") pod \"auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.147823 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.147691 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-model-cache\") pod \"auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.147987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.147881 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc4054d-d2ce-4d98-8daa-a70690cf863e-tls-certs\") pod \"auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.148047 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.147998 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnhvt\" (UniqueName: \"kubernetes.io/projected/8bc4054d-d2ce-4d98-8daa-a70690cf863e-kube-api-access-xnhvt\") pod \"auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.261027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.256542 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-dshm\") pod \"auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.261027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.257300 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-model-cache\") pod \"auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.261027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.257653 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc4054d-d2ce-4d98-8daa-a70690cf863e-tls-certs\") pod \"auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.261027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.258436 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnhvt\" (UniqueName: \"kubernetes.io/projected/8bc4054d-d2ce-4d98-8daa-a70690cf863e-kube-api-access-xnhvt\") pod \"auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.261027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.258687 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-model-cache\") pod \"auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.261027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.258728 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-home\") pod \"auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.261027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.258762 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-tmp-dir\") pod \"auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.275022 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.265336 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc4054d-d2ce-4d98-8daa-a70690cf863e-tls-certs\") pod \"auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.275270 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.269200 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-home\") pod \"auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.275402 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.269372 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-tmp-dir\") pod \"auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.284048 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.283506 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-dshm\") pod \"auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.332367 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.326446 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnhvt\" (UniqueName: \"kubernetes.io/projected/8bc4054d-d2ce-4d98-8daa-a70690cf863e-kube-api-access-xnhvt\") pod \"auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.336118 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.335466 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:45.336118 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.335731 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:45.728298 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.728265 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz"] Nov 02 17:56:45.729967 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:56:45.729932 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bc4054d_d2ce_4d98_8daa_a70690cf863e.slice/crio-c74e1f8883311fcfeebc43d6f53c81a97dec11bf14043e9e37587b3ba38fceae WatchSource:0}: Error finding container c74e1f8883311fcfeebc43d6f53c81a97dec11bf14043e9e37587b3ba38fceae: Status 404 returned error can't find the container with id c74e1f8883311fcfeebc43d6f53c81a97dec11bf14043e9e37587b3ba38fceae Nov 02 17:56:45.760270 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:45.760234 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" event={"ID":"8bc4054d-d2ce-4d98-8daa-a70690cf863e","Type":"ContainerStarted","Data":"c74e1f8883311fcfeebc43d6f53c81a97dec11bf14043e9e37587b3ba38fceae"} Nov 02 17:56:46.033668 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:46.033586 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:56:46.298033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:46.290495 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:46.732515 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:46.732488 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 17:56:46.769086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:46.769044 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" event={"ID":"8bc4054d-d2ce-4d98-8daa-a70690cf863e","Type":"ContainerStarted","Data":"bc3f9f0d925afb77c8d668ce610b6a6c4b95f8510e8ca2f941d79331e35d3cd0"} Nov 02 17:56:46.800717 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:46.800654 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" podStartSLOduration=2.800636787 podStartE2EDuration="2.800636787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:46.799762443 +0000 UTC m=+1378.588267971" watchObservedRunningTime="2026-11-02 17:56:46.800636787 +0000 UTC m=+1378.589142317" Nov 02 17:56:47.336768 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:47.336731 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:47.454976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:47.449607 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:47.625172 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:56:47.625077 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:56:48.572620 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:48.572583 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:49.481733 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:49.481694 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:50.291807 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:50.291767 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:50.657635 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:50.657546 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:51.109820 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:51.109788 5549 scope.go:122] "RemoveContainer" containerID="591f6abdc757c352f2f1c7a94878247cd9141a138fa32ba295e766ac1d44276d" Nov 02 17:56:51.268977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:51.266936 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:52.244981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:52.244690 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:53.020403 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.020363 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:53.020797 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.020780 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk"] Nov 02 17:56:53.029490 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.029454 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk"] Nov 02 17:56:53.029662 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.029567 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 17:56:53.042750 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.042707 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-7771bf-serving-cert\"" Nov 02 17:56:53.212180 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.212135 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:53.308772 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.308659 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15281fd-8907-4d15-a06e-85a772e3c837-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk\" (UID: \"c15281fd-8907-4d15-a06e-85a772e3c837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 17:56:53.308772 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.308731 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15281fd-8907-4d15-a06e-85a772e3c837-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk\" (UID: \"c15281fd-8907-4d15-a06e-85a772e3c837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 17:56:53.309359 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.308822 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15281fd-8907-4d15-a06e-85a772e3c837-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk\" (UID: \"c15281fd-8907-4d15-a06e-85a772e3c837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 17:56:53.309359 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.308862 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4qvd\" (UniqueName: \"kubernetes.io/projected/c15281fd-8907-4d15-a06e-85a772e3c837-kube-api-access-l4qvd\") pod \"maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk\" (UID: \"c15281fd-8907-4d15-a06e-85a772e3c837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 17:56:53.411493 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.411447 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15281fd-8907-4d15-a06e-85a772e3c837-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk\" (UID: \"c15281fd-8907-4d15-a06e-85a772e3c837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 17:56:53.411694 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.411515 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15281fd-8907-4d15-a06e-85a772e3c837-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk\" (UID: \"c15281fd-8907-4d15-a06e-85a772e3c837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 17:56:53.411694 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.411542 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15281fd-8907-4d15-a06e-85a772e3c837-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk\" (UID: \"c15281fd-8907-4d15-a06e-85a772e3c837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 17:56:53.411694 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.411612 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4qvd\" (UniqueName: \"kubernetes.io/projected/c15281fd-8907-4d15-a06e-85a772e3c837-kube-api-access-l4qvd\") pod \"maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk\" (UID: \"c15281fd-8907-4d15-a06e-85a772e3c837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 17:56:53.411909 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:56:53.411719 5549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-serving-cert: secret "maas-api-e2e-shared-b-w0-7771bf-serving-cert" not found Nov 02 17:56:53.411909 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:56:53.411781 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c15281fd-8907-4d15-a06e-85a772e3c837-maas-api-tls podName:c15281fd-8907-4d15-a06e-85a772e3c837 nodeName:}" failed. No retries permitted until 2026-11-02 17:56:53.911760815 +0000 UTC m=+1385.700266321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c15281fd-8907-4d15-a06e-85a772e3c837-maas-api-tls") pod "maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" (UID: "c15281fd-8907-4d15-a06e-85a772e3c837") : secret "maas-api-e2e-shared-b-w0-7771bf-serving-cert" not found Nov 02 17:56:53.412405 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.412379 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15281fd-8907-4d15-a06e-85a772e3c837-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk\" (UID: \"c15281fd-8907-4d15-a06e-85a772e3c837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 17:56:53.414116 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.414095 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15281fd-8907-4d15-a06e-85a772e3c837-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk\" (UID: \"c15281fd-8907-4d15-a06e-85a772e3c837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 17:56:53.421907 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.421874 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4qvd\" (UniqueName: \"kubernetes.io/projected/c15281fd-8907-4d15-a06e-85a772e3c837-kube-api-access-l4qvd\") pod \"maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk\" (UID: \"c15281fd-8907-4d15-a06e-85a772e3c837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 17:56:53.657036 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.649033 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:53.959618 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.959582 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15281fd-8907-4d15-a06e-85a772e3c837-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk\" (UID: \"c15281fd-8907-4d15-a06e-85a772e3c837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 17:56:53.962162 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:53.962128 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15281fd-8907-4d15-a06e-85a772e3c837-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk\" (UID: \"c15281fd-8907-4d15-a06e-85a772e3c837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 17:56:54.125108 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:54.125070 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 17:56:54.225414 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:54.211845 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:54.432463 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:54.432431 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk"] Nov 02 17:56:54.432942 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:56:54.432923 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15281fd_8907_4d15_a06e_85a772e3c837.slice/crio-d76e8349513f270716369bd3fd62b4b3f1b52a018321cb63bc4d1e0f8096ec95 WatchSource:0}: Error finding container d76e8349513f270716369bd3fd62b4b3f1b52a018321cb63bc4d1e0f8096ec95: Status 404 returned error can't find the container with id d76e8349513f270716369bd3fd62b4b3f1b52a018321cb63bc4d1e0f8096ec95 Nov 02 17:56:54.755279 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:54.755246 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" Nov 02 17:56:54.755453 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:54.755314 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" Nov 02 17:56:54.854262 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:54.854226 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" event={"ID":"c15281fd-8907-4d15-a06e-85a772e3c837","Type":"ContainerStarted","Data":"d76e8349513f270716369bd3fd62b4b3f1b52a018321cb63bc4d1e0f8096ec95"} Nov 02 17:56:54.960787 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:54.959468 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:55.336566 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:55.336479 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:55.349643 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:55.349617 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:55.883300 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:55.883258 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" event={"ID":"c15281fd-8907-4d15-a06e-85a772e3c837","Type":"ContainerStarted","Data":"b3efb0073cc9509bcb74ed53acaff1ad21be1cf4811d6a25651173059fcebbea"} Nov 02 17:56:55.884255 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:55.884194 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 17:56:55.885455 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:55.885251 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:55.939625 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:55.939556 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" podStartSLOduration=3.487020957 podStartE2EDuration="3.939536648s" totalImagesPullingTime="452.515691ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:55.912680997 +0000 UTC m=+1387.701186592" watchObservedRunningTime="2026-11-02 17:56:55.939536648 +0000 UTC m=+1387.728042177" Nov 02 17:56:55.949421 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:55.949197 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 17:56:56.894046 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:56.894002 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:57.661799 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:57.661761 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:57.843916 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:57.843875 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:58.648675 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:58.648607 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:58.974483 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:58.974448 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:59.153989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:59.151122 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:59.295611 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:59.295520 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:59.563031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:56:59.562833 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:00.399978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:00.398529 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:01.233002 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:01.226484 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:01.895671 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:01.895639 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 17:57:02.013330 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:02.013272 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:02.748286 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:02.747757 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:03.552298 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:03.552237 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:04.395715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:04.395671 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:05.136347 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:05.136291 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:06.220603 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:06.220562 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:07.166024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:07.159589 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:07.310749 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:07.310716 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:07.704259 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:07.704220 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:08.238828 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.238795 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:08.412292 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.412259 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:08.736542 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.736499 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82"] Nov 02 17:57:08.742470 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.742442 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.764931 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.764900 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82"] Nov 02 17:57:08.831093 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.831058 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-b6b6f4-openshift-default-dockercfg-rnffh\"" Nov 02 17:57:08.833037 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.833010 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.833143 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.833045 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.833143 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.833080 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.833305 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.833284 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.833409 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.833387 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.833506 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.833486 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9xqv\" (UniqueName: \"kubernetes.io/projected/40e3fde9-2048-40f8-8a84-5004a426c81f-kube-api-access-s9xqv\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.833579 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.833549 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e3fde9-2048-40f8-8a84-5004a426c81f-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.833674 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.833660 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.833715 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.833708 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.914077 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.914036 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds"] Nov 02 17:57:08.918985 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.918947 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:08.931814 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.931781 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-b6b6f4\"/\"auth-test-model-b6b6f4-kserve-self-signed-certs\"" Nov 02 17:57:08.932232 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.932187 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-b6b6f4\"/\"default-dockercfg-c2wgh\"" Nov 02 17:57:08.936416 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.936378 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.936547 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.936461 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9xqv\" (UniqueName: \"kubernetes.io/projected/40e3fde9-2048-40f8-8a84-5004a426c81f-kube-api-access-s9xqv\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.936547 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.936520 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e3fde9-2048-40f8-8a84-5004a426c81f-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.936692 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.936595 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.936692 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.936663 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.936809 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.936730 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.936809 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.936757 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.940125 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.940102 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e3fde9-2048-40f8-8a84-5004a426c81f-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.945050 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.941041 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.945050 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.941348 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.945050 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.943518 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.945304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.944369 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.945304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.944340 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.945304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.945153 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds"] Nov 02 17:57:08.945304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.944689 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.945304 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.945241 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.945676 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.945658 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-b6b6f4\"/\"openshift-service-ca.crt\"" Nov 02 17:57:08.945733 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.945702 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.945886 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.945870 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-b6b6f4\"/\"kube-root-ca.crt\"" Nov 02 17:57:08.950196 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.950170 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.954276 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.954250 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9xqv\" (UniqueName: \"kubernetes.io/projected/40e3fde9-2048-40f8-8a84-5004a426c81f-kube-api-access-s9xqv\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:08.972868 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:08.972837 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:09.046462 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.046371 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh996\" (UniqueName: \"kubernetes.io/projected/d16de1e2-a7a7-4046-90e8-85763b266321-kube-api-access-qh996\") pod \"auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.046462 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.046456 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-dshm\") pod \"auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.046665 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.046579 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-tmp-dir\") pod \"auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.046665 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.046623 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d16de1e2-a7a7-4046-90e8-85763b266321-tls-certs\") pod \"auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.046751 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.046734 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-home\") pod \"auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.046789 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.046778 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-model-cache\") pod \"auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.064441 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.064405 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:09.134470 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.134437 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:09.159987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.159385 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh996\" (UniqueName: \"kubernetes.io/projected/d16de1e2-a7a7-4046-90e8-85763b266321-kube-api-access-qh996\") pod \"auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.159987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.159527 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-dshm\") pod \"auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.159987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.159654 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-tmp-dir\") pod \"auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.159987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.159703 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d16de1e2-a7a7-4046-90e8-85763b266321-tls-certs\") pod \"auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.159987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.159796 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-home\") pod \"auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.159987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.159856 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-model-cache\") pod \"auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.160479 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.160357 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-model-cache\") pod \"auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.194976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.194324 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-tmp-dir\") pod \"auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.219975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.208401 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-home\") pod \"auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.244977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.244620 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh996\" (UniqueName: \"kubernetes.io/projected/d16de1e2-a7a7-4046-90e8-85763b266321-kube-api-access-qh996\") pod \"auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.288991 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.288289 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-dshm\") pod \"auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.303296 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.303184 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d16de1e2-a7a7-4046-90e8-85763b266321-tls-certs\") pod \"auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.562636 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.562605 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82"] Nov 02 17:57:09.563160 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:57:09.563135 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e3fde9_2048_40f8_8a84_5004a426c81f.slice/crio-ff90c870fec21a406449a68adf48f050d0b9c05d9dd57878e14e7e08b912aa77 WatchSource:0}: Error finding container ff90c870fec21a406449a68adf48f050d0b9c05d9dd57878e14e7e08b912aa77: Status 404 returned error can't find the container with id ff90c870fec21a406449a68adf48f050d0b9c05d9dd57878e14e7e08b912aa77 Nov 02 17:57:09.566438 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.566393 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:09.566545 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.566490 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:09.566545 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.566534 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:09.588250 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.588215 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:09.975069 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:09.956242 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:10.040505 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:10.040455 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds"] Nov 02 17:57:10.048155 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:57:10.048111 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd16de1e2_a7a7_4046_90e8_85763b266321.slice/crio-9d5a35bebcd222b8724c6c564ef0d65beee72d9618c67d42a327386d98f7cf17 WatchSource:0}: Error finding container 9d5a35bebcd222b8724c6c564ef0d65beee72d9618c67d42a327386d98f7cf17: Status 404 returned error can't find the container with id 9d5a35bebcd222b8724c6c564ef0d65beee72d9618c67d42a327386d98f7cf17 Nov 02 17:57:10.076432 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:10.076394 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" event={"ID":"40e3fde9-2048-40f8-8a84-5004a426c81f","Type":"ContainerStarted","Data":"544c86e6cd5ceae36bae075be436ab64478ecd91afef5957834ff94e476000d0"} Nov 02 17:57:10.076576 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:10.076460 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" event={"ID":"40e3fde9-2048-40f8-8a84-5004a426c81f","Type":"ContainerStarted","Data":"ff90c870fec21a406449a68adf48f050d0b9c05d9dd57878e14e7e08b912aa77"} Nov 02 17:57:10.109506 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:10.109457 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" podStartSLOduration=2.109439003 podStartE2EDuration="2.109439003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:10.106377171 +0000 UTC m=+1401.894882699" watchObservedRunningTime="2026-11-02 17:57:10.109439003 +0000 UTC m=+1401.897944986" Nov 02 17:57:11.065295 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:11.065259 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:11.067194 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:11.067165 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 02 17:57:11.067332 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:11.067211 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" podUID="40e3fde9-2048-40f8-8a84-5004a426c81f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 02 17:57:11.109136 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:11.109092 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" event={"ID":"d16de1e2-a7a7-4046-90e8-85763b266321","Type":"ContainerStarted","Data":"1a98c7d99b4d73b6b387f30afe43f487bd37713dd4be6d63b31186578ba3d08c"} Nov 02 17:57:11.109311 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:11.109150 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" event={"ID":"d16de1e2-a7a7-4046-90e8-85763b266321","Type":"ContainerStarted","Data":"9d5a35bebcd222b8724c6c564ef0d65beee72d9618c67d42a327386d98f7cf17"} Nov 02 17:57:11.141244 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:11.141187 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" podStartSLOduration=3.141170524 podStartE2EDuration="3.141170524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:11.139157539 +0000 UTC m=+1402.927663068" watchObservedRunningTime="2026-11-02 17:57:11.141170524 +0000 UTC m=+1402.929676051" Nov 02 17:57:11.236004 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:11.232570 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:11.391706 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:11.389424 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:11.606017 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:11.598043 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:12.039254 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:12.039217 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:12.066994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:12.065764 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 02 17:57:12.066994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:12.065834 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" podUID="40e3fde9-2048-40f8-8a84-5004a426c81f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 02 17:57:12.228974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:12.226606 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:13.074065 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:13.069426 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 02 17:57:13.074065 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:13.069494 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" podUID="40e3fde9-2048-40f8-8a84-5004a426c81f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 02 17:57:13.074065 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:13.071343 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:13.411277 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:13.411193 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:13.643982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:13.634403 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:14.065700 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:14.065659 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 02 17:57:14.065885 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:14.065720 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" podUID="40e3fde9-2048-40f8-8a84-5004a426c81f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 02 17:57:14.368716 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:14.365509 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:15.065473 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.065432 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 02 17:57:15.065665 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.065501 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" podUID="40e3fde9-2048-40f8-8a84-5004a426c81f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 02 17:57:15.227652 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.227613 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb"] Nov 02 17:57:15.233665 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.233637 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.244481 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.244454 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-9165c3\"/\"openshift-service-ca.crt\"" Nov 02 17:57:15.246351 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.246307 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-9165c3\"/\"kube-root-ca.crt\"" Nov 02 17:57:15.246539 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.246519 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-9165c3\"/\"auth-test-model-9165c3-kserve-self-signed-certs\"" Nov 02 17:57:15.246739 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.246676 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-9165c3\"/\"default-dockercfg-njhzl\"" Nov 02 17:57:15.248134 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.248110 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb"] Nov 02 17:57:15.270372 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.270322 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:15.270847 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.270822 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-tmp-dir\") pod \"auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.271012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.270891 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-dshm\") pod \"auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.271134 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.271022 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-model-cache\") pod \"auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.271134 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.271086 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-home\") pod \"auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.271300 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.271136 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbtz9\" (UniqueName: \"kubernetes.io/projected/7b70cca5-bca5-4aaa-97da-4cf46d011a44-kube-api-access-cbtz9\") pod \"auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.271300 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.271163 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b70cca5-bca5-4aaa-97da-4cf46d011a44-tls-certs\") pod \"auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.373019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.372903 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-tmp-dir\") pod \"auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.373019 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.372999 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-dshm\") pod \"auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.373554 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.373083 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-model-cache\") pod \"auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.373554 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.373130 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-home\") pod \"auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.373554 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.373175 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbtz9\" (UniqueName: \"kubernetes.io/projected/7b70cca5-bca5-4aaa-97da-4cf46d011a44-kube-api-access-cbtz9\") pod \"auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.373554 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.373200 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b70cca5-bca5-4aaa-97da-4cf46d011a44-tls-certs\") pod \"auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.373554 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.373388 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-tmp-dir\") pod \"auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.373554 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.373413 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-model-cache\") pod \"auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.373554 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.373534 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-home\") pod \"auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.375358 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.375336 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-dshm\") pod \"auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.375772 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.375750 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b70cca5-bca5-4aaa-97da-4cf46d011a44-tls-certs\") pod \"auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.407135 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.407098 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbtz9\" (UniqueName: \"kubernetes.io/projected/7b70cca5-bca5-4aaa-97da-4cf46d011a44-kube-api-access-cbtz9\") pod \"auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.426205 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.426172 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:15.571977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.571647 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:15.939866 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.939781 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q"] Nov 02 17:57:15.947221 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.947194 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:15.955654 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:15.955484 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q"] Nov 02 17:57:16.007245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.006367 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb2197f-704a-47aa-8b7b-0094baa86a96-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.007245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.006555 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.007245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.006585 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.007245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.006667 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.007245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.006750 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.007245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.006850 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.007245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.006894 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.007245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.007090 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt6js\" (UniqueName: \"kubernetes.io/projected/6eb2197f-704a-47aa-8b7b-0094baa86a96-kube-api-access-zt6js\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.007245 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.007175 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.012870 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.012846 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb"] Nov 02 17:57:16.027189 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.027150 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-1e2281-openshift-default-dockercfg-xhl4q\"" Nov 02 17:57:16.065475 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.065443 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 02 17:57:16.065627 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.065503 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" podUID="40e3fde9-2048-40f8-8a84-5004a426c81f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 02 17:57:16.108731 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.108700 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.108875 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.108738 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.108875 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.108815 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt6js\" (UniqueName: \"kubernetes.io/projected/6eb2197f-704a-47aa-8b7b-0094baa86a96-kube-api-access-zt6js\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.108875 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.108870 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.109195 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.108975 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb2197f-704a-47aa-8b7b-0094baa86a96-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.109195 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.109079 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.109195 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.109105 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.109195 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.109143 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.109195 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.109187 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.110053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.109404 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.110053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.109685 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.110053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.109691 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.110053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.109896 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.110053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.109987 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb2197f-704a-47aa-8b7b-0094baa86a96-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.112221 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.112193 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.112385 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.112367 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.128107 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.128071 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.129467 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.129435 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt6js\" (UniqueName: \"kubernetes.io/projected/6eb2197f-704a-47aa-8b7b-0094baa86a96-kube-api-access-zt6js\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.175898 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.175847 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:16.186010 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.185946 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" event={"ID":"7b70cca5-bca5-4aaa-97da-4cf46d011a44","Type":"ContainerStarted","Data":"c8525140fa1af86e98e416909d5d8bf9b1a146809b4545169a24b37f8f2c5398"} Nov 02 17:57:16.186210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.186026 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" event={"ID":"7b70cca5-bca5-4aaa-97da-4cf46d011a44","Type":"ContainerStarted","Data":"5bd1f29ee9790e1e6034e9923e508e8b7a630a5dc9281ca94de1132b4431ee8b"} Nov 02 17:57:16.212500 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.212446 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" podStartSLOduration=1.212432885 podStartE2EDuration="1.212432885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:16.21054571 +0000 UTC m=+1407.999051233" watchObservedRunningTime="2026-11-02 17:57:16.212432885 +0000 UTC m=+1408.000938412" Nov 02 17:57:16.244068 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.244029 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg"] Nov 02 17:57:16.249466 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.249432 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.264412 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.264378 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg"] Nov 02 17:57:16.277378 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.277349 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:16.317432 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.315851 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-data\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.317432 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.315990 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j589r\" (UniqueName: \"kubernetes.io/projected/0c3b2112-be30-40ae-bc4b-84d75341a926-kube-api-access-j589r\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.317432 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.316120 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-token\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.317432 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.316207 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-credential-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.317432 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.316288 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-podinfo\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.317432 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.316332 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-workload-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.317432 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.316370 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3b2112-be30-40ae-bc4b-84d75341a926-istiod-ca-cert\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.317432 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.316414 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-envoy\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.317432 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.316438 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-workload-certs\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.366975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.358610 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-b1a788-openshift-default-dockercfg-kv6gv\"" Nov 02 17:57:16.434148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.418239 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-credential-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.434148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.418333 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-podinfo\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.434148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.418379 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-workload-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.434148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.418416 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3b2112-be30-40ae-bc4b-84d75341a926-istiod-ca-cert\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.434148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.418475 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-envoy\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.434148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.418500 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-workload-certs\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.434148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.418607 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-data\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.434148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.418820 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j589r\" (UniqueName: \"kubernetes.io/projected/0c3b2112-be30-40ae-bc4b-84d75341a926-kube-api-access-j589r\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.434148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.419079 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-token\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.434148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.420032 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-credential-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.434148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.421322 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-workload-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.434148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.421978 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3b2112-be30-40ae-bc4b-84d75341a926-istiod-ca-cert\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.434148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.422217 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-workload-certs\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.434148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.424901 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-data\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.469235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.467289 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-token\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.469235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.468841 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j589r\" (UniqueName: \"kubernetes.io/projected/0c3b2112-be30-40ae-bc4b-84d75341a926-kube-api-access-j589r\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.470258 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.470227 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-envoy\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.473337 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.473311 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-podinfo\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.575752 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.575718 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:16.581340 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.581310 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:16.638290 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.635473 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q"] Nov 02 17:57:16.638290 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:57:16.636698 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb2197f_704a_47aa_8b7b_0094baa86a96.slice/crio-23ed069a88afe0d187c24453bd6701f9289c839ff80b1b40de33e2920e836394 WatchSource:0}: Error finding container 23ed069a88afe0d187c24453bd6701f9289c839ff80b1b40de33e2920e836394: Status 404 returned error can't find the container with id 23ed069a88afe0d187c24453bd6701f9289c839ff80b1b40de33e2920e836394 Nov 02 17:57:16.641914 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.641828 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:16.642031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.641928 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:16.642031 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.641988 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:16.875749 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.875713 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg"] Nov 02 17:57:16.875946 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:57:16.875925 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3b2112_be30_40ae_bc4b_84d75341a926.slice/crio-c08a66c46bfc89ecf4064819779f3392c07938cf156382f2710dfa1ba3893ef5 WatchSource:0}: Error finding container c08a66c46bfc89ecf4064819779f3392c07938cf156382f2710dfa1ba3893ef5: Status 404 returned error can't find the container with id c08a66c46bfc89ecf4064819779f3392c07938cf156382f2710dfa1ba3893ef5 Nov 02 17:57:16.878864 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.878830 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:16.878983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.878893 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:16.878983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:16.878931 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:17.071448 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:17.071364 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:17.071923 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:17.071864 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:17.072782 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:17.072763 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 17:57:17.277992 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:17.268786 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:17.291342 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:17.291301 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" event={"ID":"6eb2197f-704a-47aa-8b7b-0094baa86a96","Type":"ContainerStarted","Data":"50c5136b91fca96d8018cf7779eb32ef3abf4b742bfcc5edb50ff1951d00fea6"} Nov 02 17:57:17.291537 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:17.291370 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" event={"ID":"6eb2197f-704a-47aa-8b7b-0094baa86a96","Type":"ContainerStarted","Data":"23ed069a88afe0d187c24453bd6701f9289c839ff80b1b40de33e2920e836394"} Nov 02 17:57:17.298655 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:17.297407 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" event={"ID":"0c3b2112-be30-40ae-bc4b-84d75341a926","Type":"ContainerStarted","Data":"7875f01f9f135719a6dba7c099fdce17c752085ffa2ee3359c95775fa69edae7"} Nov 02 17:57:17.298655 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:17.297461 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" event={"ID":"0c3b2112-be30-40ae-bc4b-84d75341a926","Type":"ContainerStarted","Data":"c08a66c46bfc89ecf4064819779f3392c07938cf156382f2710dfa1ba3893ef5"} Nov 02 17:57:17.300619 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:17.300593 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg"] Nov 02 17:57:17.300786 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:17.300753 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" podUID="27067902-98d8-4f23-ba47-f5bb2fb86876" containerName="istio-proxy" containerID="cri-o://c42d4c65252f6713bb0cc470fff362f55e00bc17fc60f064603cbbf395c25b8e" gracePeriod=30 Nov 02 17:57:17.362142 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:17.358353 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" podStartSLOduration=2.358332549 podStartE2EDuration="2.358332549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:17.356403316 +0000 UTC m=+1409.144908845" watchObservedRunningTime="2026-11-02 17:57:17.358332549 +0000 UTC m=+1409.146838079" Nov 02 17:57:17.449910 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:17.449851 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" podStartSLOduration=1.449831995 podStartE2EDuration="1.449831995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:17.44707379 +0000 UTC m=+1409.235579317" watchObservedRunningTime="2026-11-02 17:57:17.449831995 +0000 UTC m=+1409.238337571" Nov 02 17:57:17.585982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:17.582306 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:17.586983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:17.586862 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 02 17:57:17.586983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:17.586937 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" podUID="0c3b2112-be30-40ae-bc4b-84d75341a926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 02 17:57:17.620057 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:57:17.619931 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:57:18.259717 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:18.259681 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:18.280600 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:18.280563 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:18.288120 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:18.288079 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 02 17:57:18.288375 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:18.288347 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" podUID="6eb2197f-704a-47aa-8b7b-0094baa86a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 02 17:57:18.587039 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:18.583438 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 02 17:57:18.587039 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:18.583504 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" podUID="0c3b2112-be30-40ae-bc4b-84d75341a926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 02 17:57:19.278593 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:19.278546 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 02 17:57:19.278803 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:19.278621 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" podUID="6eb2197f-704a-47aa-8b7b-0094baa86a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 02 17:57:19.416982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:19.414465 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:19.613098 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:19.609202 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 02 17:57:19.613098 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:19.609264 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" podUID="0c3b2112-be30-40ae-bc4b-84d75341a926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 02 17:57:19.630980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:19.618007 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:19.646474 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:19.646448 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:20.282034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:20.278928 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 02 17:57:20.282034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:20.279005 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" podUID="6eb2197f-704a-47aa-8b7b-0094baa86a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 02 17:57:20.383987 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:20.383664 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:20.533039 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:20.532357 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 17:57:20.613977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:20.601394 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 02 17:57:20.613977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:20.601456 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" podUID="0c3b2112-be30-40ae-bc4b-84d75341a926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 02 17:57:21.279018 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:21.278751 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 02 17:57:21.279018 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:21.278827 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" podUID="6eb2197f-704a-47aa-8b7b-0094baa86a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 02 17:57:21.375978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:21.369879 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:21.593041 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:21.589316 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 02 17:57:21.593041 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:21.589378 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" podUID="0c3b2112-be30-40ae-bc4b-84d75341a926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 02 17:57:22.288977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:22.278918 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 02 17:57:22.288977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:22.279013 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" podUID="6eb2197f-704a-47aa-8b7b-0094baa86a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 02 17:57:22.402974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:22.397507 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:22.542125 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:22.537132 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:57:22.593981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:22.576116 5549 generic.go:356] "Generic (PLEG): container finished" podID="27067902-98d8-4f23-ba47-f5bb2fb86876" containerID="c42d4c65252f6713bb0cc470fff362f55e00bc17fc60f064603cbbf395c25b8e" exitCode=0 Nov 02 17:57:22.593981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:22.576229 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" event={"ID":"27067902-98d8-4f23-ba47-f5bb2fb86876","Type":"ContainerDied","Data":"c42d4c65252f6713bb0cc470fff362f55e00bc17fc60f064603cbbf395c25b8e"} Nov 02 17:57:22.593981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:22.589433 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 02 17:57:22.593981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:22.589506 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" podUID="0c3b2112-be30-40ae-bc4b-84d75341a926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 02 17:57:23.073178 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.066084 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw"] Nov 02 17:57:23.073178 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.072769 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.074148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.074114 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw"] Nov 02 17:57:23.149116 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.149083 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-9165c3-openshift-default-dockercfg-7wchl\"" Nov 02 17:57:23.236308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.213356 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.236308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.213485 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.236308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.213531 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsgb8\" (UniqueName: \"kubernetes.io/projected/e56ccdbd-6737-4a84-9d80-59018a6b77ff-kube-api-access-dsgb8\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.236308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.213692 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.236308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.213730 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.236308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.213773 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.236308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.213819 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.236308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.213929 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.236308 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.213999 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.253779 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.253749 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:57:23.278840 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.278795 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 02 17:57:23.279054 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.278874 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" podUID="6eb2197f-704a-47aa-8b7b-0094baa86a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 02 17:57:23.348028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.347896 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27067902-98d8-4f23-ba47-f5bb2fb86876-kube-api-access-t2hzv\" (UniqueName: \"kubernetes.io/projected/27067902-98d8-4f23-ba47-f5bb2fb86876-kube-api-access-t2hzv\") pod \"27067902-98d8-4f23-ba47-f5bb2fb86876\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " Nov 02 17:57:23.348028 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.348006 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27067902-98d8-4f23-ba47-f5bb2fb86876-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27067902-98d8-4f23-ba47-f5bb2fb86876-istiod-ca-cert\") pod \"27067902-98d8-4f23-ba47-f5bb2fb86876\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " Nov 02 17:57:23.348666 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.348082 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-workload-certs\") pod \"27067902-98d8-4f23-ba47-f5bb2fb86876\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " Nov 02 17:57:23.348666 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.348137 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-workload-socket\") pod \"27067902-98d8-4f23-ba47-f5bb2fb86876\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " Nov 02 17:57:23.348666 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.348171 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-data\") pod \"27067902-98d8-4f23-ba47-f5bb2fb86876\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " Nov 02 17:57:23.348666 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.348303 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-podinfo\") pod \"27067902-98d8-4f23-ba47-f5bb2fb86876\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " Nov 02 17:57:23.348666 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.348376 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-token\" (UniqueName: \"kubernetes.io/projected/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-token\") pod \"27067902-98d8-4f23-ba47-f5bb2fb86876\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " Nov 02 17:57:23.348666 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.348516 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-envoy\") pod \"27067902-98d8-4f23-ba47-f5bb2fb86876\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " Nov 02 17:57:23.348666 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.348571 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-credential-socket\") pod \"27067902-98d8-4f23-ba47-f5bb2fb86876\" (UID: \"27067902-98d8-4f23-ba47-f5bb2fb86876\") " Nov 02 17:57:23.349097 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.349060 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.349260 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.349120 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.349260 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.349169 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.349714 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.349687 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.349799 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.349719 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27067902-98d8-4f23-ba47-f5bb2fb86876-istiod-ca-cert" pod "27067902-98d8-4f23-ba47-f5bb2fb86876" (UID: "27067902-98d8-4f23-ba47-f5bb2fb86876"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:23.349920 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.349897 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.350027 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.350001 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.350134 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.350115 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.350333 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.350310 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-credential-socket" pod "27067902-98d8-4f23-ba47-f5bb2fb86876" (UID: "27067902-98d8-4f23-ba47-f5bb2fb86876"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.350700 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.350678 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.351155 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.351130 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-data" pod "27067902-98d8-4f23-ba47-f5bb2fb86876" (UID: "27067902-98d8-4f23-ba47-f5bb2fb86876"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.351360 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.351337 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-workload-certs" pod "27067902-98d8-4f23-ba47-f5bb2fb86876" (UID: "27067902-98d8-4f23-ba47-f5bb2fb86876"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.351360 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.350118 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.351500 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.351469 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.351588 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.351566 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-workload-socket" pod "27067902-98d8-4f23-ba47-f5bb2fb86876" (UID: "27067902-98d8-4f23-ba47-f5bb2fb86876"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.351756 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.351731 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.351837 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.351800 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsgb8\" (UniqueName: \"kubernetes.io/projected/e56ccdbd-6737-4a84-9d80-59018a6b77ff-kube-api-access-dsgb8\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.351937 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.351912 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.352443 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.351922 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.352613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.352597 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27067902-98d8-4f23-ba47-f5bb2fb86876-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.352744 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.352730 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.352856 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.352842 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.352988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.352973 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.353097 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.352358 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.354221 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.354196 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-token" pod "27067902-98d8-4f23-ba47-f5bb2fb86876" (UID: "27067902-98d8-4f23-ba47-f5bb2fb86876"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:23.355976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.355911 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27067902-98d8-4f23-ba47-f5bb2fb86876-kube-api-access-t2hzv" pod "27067902-98d8-4f23-ba47-f5bb2fb86876" (UID: "27067902-98d8-4f23-ba47-f5bb2fb86876"). InnerVolumeSpecName "kube-api-access-t2hzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:23.356173 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.356130 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-envoy" pod "27067902-98d8-4f23-ba47-f5bb2fb86876" (UID: "27067902-98d8-4f23-ba47-f5bb2fb86876"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.356524 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.356495 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.358507 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.358300 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-podinfo" pod "27067902-98d8-4f23-ba47-f5bb2fb86876" (UID: "27067902-98d8-4f23-ba47-f5bb2fb86876"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:57:23.358507 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.358490 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.368984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.368938 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsgb8\" (UniqueName: \"kubernetes.io/projected/e56ccdbd-6737-4a84-9d80-59018a6b77ff-kube-api-access-dsgb8\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.369282 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.369258 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.454843 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.454801 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.454843 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.454844 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.455085 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.454861 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27067902-98d8-4f23-ba47-f5bb2fb86876-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.455085 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.454878 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2hzv\" (UniqueName: \"kubernetes.io/projected/27067902-98d8-4f23-ba47-f5bb2fb86876-kube-api-access-t2hzv\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.490992 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.482483 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:23.587997 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:23.585622 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:24.064088 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.012645 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5"] Nov 02 17:57:24.064088 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.013977 5549 cpu_manager.go:414] "containerMap: removing container" podUID="27067902-98d8-4f23-ba47-f5bb2fb86876" containerName="istio-proxy" Nov 02 17:57:24.064088 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.013998 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27067902-98d8-4f23-ba47-f5bb2fb86876" containerName="istio-proxy" Nov 02 17:57:24.064088 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.014228 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="27067902-98d8-4f23-ba47-f5bb2fb86876" containerName="istio-proxy" Nov 02 17:57:24.177603 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.177567 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.257218 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.257187 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:24.533336 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.531625 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.533336 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.531686 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.533336 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.531720 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.533336 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.531747 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.533336 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.531771 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.533336 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.531888 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6425q\" (UniqueName: \"kubernetes.io/projected/8664bcf8-87ef-4033-a3f8-22db481e826c-kube-api-access-6425q\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.533336 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.531919 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8664bcf8-87ef-4033-a3f8-22db481e826c-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.533336 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.531995 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.533336 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.532056 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.580054 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.542027 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" Nov 02 17:57:24.625106 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.596141 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:24.625106 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.624529 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg" event={"ID":"27067902-98d8-4f23-ba47-f5bb2fb86876","Type":"ContainerDied","Data":"3d9262e70cb1e4b6fe05137bf385823754eaf8b36c664c59c2c0d467ae36fa1f"} Nov 02 17:57:24.625106 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.624588 5549 scope.go:122] "RemoveContainer" containerID="c42d4c65252f6713bb0cc470fff362f55e00bc17fc60f064603cbbf395c25b8e" Nov 02 17:57:24.627735 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.627707 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:24.627892 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.627802 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 17:57:24.627892 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.627838 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5"] Nov 02 17:57:24.637076 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.629083 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:24.637361 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.636576 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.638416 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.638390 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.638585 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.638568 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.638705 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.638687 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.638988 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.638967 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6425q\" (UniqueName: \"kubernetes.io/projected/8664bcf8-87ef-4033-a3f8-22db481e826c-kube-api-access-6425q\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.639142 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.639123 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8664bcf8-87ef-4033-a3f8-22db481e826c-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.639293 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.639277 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.639457 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.639443 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.639859 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.639842 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.644278 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.637036 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.644980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.644882 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.645710 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.645680 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.649329 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.648872 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.651712 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.651688 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8664bcf8-87ef-4033-a3f8-22db481e826c-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.652151 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.652116 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.677841 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.676279 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 17:57:24.693700 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.693610 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.709211 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.697773 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6425q\" (UniqueName: \"kubernetes.io/projected/8664bcf8-87ef-4033-a3f8-22db481e826c-kube-api-access-6425q\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.724847 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:24.722617 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:24.787627 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:57:24.787557 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27067902_98d8_4f23_ba47_f5bb2fb86876.slice/crio-3d9262e70cb1e4b6fe05137bf385823754eaf8b36c664c59c2c0d467ae36fa1f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27067902_98d8_4f23_ba47_f5bb2fb86876.slice\": RecentStats: unable to find data in memory cache]" Nov 02 17:57:24.845734 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:57:24.845689 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27067902_98d8_4f23_ba47_f5bb2fb86876.slice\": RecentStats: unable to find data in memory cache]" Nov 02 17:57:25.076348 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.076319 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:25.159443 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.159401 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg"] Nov 02 17:57:25.187387 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.187354 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-fj7hg"] Nov 02 17:57:25.303990 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.303939 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw"] Nov 02 17:57:25.311366 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.310045 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:25.311591 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.311561 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:25.311750 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.311727 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:25.470114 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:57:25.470085 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8664bcf8_87ef_4033_a3f8_22db481e826c.slice/crio-9675e3690c9bb27d32fffb5f2e1747dd63a75bf592b8de00075b398226040378 WatchSource:0}: Error finding container 9675e3690c9bb27d32fffb5f2e1747dd63a75bf592b8de00075b398226040378: Status 404 returned error can't find the container with id 9675e3690c9bb27d32fffb5f2e1747dd63a75bf592b8de00075b398226040378 Nov 02 17:57:25.470627 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.470603 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5"] Nov 02 17:57:25.473235 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.473196 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:25.473347 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.473282 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:25.473347 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.473325 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:25.554841 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.554805 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" event={"ID":"e56ccdbd-6737-4a84-9d80-59018a6b77ff","Type":"ContainerStarted","Data":"56a3e72ae6988fda9241dbdff978cbce31c14d0f3d38c950c58fb642dfb66e29"} Nov 02 17:57:25.555478 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.555422 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" event={"ID":"e56ccdbd-6737-4a84-9d80-59018a6b77ff","Type":"ContainerStarted","Data":"c316867731343119c1f44bbe0539bee32e9e9f2845f36cf228a935b85a20304d"} Nov 02 17:57:25.557750 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.556961 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" event={"ID":"8664bcf8-87ef-4033-a3f8-22db481e826c","Type":"ContainerStarted","Data":"954283cc3e3e214760277e957f41171668dd0aa0fb98a7b8de87c79b5d15c860"} Nov 02 17:57:25.557750 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.557013 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" event={"ID":"8664bcf8-87ef-4033-a3f8-22db481e826c","Type":"ContainerStarted","Data":"9675e3690c9bb27d32fffb5f2e1747dd63a75bf592b8de00075b398226040378"} Nov 02 17:57:25.575092 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.574989 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:25.582470 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.582365 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" podStartSLOduration=3.582347827 podStartE2EDuration="3.582347827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:25.579020184 +0000 UTC m=+1417.367525719" watchObservedRunningTime="2026-11-02 17:57:25.582347827 +0000 UTC m=+1417.370853358" Nov 02 17:57:25.588317 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.588290 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:25.604576 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:25.604524 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" podStartSLOduration=2.604510597 podStartE2EDuration="2.604510597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:25.601196946 +0000 UTC m=+1417.389702478" watchObservedRunningTime="2026-11-02 17:57:25.604510597 +0000 UTC m=+1417.393016168" Nov 02 17:57:26.077039 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.076999 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:26.078767 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.078741 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Nov 02 17:57:26.078947 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.078787 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" podUID="8664bcf8-87ef-4033-a3f8-22db481e826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Nov 02 17:57:26.350846 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.350761 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27067902-98d8-4f23-ba47-f5bb2fb86876" path="/var/lib/kubelet/pods/27067902-98d8-4f23-ba47-f5bb2fb86876/volumes" Nov 02 17:57:26.351232 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.351211 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7"] Nov 02 17:57:26.356265 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.356244 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7"] Nov 02 17:57:26.356388 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.356340 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.362244 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.362217 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-1e2281\"/\"kube-root-ca.crt\"" Nov 02 17:57:26.367112 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.366961 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-1e2281\"/\"openshift-service-ca.crt\"" Nov 02 17:57:26.410898 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.410850 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-1e2281\"/\"default-dockercfg-sd4jw\"" Nov 02 17:57:26.411319 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.411285 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-1e2281\"/\"auth-test-model-1e2281-kserve-self-signed-certs\"" Nov 02 17:57:26.450585 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.450554 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-model-cache\") pod \"auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.450779 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.450594 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-home\") pod \"auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.450779 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.450717 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eecd66e-2f57-4935-8eb4-000f22375a2a-tls-certs\") pod \"auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.450915 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.450796 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-tmp-dir\") pod \"auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.450915 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.450836 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-dshm\") pod \"auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.451023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.450987 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld97v\" (UniqueName: \"kubernetes.io/projected/8eecd66e-2f57-4935-8eb4-000f22375a2a-kube-api-access-ld97v\") pod \"auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.499983 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.497425 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:26.503050 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.503015 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:26.519116 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.519071 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 02 17:57:26.519389 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.519354 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" podUID="e56ccdbd-6737-4a84-9d80-59018a6b77ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 02 17:57:26.553801 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.553754 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld97v\" (UniqueName: \"kubernetes.io/projected/8eecd66e-2f57-4935-8eb4-000f22375a2a-kube-api-access-ld97v\") pod \"auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.555148 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.555100 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-model-cache\") pod \"auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.556314 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.556288 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-home\") pod \"auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.569866 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.556218 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-model-cache\") pod \"auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.573763 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.556838 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-home\") pod \"auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.574215 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.574188 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eecd66e-2f57-4935-8eb4-000f22375a2a-tls-certs\") pod \"auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.575896 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.575868 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-tmp-dir\") pod \"auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.583682 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.583651 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-dshm\") pod \"auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.584854 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.583548 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-tmp-dir\") pod \"auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.684773 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.682657 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:26.743253 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.743209 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld97v\" (UniqueName: \"kubernetes.io/projected/8eecd66e-2f57-4935-8eb4-000f22375a2a-kube-api-access-ld97v\") pod \"auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.757789 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.754668 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eecd66e-2f57-4935-8eb4-000f22375a2a-tls-certs\") pod \"auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.765827 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.758276 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-dshm\") pod \"auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:26.977977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:26.974027 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:27.061353 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:27.061325 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 17:57:27.077273 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:27.077243 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Nov 02 17:57:27.077500 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:27.077466 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" podUID="8664bcf8-87ef-4033-a3f8-22db481e826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Nov 02 17:57:27.501677 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:27.501217 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 02 17:57:27.501677 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:27.501277 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" podUID="e56ccdbd-6737-4a84-9d80-59018a6b77ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 02 17:57:27.670346 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:27.670297 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:27.967780 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:27.958382 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7"] Nov 02 17:57:28.126341 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:28.079119 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Nov 02 17:57:28.126341 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:28.079175 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" podUID="8664bcf8-87ef-4033-a3f8-22db481e826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Nov 02 17:57:28.359792 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:28.358429 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:28.489471 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:28.489377 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 02 17:57:28.513589 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:28.513540 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" podUID="e56ccdbd-6737-4a84-9d80-59018a6b77ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 02 17:57:28.757236 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:28.757148 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" event={"ID":"8eecd66e-2f57-4935-8eb4-000f22375a2a","Type":"ContainerStarted","Data":"479400edc13e2eaa4b309a5c6e457808f7ca76efa9c09e5142d8a313598cac1f"} Nov 02 17:57:28.757236 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:28.757209 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" event={"ID":"8eecd66e-2f57-4935-8eb4-000f22375a2a","Type":"ContainerStarted","Data":"d5c535fffc720ce87a9a4da2fe9bd9784d37b5c4ce70ded983e325593e7c0fb3"} Nov 02 17:57:28.809978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:28.805283 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" podStartSLOduration=2.805259339 podStartE2EDuration="2.805259339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:28.777417262 +0000 UTC m=+1420.565922790" watchObservedRunningTime="2026-11-02 17:57:28.805259339 +0000 UTC m=+1420.593764868" Nov 02 17:57:29.055034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:29.054463 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:29.131195 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:29.102847 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Nov 02 17:57:29.131195 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:29.102931 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" podUID="8664bcf8-87ef-4033-a3f8-22db481e826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Nov 02 17:57:29.502478 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:29.502436 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 02 17:57:29.502753 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:29.502718 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" podUID="e56ccdbd-6737-4a84-9d80-59018a6b77ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 02 17:57:30.077389 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:30.077350 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Nov 02 17:57:30.077863 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:30.077419 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" podUID="8664bcf8-87ef-4033-a3f8-22db481e826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Nov 02 17:57:30.500423 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:30.500376 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 02 17:57:30.500751 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:30.500715 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" podUID="e56ccdbd-6737-4a84-9d80-59018a6b77ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 02 17:57:30.561003 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:30.557561 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:31.077667 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:31.077635 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Nov 02 17:57:31.078151 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:31.078125 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" podUID="8664bcf8-87ef-4033-a3f8-22db481e826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Nov 02 17:57:31.498976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:31.488385 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 02 17:57:31.498976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:31.488464 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" podUID="e56ccdbd-6737-4a84-9d80-59018a6b77ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 02 17:57:32.005024 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:32.002613 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:32.078040 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:32.077933 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Nov 02 17:57:32.078501 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:32.078067 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" podUID="8664bcf8-87ef-4033-a3f8-22db481e826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Nov 02 17:57:32.490387 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:32.490356 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:32.490830 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:32.490808 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:32.491626 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:32.491606 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 17:57:33.082136 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:33.082106 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:33.082596 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:33.082580 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:33.083499 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:33.083478 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 17:57:33.625670 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:33.625634 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:34.837546 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:34.837505 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:35.433456 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:35.433424 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:36.272976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:36.272052 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:36.976920 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:36.976889 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:36.990004 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:36.989981 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:37.055029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:37.050022 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:37.862975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:37.861853 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:37.955520 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:37.955492 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 17:57:38.660647 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:38.660607 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:39.524388 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:39.524348 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:40.285978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:40.278621 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:41.190976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:41.172731 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:42.079979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:42.079648 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:43.068893 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:43.068854 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:43.908012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:43.906457 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:44.100316 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:44.100277 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:45.100704 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:45.100666 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:45.940068 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:45.939932 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:46.846976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:46.845261 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:47.562789 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.562756 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:47.581271 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.581232 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh"] Nov 02 17:57:47.587521 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.587499 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:47.598001 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:57:47.597485 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:57:47.599556 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.599532 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-2961d5\"/\"auth-test-model-2961d5-kserve-self-signed-certs\"" Nov 02 17:57:47.600642 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.600621 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-2961d5\"/\"kube-root-ca.crt\"" Nov 02 17:57:47.600823 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.600700 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-2961d5\"/\"openshift-service-ca.crt\"" Nov 02 17:57:47.600823 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.600774 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-2961d5\"/\"default-dockercfg-g97kr\"" Nov 02 17:57:47.602862 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.602680 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh"] Nov 02 17:57:47.731327 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.731286 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-dshm\") pod \"auth-test-model-2961d5-kserve-9bb54dc6f-b89lh\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:47.731327 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.731335 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-model-cache\") pod \"auth-test-model-2961d5-kserve-9bb54dc6f-b89lh\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:47.731607 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.731589 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-tls-certs\") pod \"auth-test-model-2961d5-kserve-9bb54dc6f-b89lh\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:47.731708 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.731687 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tclm9\" (UniqueName: \"kubernetes.io/projected/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-kube-api-access-tclm9\") pod \"auth-test-model-2961d5-kserve-9bb54dc6f-b89lh\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:47.731768 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.731719 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-tmp-dir\") pod \"auth-test-model-2961d5-kserve-9bb54dc6f-b89lh\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:47.731823 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.731787 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-home\") pod \"auth-test-model-2961d5-kserve-9bb54dc6f-b89lh\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:47.733407 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.733381 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:47.841029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.840282 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-dshm\") pod \"auth-test-model-2961d5-kserve-9bb54dc6f-b89lh\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:47.841029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.840333 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-model-cache\") pod \"auth-test-model-2961d5-kserve-9bb54dc6f-b89lh\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:47.841029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.840474 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-tls-certs\") pod \"auth-test-model-2961d5-kserve-9bb54dc6f-b89lh\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:47.841029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.840523 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tclm9\" (UniqueName: \"kubernetes.io/projected/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-kube-api-access-tclm9\") pod \"auth-test-model-2961d5-kserve-9bb54dc6f-b89lh\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:47.841029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.840549 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-tmp-dir\") pod \"auth-test-model-2961d5-kserve-9bb54dc6f-b89lh\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:47.841029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.840608 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-home\") pod \"auth-test-model-2961d5-kserve-9bb54dc6f-b89lh\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:47.841478 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.841124 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-home\") pod \"auth-test-model-2961d5-kserve-9bb54dc6f-b89lh\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:47.844977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.842061 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-model-cache\") pod \"auth-test-model-2961d5-kserve-9bb54dc6f-b89lh\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:47.844977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.842241 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-tmp-dir\") pod \"auth-test-model-2961d5-kserve-9bb54dc6f-b89lh\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:47.949014 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.945184 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-dshm\") pod \"auth-test-model-2961d5-kserve-9bb54dc6f-b89lh\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:47.949014 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.945743 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-tls-certs\") pod \"auth-test-model-2961d5-kserve-9bb54dc6f-b89lh\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:47.952971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.951584 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:47.952971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:47.952297 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tclm9\" (UniqueName: \"kubernetes.io/projected/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-kube-api-access-tclm9\") pod \"auth-test-model-2961d5-kserve-9bb54dc6f-b89lh\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:48.213929 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:48.213890 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:48.509416 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:57:48.509385 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a75b540_c8a0_4d10_ac48_5e645c73d0e4.slice/crio-53a6131f6a3ca04b9fb07bac29e3e223105f726c474005ac8010f101e65c5a50 WatchSource:0}: Error finding container 53a6131f6a3ca04b9fb07bac29e3e223105f726c474005ac8010f101e65c5a50: Status 404 returned error can't find the container with id 53a6131f6a3ca04b9fb07bac29e3e223105f726c474005ac8010f101e65c5a50 Nov 02 17:57:48.513496 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:48.513469 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh"] Nov 02 17:57:48.850021 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:48.849916 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:49.054241 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:49.054205 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" event={"ID":"3a75b540-c8a0-4d10-ac48-5e645c73d0e4","Type":"ContainerStarted","Data":"f5d954c0f6f5a55b071fcd85ef66e7bb59b948a750ba39555ac4ae9d81a0f1c7"} Nov 02 17:57:49.054241 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:49.054254 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" event={"ID":"3a75b540-c8a0-4d10-ac48-5e645c73d0e4","Type":"ContainerStarted","Data":"53a6131f6a3ca04b9fb07bac29e3e223105f726c474005ac8010f101e65c5a50"} Nov 02 17:57:49.077933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:49.077885 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" podStartSLOduration=2.077870714 podStartE2EDuration="2.077870714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:49.074536147 +0000 UTC m=+1440.863041675" watchObservedRunningTime="2026-11-02 17:57:49.077870714 +0000 UTC m=+1440.866376297" Nov 02 17:57:49.220975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:49.217097 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:51.616331 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:51.616301 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:52.137978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:52.133906 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:53.079973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:53.073811 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:53.647251 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:53.647216 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:53.809381 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:53.809343 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:54.701181 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:54.701146 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:55.535976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:55.529573 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:56.346973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:56.337113 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:57.051973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:57.051561 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:57.829647 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:57.829599 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:58.214869 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:58.214832 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:58.227361 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:58.227336 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:58.608032 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:58.607805 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:59.007008 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:59.006938 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:59.167480 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:59.167448 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 17:57:59.294883 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:59.294805 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:59.619750 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:59.619656 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:59.811999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:57:59.807402 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:00.565041 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:00.563927 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:01.042896 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.042856 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg"] Nov 02 17:58:01.664025 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.608293 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg"] Nov 02 17:58:01.664686 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.664656 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9"] Nov 02 17:58:01.845982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.647563 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.845982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.797298 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435cec96-4072-46ce-852a-d1faec26752f-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.845982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.835698 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.845982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.835736 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.845982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.836026 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.845982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.836093 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435cec96-4072-46ce-852a-d1faec26752f-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.845982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.836330 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.845982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.836427 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435cec96-4072-46ce-852a-d1faec26752f-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.845982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.836461 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.845982 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.836515 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptrq4\" (UniqueName: \"kubernetes.io/projected/435cec96-4072-46ce-852a-d1faec26752f-kube-api-access-ptrq4\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.888517 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.888438 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9"] Nov 02 17:58:01.888859 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.888841 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:01.959974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.949841 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:01.959974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.953790 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.959974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.953875 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435cec96-4072-46ce-852a-d1faec26752f-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.959974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.953912 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.959974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.953974 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptrq4\" (UniqueName: \"kubernetes.io/projected/435cec96-4072-46ce-852a-d1faec26752f-kube-api-access-ptrq4\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.959974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.954072 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435cec96-4072-46ce-852a-d1faec26752f-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.959974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.954104 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.959974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.954131 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.959974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.954319 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.959974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.954370 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435cec96-4072-46ce-852a-d1faec26752f-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.959974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.955223 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.981160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.974084 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435cec96-4072-46ce-852a-d1faec26752f-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.981160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.974422 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.981160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.976338 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:01.981160 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:01.976562 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:02.051108 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.027788 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptrq4\" (UniqueName: \"kubernetes.io/projected/435cec96-4072-46ce-852a-d1faec26752f-kube-api-access-ptrq4\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:02.051108 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.039284 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:02.057334 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.057217 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54201446-6418-4b65-b291-9263a4781e30-istio-token\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.057813 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.057782 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54201446-6418-4b65-b291-9263a4781e30-istio-podinfo\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.058131 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.058112 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-workload-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.074046 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.058408 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-istio-envoy\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.074374 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.074343 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-istio-data\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.074527 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.074508 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t5cp\" (UniqueName: \"kubernetes.io/projected/54201446-6418-4b65-b291-9263a4781e30-kube-api-access-6t5cp\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.074641 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.074628 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54201446-6418-4b65-b291-9263a4781e30-istiod-ca-cert\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.074781 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.074760 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-credential-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.074984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.074945 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-workload-certs\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.094703 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.094666 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435cec96-4072-46ce-852a-d1faec26752f-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:02.181154 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.181103 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-istio-data\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.181431 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.181409 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t5cp\" (UniqueName: \"kubernetes.io/projected/54201446-6418-4b65-b291-9263a4781e30-kube-api-access-6t5cp\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.181579 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.181564 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54201446-6418-4b65-b291-9263a4781e30-istiod-ca-cert\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.181723 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.181708 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-credential-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.181900 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.181885 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-workload-certs\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.196661 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.195584 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54201446-6418-4b65-b291-9263a4781e30-istiod-ca-cert\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.196661 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.195191 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-credential-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.196661 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.195629 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54201446-6418-4b65-b291-9263a4781e30-istio-token\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.196661 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.195504 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-workload-certs\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.196661 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.195820 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54201446-6418-4b65-b291-9263a4781e30-istio-podinfo\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.196661 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.194535 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-istio-data\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.196661 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.196011 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-workload-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.196661 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.196243 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-istio-envoy\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.197642 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.197594 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-workload-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.208083 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.208049 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435cec96-4072-46ce-852a-d1faec26752f-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:02.249383 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.240737 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54201446-6418-4b65-b291-9263a4781e30-istio-token\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.249383 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.241671 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:02.298594 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.266189 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-istio-envoy\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.298594 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.268987 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54201446-6418-4b65-b291-9263a4781e30-istio-podinfo\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.298594 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.283827 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t5cp\" (UniqueName: \"kubernetes.io/projected/54201446-6418-4b65-b291-9263a4781e30-kube-api-access-6t5cp\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.377999 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.377352 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:02.559757 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.559723 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:02.741097 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.740809 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:02.955785 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.955758 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg"] Nov 02 17:58:02.957249 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:58:02.957202 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435cec96_4072_46ce_852a_d1faec26752f.slice/crio-fcc55ad598878c680d557bf17c2347ad2ffa9cc9d2984f7550ad209f911a323c WatchSource:0}: Error finding container fcc55ad598878c680d557bf17c2347ad2ffa9cc9d2984f7550ad209f911a323c: Status 404 returned error can't find the container with id fcc55ad598878c680d557bf17c2347ad2ffa9cc9d2984f7550ad209f911a323c Nov 02 17:58:02.961770 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:58:02.961742 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54201446_6418_4b65_b291_9263a4781e30.slice/crio-accf20606a49512d0b028d5eabeca19ce3683e48ddd4bc3a0039ddc7cef5f056 WatchSource:0}: Error finding container accf20606a49512d0b028d5eabeca19ce3683e48ddd4bc3a0039ddc7cef5f056: Status 404 returned error can't find the container with id accf20606a49512d0b028d5eabeca19ce3683e48ddd4bc3a0039ddc7cef5f056 Nov 02 17:58:02.961860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.961827 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:02.961910 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.961902 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:02.962039 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.961933 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:02.962215 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.962198 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9"] Nov 02 17:58:02.964588 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.964559 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:02.964686 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.964620 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:02.964686 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:02.964649 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:03.090975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:03.064341 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:03.840560 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:03.840521 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" event={"ID":"54201446-6418-4b65-b291-9263a4781e30","Type":"ContainerStarted","Data":"4ac12647202ab8c7c8f309aee5b5de594ec37e69391e30ee210cd434b02936f0"} Nov 02 17:58:03.840938 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:03.840625 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" event={"ID":"54201446-6418-4b65-b291-9263a4781e30","Type":"ContainerStarted","Data":"accf20606a49512d0b028d5eabeca19ce3683e48ddd4bc3a0039ddc7cef5f056"} Nov 02 17:58:03.842847 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:03.842812 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" event={"ID":"435cec96-4072-46ce-852a-d1faec26752f","Type":"ContainerStarted","Data":"4494a059d70a7d4df5e9cc4563e9a6c11f64db3331306cc396a4a56c0650dada"} Nov 02 17:58:03.842975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:03.842865 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" event={"ID":"435cec96-4072-46ce-852a-d1faec26752f","Type":"ContainerStarted","Data":"fcc55ad598878c680d557bf17c2347ad2ffa9cc9d2984f7550ad209f911a323c"} Nov 02 17:58:03.870705 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:03.870654 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" podStartSLOduration=2.87063895 podStartE2EDuration="2.87063895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:03.867532207 +0000 UTC m=+1455.656037776" watchObservedRunningTime="2026-11-02 17:58:03.87063895 +0000 UTC m=+1455.659144477" Nov 02 17:58:03.931976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:03.920477 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" podStartSLOduration=2.920456356 podStartE2EDuration="2.920456356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:03.89925752 +0000 UTC m=+1455.687763060" watchObservedRunningTime="2026-11-02 17:58:03.920456356 +0000 UTC m=+1455.708961885" Nov 02 17:58:04.250446 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:04.250310 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:04.251479 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:04.251454 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 17:58:04.251597 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:04.251505 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" podUID="435cec96-4072-46ce-852a-d1faec26752f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 17:58:04.387996 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:04.387612 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:04.399976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:04.398745 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 02 17:58:04.399976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:04.398816 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" podUID="54201446-6418-4b65-b291-9263a4781e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 02 17:58:04.410979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:04.401781 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:05.250167 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:05.250129 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 17:58:05.250576 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:05.250189 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" podUID="435cec96-4072-46ce-852a-d1faec26752f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 17:58:05.378285 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:05.378252 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 02 17:58:05.378467 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:05.378303 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" podUID="54201446-6418-4b65-b291-9263a4781e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 02 17:58:05.772017 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:05.771944 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:06.250792 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:06.250762 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 17:58:06.251220 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:06.250810 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" podUID="435cec96-4072-46ce-852a-d1faec26752f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 17:58:06.378650 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:06.378610 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 02 17:58:06.378816 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:06.378670 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" podUID="54201446-6418-4b65-b291-9263a4781e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 02 17:58:06.999977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:06.999751 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:07.266602 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:07.265911 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 17:58:07.266602 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:07.265983 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" podUID="435cec96-4072-46ce-852a-d1faec26752f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 17:58:07.309976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:07.307324 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:07.386986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:07.385370 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 02 17:58:07.386986 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:07.385502 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" podUID="54201446-6418-4b65-b291-9263a4781e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 02 17:58:07.749967 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:07.749913 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:08.112029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.111356 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v"] Nov 02 17:58:08.139975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.135600 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v"] Nov 02 17:58:08.139975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.135732 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.253821 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.252338 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 17:58:08.253821 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.252413 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" podUID="435cec96-4072-46ce-852a-d1faec26752f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 17:58:08.256462 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.256423 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfr9w\" (UniqueName: \"kubernetes.io/projected/47851f7c-7870-4a63-bb42-6681e40bf144-kube-api-access-cfr9w\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.256589 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.256481 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.256589 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.256521 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.256589 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.256552 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.256694 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.256630 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.256694 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.256670 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47851f7c-7870-4a63-bb42-6681e40bf144-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.256897 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.256880 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47851f7c-7870-4a63-bb42-6681e40bf144-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.257000 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.256984 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.257043 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.257017 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47851f7c-7870-4a63-bb42-6681e40bf144-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.358055 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.358014 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47851f7c-7870-4a63-bb42-6681e40bf144-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.358486 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.358104 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.358486 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.358132 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47851f7c-7870-4a63-bb42-6681e40bf144-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.358486 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.358200 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfr9w\" (UniqueName: \"kubernetes.io/projected/47851f7c-7870-4a63-bb42-6681e40bf144-kube-api-access-cfr9w\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.358486 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.358230 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.358486 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.358268 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.358486 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.358299 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.358486 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.358381 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.358486 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.358426 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47851f7c-7870-4a63-bb42-6681e40bf144-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.358989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.358708 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.358989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.358764 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.358989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.358912 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47851f7c-7870-4a63-bb42-6681e40bf144-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.358989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.358932 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.359171 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.359044 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.360477 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.360458 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.360825 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.360807 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47851f7c-7870-4a63-bb42-6681e40bf144-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.371095 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.371044 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfr9w\" (UniqueName: \"kubernetes.io/projected/47851f7c-7870-4a63-bb42-6681e40bf144-kube-api-access-cfr9w\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.400980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.394568 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 02 17:58:08.400980 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.394627 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" podUID="54201446-6418-4b65-b291-9263a4781e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 02 17:58:08.404979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.401463 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47851f7c-7870-4a63-bb42-6681e40bf144-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.478490 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.478456 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:08.865995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:08.864835 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:09.025103 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:09.023161 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v"] Nov 02 17:58:09.048753 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:09.048682 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:09.048989 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:09.048967 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:09.049138 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:09.049121 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:09.252534 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:09.250936 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 17:58:09.252534 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:09.251041 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" podUID="435cec96-4072-46ce-852a-d1faec26752f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 17:58:09.378475 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:09.378431 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 02 17:58:09.378830 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:09.378492 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" podUID="54201446-6418-4b65-b291-9263a4781e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 02 17:58:10.031529 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:10.021715 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:10.044973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:10.041091 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" event={"ID":"47851f7c-7870-4a63-bb42-6681e40bf144","Type":"ContainerStarted","Data":"82e08c1c0d8c5f64f2992ed635b56b2855a7f41e759d919ddf67974580a3ba54"} Nov 02 17:58:10.044973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:10.041161 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" event={"ID":"47851f7c-7870-4a63-bb42-6681e40bf144","Type":"ContainerStarted","Data":"149242ff41fd6f1a0106b5fbe1ce4ba90c64803c5e91c35431b03dadc6ccc5f9"} Nov 02 17:58:10.255408 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:10.255375 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:10.255710 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:10.255686 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:10.256482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:10.256459 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 17:58:10.308624 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:10.308496 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" podStartSLOduration=2.308477125 podStartE2EDuration="2.308477125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:10.086389102 +0000 UTC m=+1461.874894642" watchObservedRunningTime="2026-11-02 17:58:10.308477125 +0000 UTC m=+1462.096982657" Nov 02 17:58:10.405033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:10.396348 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:10.405033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:10.398119 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:10.405033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:10.404234 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 17:58:10.478823 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:10.478791 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:10.499773 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:10.499736 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 02 17:58:10.500124 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:10.500090 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" podUID="47851f7c-7870-4a63-bb42-6681e40bf144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 02 17:58:11.249732 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:11.249690 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:11.504034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:11.501367 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 02 17:58:11.504034 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:11.501432 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" podUID="47851f7c-7870-4a63-bb42-6681e40bf144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 02 17:58:12.221977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:12.213480 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:12.485066 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:12.483338 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 02 17:58:12.485066 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:12.483397 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" podUID="47851f7c-7870-4a63-bb42-6681e40bf144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 02 17:58:12.767518 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:12.767432 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:13.137941 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:13.137844 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:13.487978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:13.483273 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 02 17:58:13.487978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:13.483337 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" podUID="47851f7c-7870-4a63-bb42-6681e40bf144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 02 17:58:14.151440 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:14.147542 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:14.478825 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:14.478792 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 02 17:58:14.479073 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:14.478846 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" podUID="47851f7c-7870-4a63-bb42-6681e40bf144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 02 17:58:15.023224 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:15.023190 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:15.478821 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:15.478788 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 02 17:58:15.479352 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:15.478853 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" podUID="47851f7c-7870-4a63-bb42-6681e40bf144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 02 17:58:15.956793 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:15.956753 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:16.484271 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:16.484236 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:16.484734 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:16.484702 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:16.485328 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:16.485309 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 17:58:16.854732 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:16.854645 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:17.588263 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:58:17.588218 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:58:17.791296 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:17.787667 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:18.740042 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:18.733852 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:19.427241 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:19.427200 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:20.052012 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:20.048452 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:20.857778 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:20.857735 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:21.616801 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:21.616764 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:22.295973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:22.288186 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:23.036156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.035916 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:23.128067 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.128024 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj"] Nov 02 17:58:23.134233 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.134206 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.142802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.142775 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj"] Nov 02 17:58:23.237339 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.237294 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852a217c-3306-4428-a34c-0414d04eaba4-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.237553 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.237380 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852a217c-3306-4428-a34c-0414d04eaba4-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.237553 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.237492 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.237697 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.237598 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgwl9\" (UniqueName: \"kubernetes.io/projected/852a217c-3306-4428-a34c-0414d04eaba4-kube-api-access-cgwl9\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.237697 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.237639 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.237821 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.237722 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.237821 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.237764 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.237933 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.237879 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852a217c-3306-4428-a34c-0414d04eaba4-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.238051 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.237937 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.339069 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.338942 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgwl9\" (UniqueName: \"kubernetes.io/projected/852a217c-3306-4428-a34c-0414d04eaba4-kube-api-access-cgwl9\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.339069 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.339042 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.339337 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.339160 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.339337 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.339188 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.339337 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.339250 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852a217c-3306-4428-a34c-0414d04eaba4-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.339337 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.339302 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.339557 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.339370 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852a217c-3306-4428-a34c-0414d04eaba4-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.339557 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.339441 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852a217c-3306-4428-a34c-0414d04eaba4-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.339557 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.339477 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.339723 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.339642 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.339788 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.339739 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.339941 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.339922 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.340039 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.340014 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.340130 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.340104 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852a217c-3306-4428-a34c-0414d04eaba4-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.341560 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.341534 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.341845 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.341827 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852a217c-3306-4428-a34c-0414d04eaba4-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.349001 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.348980 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgwl9\" (UniqueName: \"kubernetes.io/projected/852a217c-3306-4428-a34c-0414d04eaba4-kube-api-access-cgwl9\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.349100 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.349081 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852a217c-3306-4428-a34c-0414d04eaba4-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.458498 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.458463 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:23.748451 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:58:23.748412 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852a217c_3306_4428_a34c_0414d04eaba4.slice/crio-84bbdb2c07230d8f585e7923e05ca0c823bf0f6f485ba2267c701b8e39ccebd4 WatchSource:0}: Error finding container 84bbdb2c07230d8f585e7923e05ca0c823bf0f6f485ba2267c701b8e39ccebd4: Status 404 returned error can't find the container with id 84bbdb2c07230d8f585e7923e05ca0c823bf0f6f485ba2267c701b8e39ccebd4 Nov 02 17:58:23.749053 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.748897 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj"] Nov 02 17:58:23.751127 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.751092 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:23.751243 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.751170 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:23.751243 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.751217 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:23.857061 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:23.853803 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:24.208801 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:24.208760 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" event={"ID":"852a217c-3306-4428-a34c-0414d04eaba4","Type":"ContainerStarted","Data":"531a6762dc62bb1c952c8e83f2260673b80da9c7463fea90a69ae20187d13f0f"} Nov 02 17:58:24.209242 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:24.208864 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" event={"ID":"852a217c-3306-4428-a34c-0414d04eaba4","Type":"ContainerStarted","Data":"84bbdb2c07230d8f585e7923e05ca0c823bf0f6f485ba2267c701b8e39ccebd4"} Nov 02 17:58:24.230087 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:24.230026 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" podStartSLOduration=1.23000761 podStartE2EDuration="1.23000761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:24.228655068 +0000 UTC m=+1476.017160594" watchObservedRunningTime="2026-11-02 17:58:24.23000761 +0000 UTC m=+1476.018513139" Nov 02 17:58:24.459555 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:24.459464 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:24.461081 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:24.461051 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 17:58:24.461201 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:24.461114 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" podUID="852a217c-3306-4428-a34c-0414d04eaba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 17:58:24.531033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:24.530249 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:24.942461 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:24.942368 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:25.108358 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:25.107594 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:25.256977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:25.256676 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:25.459632 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:25.459593 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 17:58:25.459832 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:25.459663 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" podUID="852a217c-3306-4428-a34c-0414d04eaba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 17:58:26.020020 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:26.015865 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:26.459465 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:26.459422 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 17:58:26.459971 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:26.459479 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" podUID="852a217c-3306-4428-a34c-0414d04eaba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 17:58:26.851192 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:26.851102 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:27.459749 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:27.459712 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 17:58:27.460246 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:27.459782 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" podUID="852a217c-3306-4428-a34c-0414d04eaba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 17:58:27.538994 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:27.538946 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:28.400609 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:28.400509 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:28.461701 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:28.461661 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 17:58:28.462322 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:28.462290 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" podUID="852a217c-3306-4428-a34c-0414d04eaba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 17:58:29.031944 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:29.031902 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:29.459472 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:29.459430 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 17:58:29.459644 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:29.459496 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" podUID="852a217c-3306-4428-a34c-0414d04eaba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 17:58:29.709285 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:29.709232 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:30.345920 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:30.345878 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:30.486761 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:30.486732 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:30.487740 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:30.487704 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:30.490023 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:30.489946 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 17:58:30.692411 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:30.692334 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:30.864623 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:30.864587 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:31.420091 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:31.417193 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:31.572255 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:31.572213 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:32.137978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:32.133821 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:32.945289 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:32.945236 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:33.738029 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:33.734588 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:34.403327 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:34.403284 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:35.148664 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:35.148622 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:35.844043 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:35.844003 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:36.577475 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:36.577430 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:37.077282 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:37.077250 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:37.506725 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:37.506688 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:37.849375 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:37.849300 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:37.993400 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:37.993363 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:38.378935 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:38.378887 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:38.524978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:38.524760 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:38.849009 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:38.828572 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:39.402026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:39.401919 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:39.549327 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:39.549294 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:39.925220 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:39.925176 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:40.118977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:40.111891 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:40.558973 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:40.558757 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:41.202201 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:41.202164 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:58:41.243069 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:41.243034 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:41.830396 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:41.830364 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:41.988523 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:41.988493 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:42.159473 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:42.159382 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:42.305482 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:42.305445 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:43.254541 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:43.254505 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:44.029368 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:44.029329 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:44.216659 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:44.216616 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:44.939613 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:44.939576 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:45.684825 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:45.671456 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:46.447995 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:46.443618 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:47.113237 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:47.113198 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:47.588543 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:58:47.588508 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:58:47.911769 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:47.911672 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:48.648543 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:48.648505 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:49.456975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:49.452547 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:49.614094 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:49.614061 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 17:58:49.838537 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:49.838464 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 17:58:50.260877 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:50.260832 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:51.023860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:51.023779 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:51.724033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:51.723986 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:52.433975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:52.431562 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:53.199970 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:53.199912 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:53.841738 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:53.841690 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:54.531062 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:54.531024 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:54.876030 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:54.873768 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:55.133151 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:55.113351 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:55.265936 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:55.265884 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:55.434699 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:55.434608 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:55.649427 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:55.649392 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:55.793080 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:55.793027 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:56.388004 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:56.380262 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:56.535167 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:56.535133 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:57.231915 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:57.231870 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:58.048290 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:58.048229 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:58.820369 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:58.820334 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:59.080657 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:59.080561 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:59.306243 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:59.306211 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:59.506981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:58:59.506492 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:00.216979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:00.210907 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:00.469598 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:00.469511 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:00.969562 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:00.969515 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:01.856543 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:01.856504 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:02.754349 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:02.754307 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:03.304698 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:03.304662 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:03.468156 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:03.468122 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:03.637805 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:03.637543 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:04.417089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:04.417050 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:05.150944 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:05.150893 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:05.920914 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:05.920853 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:06.605888 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:06.605810 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:07.604596 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:07.604557 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:08.274520 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:08.272786 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:09.191979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:09.186548 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:10.029922 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:10.029878 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:10.724940 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:10.718536 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:11.398767 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:11.398726 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:12.104490 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:12.104452 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:12.722210 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:12.722172 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:13.623981 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:13.622565 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:14.312033 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:14.310561 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:14.950975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:14.950594 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:15.652395 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:15.652354 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:16.372002 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:16.371826 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:17.149972 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:17.149828 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:17.590313 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:59:17.590272 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:59:17.851734 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:17.851568 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:18.528998 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:18.528936 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:19.230123 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:19.230087 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:19.939870 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:19.939829 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:20.689139 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:20.689083 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:21.350606 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:21.350567 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:22.013675 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:22.013580 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:22.677976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:22.673438 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:23.400590 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:23.400546 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:24.051026 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:24.047476 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:24.855366 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:24.855324 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:25.524706 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:25.524649 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:26.228051 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:26.227995 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:26.910514 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:26.910423 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:27.651550 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:27.651512 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:28.355974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:28.352850 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:29.016844 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:29.016774 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:29.743979 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:29.736579 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:30.399753 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:30.399280 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:31.124991 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:31.116417 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:31.774265 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:31.774226 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:32.566061 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:32.564133 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:33.244730 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:33.244680 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:34.050000 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:34.049419 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:34.422701 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:34.422615 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 17:59:34.741575 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:34.741531 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:35.840977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:35.834571 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:37.361375 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:37.361333 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:38.682918 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:38.682861 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:39.801770 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:39.801730 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:40.257910 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:40.257880 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:40.322531 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:40.322499 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7347-4cf539"] Nov 02 17:59:40.327300 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:40.327279 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-4cf539" Nov 02 17:59:40.337051 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:40.337022 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zf9kl\"" Nov 02 17:59:40.340918 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:40.340897 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7347-4cf539"] Nov 02 17:59:40.409742 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:40.409708 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2dxr\" (UniqueName: \"kubernetes.io/projected/d4c072d5-46a9-430a-b34a-6b4a2903bb59-kube-api-access-t2dxr\") pod \"test-curl-7347-4cf539\" (UID: \"d4c072d5-46a9-430a-b34a-6b4a2903bb59\") " pod="opendatahub/test-curl-7347-4cf539" Nov 02 17:59:40.511594 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:40.511500 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2dxr\" (UniqueName: \"kubernetes.io/projected/d4c072d5-46a9-430a-b34a-6b4a2903bb59-kube-api-access-t2dxr\") pod \"test-curl-7347-4cf539\" (UID: \"d4c072d5-46a9-430a-b34a-6b4a2903bb59\") " pod="opendatahub/test-curl-7347-4cf539" Nov 02 17:59:40.524772 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:40.524732 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2dxr\" (UniqueName: \"kubernetes.io/projected/d4c072d5-46a9-430a-b34a-6b4a2903bb59-kube-api-access-t2dxr\") pod \"test-curl-7347-4cf539\" (UID: \"d4c072d5-46a9-430a-b34a-6b4a2903bb59\") " pod="opendatahub/test-curl-7347-4cf539" Nov 02 17:59:40.563821 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:40.563707 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:40.642860 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:40.642819 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-4cf539" Nov 02 17:59:40.931070 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:40.931039 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7347-4cf539"] Nov 02 17:59:40.932270 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:59:40.932242 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4c072d5_46a9_430a_b34a_6b4a2903bb59.slice/crio-f2a1de7d603fefdb7d2f2307c6d3637537761ca0594d8f8cd3815a45f8bd6b32 WatchSource:0}: Error finding container f2a1de7d603fefdb7d2f2307c6d3637537761ca0594d8f8cd3815a45f8bd6b32: Status 404 returned error can't find the container with id f2a1de7d603fefdb7d2f2307c6d3637537761ca0594d8f8cd3815a45f8bd6b32 Nov 02 17:59:41.313977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:41.313562 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:41.606078 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:41.605982 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7347-4cf539" event={"ID":"d4c072d5-46a9-430a-b34a-6b4a2903bb59","Type":"ContainerStarted","Data":"6f20153aefe7f0a53f6ecf87adadf28fe48542015819bf75e055cb9d981fd14f"} Nov 02 17:59:41.606078 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:41.606031 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7347-4cf539" event={"ID":"d4c072d5-46a9-430a-b34a-6b4a2903bb59","Type":"ContainerStarted","Data":"f2a1de7d603fefdb7d2f2307c6d3637537761ca0594d8f8cd3815a45f8bd6b32"} Nov 02 17:59:41.623060 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:41.623005 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7347-4cf539" podStartSLOduration=1.622991233 podStartE2EDuration="1.622991233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:41.621021472 +0000 UTC m=+1553.409527001" watchObservedRunningTime="2026-11-02 17:59:41.622991233 +0000 UTC m=+1553.411496754" Nov 02 17:59:42.131978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:42.131650 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:42.408661 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:42.408578 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7347-4cf539"] Nov 02 17:59:42.415869 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:42.415836 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7347-4cf539"] Nov 02 17:59:42.582224 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:42.582191 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7347-8ca5b6"] Nov 02 17:59:42.583087 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:42.583070 5549 cpu_manager.go:414] "containerMap: removing container" podUID="d4c072d5-46a9-430a-b34a-6b4a2903bb59" containerName="test-curl-7347-4cf539" Nov 02 17:59:42.583145 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:42.583089 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c072d5-46a9-430a-b34a-6b4a2903bb59" containerName="test-curl-7347-4cf539" Nov 02 17:59:42.583243 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:42.583234 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c072d5-46a9-430a-b34a-6b4a2903bb59" containerName="test-curl-7347-4cf539" Nov 02 17:59:42.587229 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:42.587200 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-8ca5b6" Nov 02 17:59:42.593856 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:42.593831 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7347-8ca5b6"] Nov 02 17:59:42.757379 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:42.757338 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g2s4\" (UniqueName: \"kubernetes.io/projected/3745e71c-2bff-42a0-a57f-f8e4d7f9b65a-kube-api-access-8g2s4\") pod \"test-curl-7347-8ca5b6\" (UID: \"3745e71c-2bff-42a0-a57f-f8e4d7f9b65a\") " pod="opendatahub/test-curl-7347-8ca5b6" Nov 02 17:59:42.847977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:42.847614 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:42.865817 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:42.862093 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g2s4\" (UniqueName: \"kubernetes.io/projected/3745e71c-2bff-42a0-a57f-f8e4d7f9b65a-kube-api-access-8g2s4\") pod \"test-curl-7347-8ca5b6\" (UID: \"3745e71c-2bff-42a0-a57f-f8e4d7f9b65a\") " pod="opendatahub/test-curl-7347-8ca5b6" Nov 02 17:59:42.906778 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:42.905177 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g2s4\" (UniqueName: \"kubernetes.io/projected/3745e71c-2bff-42a0-a57f-f8e4d7f9b65a-kube-api-access-8g2s4\") pod \"test-curl-7347-8ca5b6\" (UID: \"3745e71c-2bff-42a0-a57f-f8e4d7f9b65a\") " pod="opendatahub/test-curl-7347-8ca5b6" Nov 02 17:59:42.906778 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:42.905472 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-8ca5b6" Nov 02 17:59:43.235271 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:43.235238 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7347-8ca5b6"] Nov 02 17:59:43.235643 ip-10-0-142-133 kubenswrapper[5549]: W1102 17:59:43.235389 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3745e71c_2bff_42a0_a57f_f8e4d7f9b65a.slice/crio-6dc6474e0b2e5596ad8288e79375fafdce027f2768a6f797285cc36d38aed623 WatchSource:0}: Error finding container 6dc6474e0b2e5596ad8288e79375fafdce027f2768a6f797285cc36d38aed623: Status 404 returned error can't find the container with id 6dc6474e0b2e5596ad8288e79375fafdce027f2768a6f797285cc36d38aed623 Nov 02 17:59:43.636035 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:43.632648 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:43.647241 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:43.647195 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7347-4cf539" podUID="d4c072d5-46a9-430a-b34a-6b4a2903bb59" containerName="test-curl-7347-4cf539" containerID="cri-o://6f20153aefe7f0a53f6ecf87adadf28fe48542015819bf75e055cb9d981fd14f" gracePeriod=2 Nov 02 17:59:43.647631 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:43.647581 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7347-8ca5b6" event={"ID":"3745e71c-2bff-42a0-a57f-f8e4d7f9b65a","Type":"ContainerStarted","Data":"2ff760411c9b0a7aa3ff27af5d8289d1596cc3fed685a13db562e46af4c840c0"} Nov 02 17:59:43.647804 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:43.647787 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7347-8ca5b6" event={"ID":"3745e71c-2bff-42a0-a57f-f8e4d7f9b65a","Type":"ContainerStarted","Data":"6dc6474e0b2e5596ad8288e79375fafdce027f2768a6f797285cc36d38aed623"} Nov 02 17:59:43.653247 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:59:43.653200 5549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-4cf539\" is forbidden: User \"system:node:ip-10-0-142-133.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-133.us-east-2.compute.internal' and this object" podUID="d4c072d5-46a9-430a-b34a-6b4a2903bb59" pod="opendatahub/test-curl-7347-4cf539" Nov 02 17:59:43.685077 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:59:43.685028 5549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-4cf539\" is forbidden: User \"system:node:ip-10-0-142-133.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-133.us-east-2.compute.internal' and this object" podUID="d4c072d5-46a9-430a-b34a-6b4a2903bb59" pod="opendatahub/test-curl-7347-4cf539" Nov 02 17:59:43.692796 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:43.692668 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7347-8ca5b6" podStartSLOduration=1.692645673 podStartE2EDuration="1.692645673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:43.675909329 +0000 UTC m=+1555.464414860" watchObservedRunningTime="2026-11-02 17:59:43.692645673 +0000 UTC m=+1555.481151207" Nov 02 17:59:44.382976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:44.376299 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:44.458946 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:44.457816 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7347-8ca5b6"] Nov 02 17:59:44.464564 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:44.464452 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7347-8ca5b6"] Nov 02 17:59:45.093396 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:45.093358 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:45.251317 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:45.251282 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:45.552550 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:45.552479 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:45.714560 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:45.714467 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7347-8ca5b6" podUID="3745e71c-2bff-42a0-a57f-f8e4d7f9b65a" containerName="test-curl-7347-8ca5b6" containerID="cri-o://2ff760411c9b0a7aa3ff27af5d8289d1596cc3fed685a13db562e46af4c840c0" gracePeriod=2 Nov 02 17:59:45.720312 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:59:45.720274 5549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-8ca5b6\" is forbidden: User \"system:node:ip-10-0-142-133.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-133.us-east-2.compute.internal' and this object" podUID="3745e71c-2bff-42a0-a57f-f8e4d7f9b65a" pod="opendatahub/test-curl-7347-8ca5b6" Nov 02 17:59:45.784978 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:45.779016 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:46.190783 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:46.190755 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-4cf539" Nov 02 17:59:46.193353 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:59:46.193322 5549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-8ca5b6\" is forbidden: User \"system:node:ip-10-0-142-133.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-133.us-east-2.compute.internal' and this object" podUID="3745e71c-2bff-42a0-a57f-f8e4d7f9b65a" pod="opendatahub/test-curl-7347-8ca5b6" Nov 02 17:59:46.234155 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:46.233886 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:46.285089 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:46.265830 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4c072d5-46a9-430a-b34a-6b4a2903bb59-kube-api-access-t2dxr\" (UniqueName: \"kubernetes.io/projected/d4c072d5-46a9-430a-b34a-6b4a2903bb59-kube-api-access-t2dxr\") pod \"d4c072d5-46a9-430a-b34a-6b4a2903bb59\" (UID: \"d4c072d5-46a9-430a-b34a-6b4a2903bb59\") " Nov 02 17:59:46.289984 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:46.285503 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c072d5-46a9-430a-b34a-6b4a2903bb59-kube-api-access-t2dxr" pod "d4c072d5-46a9-430a-b34a-6b4a2903bb59" (UID: "d4c072d5-46a9-430a-b34a-6b4a2903bb59"). InnerVolumeSpecName "kube-api-access-t2dxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:59:46.354603 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:46.354564 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4c072d5-46a9-430a-b34a-6b4a2903bb59" path="/var/lib/kubelet/pods/d4c072d5-46a9-430a-b34a-6b4a2903bb59/volumes" Nov 02 17:59:46.386471 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:46.386431 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2dxr\" (UniqueName: \"kubernetes.io/projected/d4c072d5-46a9-430a-b34a-6b4a2903bb59-kube-api-access-t2dxr\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:46.725098 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:46.725065 5549 generic.go:356] "Generic (PLEG): container finished" podID="d4c072d5-46a9-430a-b34a-6b4a2903bb59" containerID="6f20153aefe7f0a53f6ecf87adadf28fe48542015819bf75e055cb9d981fd14f" exitCode=137 Nov 02 17:59:46.725538 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:46.725124 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-4cf539" Nov 02 17:59:46.725538 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:46.725168 5549 scope.go:122] "RemoveContainer" containerID="6f20153aefe7f0a53f6ecf87adadf28fe48542015819bf75e055cb9d981fd14f" Nov 02 17:59:46.735795 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:46.735773 5549 scope.go:122] "RemoveContainer" containerID="6f20153aefe7f0a53f6ecf87adadf28fe48542015819bf75e055cb9d981fd14f" Nov 02 17:59:46.736140 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:59:46.736117 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f20153aefe7f0a53f6ecf87adadf28fe48542015819bf75e055cb9d981fd14f\": container with ID starting with 6f20153aefe7f0a53f6ecf87adadf28fe48542015819bf75e055cb9d981fd14f not found: ID does not exist" containerID="6f20153aefe7f0a53f6ecf87adadf28fe48542015819bf75e055cb9d981fd14f" Nov 02 17:59:46.736206 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:46.736150 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f20153aefe7f0a53f6ecf87adadf28fe48542015819bf75e055cb9d981fd14f"} err="failed to get container status \"6f20153aefe7f0a53f6ecf87adadf28fe48542015819bf75e055cb9d981fd14f\": rpc error: code = NotFound desc = could not find container \"6f20153aefe7f0a53f6ecf87adadf28fe48542015819bf75e055cb9d981fd14f\": container with ID starting with 6f20153aefe7f0a53f6ecf87adadf28fe48542015819bf75e055cb9d981fd14f not found: ID does not exist" Nov 02 17:59:47.120609 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:47.118692 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:47.588921 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:59:47.588885 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 17:59:47.976802 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:47.976772 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-8ca5b6" Nov 02 17:59:48.142776 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:48.126218 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3745e71c-2bff-42a0-a57f-f8e4d7f9b65a-kube-api-access-8g2s4\" (UniqueName: \"kubernetes.io/projected/3745e71c-2bff-42a0-a57f-f8e4d7f9b65a-kube-api-access-8g2s4\") pod \"3745e71c-2bff-42a0-a57f-f8e4d7f9b65a\" (UID: \"3745e71c-2bff-42a0-a57f-f8e4d7f9b65a\") " Nov 02 17:59:48.142776 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:48.134864 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3745e71c-2bff-42a0-a57f-f8e4d7f9b65a-kube-api-access-8g2s4" pod "3745e71c-2bff-42a0-a57f-f8e4d7f9b65a" (UID: "3745e71c-2bff-42a0-a57f-f8e4d7f9b65a"). InnerVolumeSpecName "kube-api-access-8g2s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:59:48.142776 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:48.135976 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:48.228946 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:48.228905 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g2s4\" (UniqueName: \"kubernetes.io/projected/3745e71c-2bff-42a0-a57f-f8e4d7f9b65a-kube-api-access-8g2s4\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:48.321653 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:48.321623 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3745e71c-2bff-42a0-a57f-f8e4d7f9b65a" path="/var/lib/kubelet/pods/3745e71c-2bff-42a0-a57f-f8e4d7f9b65a/volumes" Nov 02 17:59:48.742842 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:48.742799 5549 generic.go:356] "Generic (PLEG): container finished" podID="3745e71c-2bff-42a0-a57f-f8e4d7f9b65a" containerID="2ff760411c9b0a7aa3ff27af5d8289d1596cc3fed685a13db562e46af4c840c0" exitCode=137 Nov 02 17:59:48.743086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:48.742849 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-8ca5b6" Nov 02 17:59:48.743086 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:48.742868 5549 scope.go:122] "RemoveContainer" containerID="2ff760411c9b0a7aa3ff27af5d8289d1596cc3fed685a13db562e46af4c840c0" Nov 02 17:59:48.753360 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:48.753166 5549 scope.go:122] "RemoveContainer" containerID="2ff760411c9b0a7aa3ff27af5d8289d1596cc3fed685a13db562e46af4c840c0" Nov 02 17:59:48.753555 ip-10-0-142-133 kubenswrapper[5549]: E1102 17:59:48.753529 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ff760411c9b0a7aa3ff27af5d8289d1596cc3fed685a13db562e46af4c840c0\": container with ID starting with 2ff760411c9b0a7aa3ff27af5d8289d1596cc3fed685a13db562e46af4c840c0 not found: ID does not exist" containerID="2ff760411c9b0a7aa3ff27af5d8289d1596cc3fed685a13db562e46af4c840c0" Nov 02 17:59:48.753606 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:48.753565 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ff760411c9b0a7aa3ff27af5d8289d1596cc3fed685a13db562e46af4c840c0"} err="failed to get container status \"2ff760411c9b0a7aa3ff27af5d8289d1596cc3fed685a13db562e46af4c840c0\": rpc error: code = NotFound desc = could not find container \"2ff760411c9b0a7aa3ff27af5d8289d1596cc3fed685a13db562e46af4c840c0\": container with ID starting with 2ff760411c9b0a7aa3ff27af5d8289d1596cc3fed685a13db562e46af4c840c0 not found: ID does not exist" Nov 02 17:59:48.975431 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:48.975151 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:49.146260 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:49.146163 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:49.914976 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:49.908677 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:50.373617 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:50.373584 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:50.669048 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:50.663234 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:51.214069 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:51.202268 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:52.173975 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:52.167867 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:53.058145 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:53.058099 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:53.951761 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:53.951720 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:54.932686 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:54.932650 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:56.000560 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:56.000482 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:56.858974 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:56.858165 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:57.565977 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:57.564456 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:58.266398 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:58.266359 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:59.058269 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:59.058230 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:59.203596 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:59.203563 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:59.339650 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:59.339561 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:59.858831 ip-10-0-142-133 kubenswrapper[5549]: I1102 17:59:59.858784 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:00.282319 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:00.282278 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm"] Nov 02 18:00:00.283164 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:00.283144 5549 cpu_manager.go:414] "containerMap: removing container" podUID="3745e71c-2bff-42a0-a57f-f8e4d7f9b65a" containerName="test-curl-7347-8ca5b6" Nov 02 18:00:00.283164 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:00.283161 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3745e71c-2bff-42a0-a57f-f8e4d7f9b65a" containerName="test-curl-7347-8ca5b6" Nov 02 18:00:00.283371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:00.283301 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3745e71c-2bff-42a0-a57f-f8e4d7f9b65a" containerName="test-curl-7347-8ca5b6" Nov 02 18:00:00.289596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:00.289571 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" Nov 02 18:00:00.305168 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:00.304729 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm"] Nov 02 18:00:00.394677 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:00.394646 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\"" Nov 02 18:00:00.442207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:00.442169 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2abfa50d-b0bd-40c0-9d31-c4d067b9d06e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm\" (UID: \"2abfa50d-b0bd-40c0-9d31-c4d067b9d06e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" Nov 02 18:00:00.442485 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:00.442463 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txvjz\" (UniqueName: \"kubernetes.io/projected/2abfa50d-b0bd-40c0-9d31-c4d067b9d06e-kube-api-access-txvjz\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm\" (UID: \"2abfa50d-b0bd-40c0-9d31-c4d067b9d06e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" Nov 02 18:00:00.543640 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:00.543548 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2abfa50d-b0bd-40c0-9d31-c4d067b9d06e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm\" (UID: \"2abfa50d-b0bd-40c0-9d31-c4d067b9d06e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" Nov 02 18:00:00.543852 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:00.543832 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txvjz\" (UniqueName: \"kubernetes.io/projected/2abfa50d-b0bd-40c0-9d31-c4d067b9d06e-kube-api-access-txvjz\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm\" (UID: \"2abfa50d-b0bd-40c0-9d31-c4d067b9d06e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" Nov 02 18:00:00.544250 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:00.544229 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2abfa50d-b0bd-40c0-9d31-c4d067b9d06e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm\" (UID: \"2abfa50d-b0bd-40c0-9d31-c4d067b9d06e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" Nov 02 18:00:00.553055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:00.553026 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txvjz\" (UniqueName: \"kubernetes.io/projected/2abfa50d-b0bd-40c0-9d31-c4d067b9d06e-kube-api-access-txvjz\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm\" (UID: \"2abfa50d-b0bd-40c0-9d31-c4d067b9d06e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" Nov 02 18:00:00.609039 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:00.609007 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" Nov 02 18:00:00.717654 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:00.717585 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:00.877686 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:00.871614 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:00.950419 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:00:00.950387 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abfa50d_b0bd_40c0_9d31_c4d067b9d06e.slice/crio-ab23f6e48008d9442e151d8a37ca18bc53d2ca60ba13f29dd7bc534f87ff8362 WatchSource:0}: Error finding container ab23f6e48008d9442e151d8a37ca18bc53d2ca60ba13f29dd7bc534f87ff8362: Status 404 returned error can't find the container with id ab23f6e48008d9442e151d8a37ca18bc53d2ca60ba13f29dd7bc534f87ff8362 Nov 02 18:00:00.951714 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:00.951690 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm"] Nov 02 18:00:00.966897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:00.966861 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" event={"ID":"2abfa50d-b0bd-40c0-9d31-c4d067b9d06e","Type":"ContainerStarted","Data":"ab23f6e48008d9442e151d8a37ca18bc53d2ca60ba13f29dd7bc534f87ff8362"} Nov 02 18:00:01.971255 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:01.966201 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:02.040067 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:02.032104 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" event={"ID":"2abfa50d-b0bd-40c0-9d31-c4d067b9d06e","Type":"ContainerStarted","Data":"539c870a2f96f051156b7aa74d2eecceee0275d48f717c7c3e9e2dba73c712a2"} Nov 02 18:00:02.056208 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:02.056149 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" podStartSLOduration=2.056130263 podStartE2EDuration="2.056130263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:02.050460092 +0000 UTC m=+1573.838965622" watchObservedRunningTime="2026-11-02 18:00:02.056130263 +0000 UTC m=+1573.844635805" Nov 02 18:00:02.817707 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:02.806978 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:03.046633 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:03.046556 5549 generic.go:356] "Generic (PLEG): container finished" podID="2abfa50d-b0bd-40c0-9d31-c4d067b9d06e" containerID="539c870a2f96f051156b7aa74d2eecceee0275d48f717c7c3e9e2dba73c712a2" exitCode=60 Nov 02 18:00:03.047150 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:03.046691 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" event={"ID":"2abfa50d-b0bd-40c0-9d31-c4d067b9d06e","Type":"ContainerDied","Data":"539c870a2f96f051156b7aa74d2eecceee0275d48f717c7c3e9e2dba73c712a2"} Nov 02 18:00:03.047150 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:03.047017 5549 scope.go:122] "RemoveContainer" containerID="539c870a2f96f051156b7aa74d2eecceee0275d48f717c7c3e9e2dba73c712a2" Nov 02 18:00:03.727742 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:03.727110 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:04.057086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:04.057049 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" event={"ID":"2abfa50d-b0bd-40c0-9d31-c4d067b9d06e","Type":"ContainerStarted","Data":"2af8e257c49dab53cdfe5edb571a77b5ef6290d7857c64eaaf78b0defaf347f0"} Nov 02 18:00:04.573975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:04.571096 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:05.067597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:05.067563 5549 generic.go:356] "Generic (PLEG): container finished" podID="2abfa50d-b0bd-40c0-9d31-c4d067b9d06e" containerID="2af8e257c49dab53cdfe5edb571a77b5ef6290d7857c64eaaf78b0defaf347f0" exitCode=60 Nov 02 18:00:05.068104 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:05.067632 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" event={"ID":"2abfa50d-b0bd-40c0-9d31-c4d067b9d06e","Type":"ContainerDied","Data":"2af8e257c49dab53cdfe5edb571a77b5ef6290d7857c64eaaf78b0defaf347f0"} Nov 02 18:00:05.068104 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:05.067696 5549 scope.go:122] "RemoveContainer" containerID="539c870a2f96f051156b7aa74d2eecceee0275d48f717c7c3e9e2dba73c712a2" Nov 02 18:00:05.068223 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:05.068105 5549 scope.go:122] "RemoveContainer" containerID="2af8e257c49dab53cdfe5edb571a77b5ef6290d7857c64eaaf78b0defaf347f0" Nov 02 18:00:05.068576 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:00:05.068550 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm_odh-ai-gateway-infra(2abfa50d-b0bd-40c0-9d31-c4d067b9d06e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" podUID="2abfa50d-b0bd-40c0-9d31-c4d067b9d06e" Nov 02 18:00:05.220545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:05.216976 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:06.040980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:06.038483 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:06.141860 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:06.141829 5549 scope.go:122] "RemoveContainer" containerID="2af8e257c49dab53cdfe5edb571a77b5ef6290d7857c64eaaf78b0defaf347f0" Nov 02 18:00:06.142287 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:00:06.142244 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm_odh-ai-gateway-infra(2abfa50d-b0bd-40c0-9d31-c4d067b9d06e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" podUID="2abfa50d-b0bd-40c0-9d31-c4d067b9d06e" Nov 02 18:00:06.969750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:06.969701 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:07.688926 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:07.688887 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:08.725362 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:08.725322 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:09.523892 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:09.523855 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:10.414043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:10.410670 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:10.667770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:10.667683 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:10.822669 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:10.822632 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:11.208985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:11.201878 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:11.945768 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:11.943143 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:12.641438 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:12.641395 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:13.342999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:13.331540 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:14.224048 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:14.211575 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:14.990209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:14.990168 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:15.312569 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:15.312490 5549 scope.go:122] "RemoveContainer" containerID="2af8e257c49dab53cdfe5edb571a77b5ef6290d7857c64eaaf78b0defaf347f0" Nov 02 18:00:15.861028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:15.849478 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:16.285398 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:16.285365 5549 generic.go:356] "Generic (PLEG): container finished" podID="2abfa50d-b0bd-40c0-9d31-c4d067b9d06e" containerID="fff3a05e71ecd7c0b636ee658d21ee93308ffa7ad711fb443f4929820e15febd" exitCode=60 Nov 02 18:00:16.285804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:16.285440 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" event={"ID":"2abfa50d-b0bd-40c0-9d31-c4d067b9d06e","Type":"ContainerDied","Data":"fff3a05e71ecd7c0b636ee658d21ee93308ffa7ad711fb443f4929820e15febd"} Nov 02 18:00:16.285804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:16.285503 5549 scope.go:122] "RemoveContainer" containerID="2af8e257c49dab53cdfe5edb571a77b5ef6290d7857c64eaaf78b0defaf347f0" Nov 02 18:00:16.286064 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:16.286038 5549 scope.go:122] "RemoveContainer" containerID="fff3a05e71ecd7c0b636ee658d21ee93308ffa7ad711fb443f4929820e15febd" Nov 02 18:00:16.286537 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:00:16.286452 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm_odh-ai-gateway-infra(2abfa50d-b0bd-40c0-9d31-c4d067b9d06e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" podUID="2abfa50d-b0bd-40c0-9d31-c4d067b9d06e" Nov 02 18:00:16.643771 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:16.643727 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:17.329317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:17.329283 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm"] Nov 02 18:00:17.487446 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:17.487255 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:17.598888 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:17.598857 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" Nov 02 18:00:17.605124 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:00:17.605091 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:00:17.672326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:17.672277 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2abfa50d-b0bd-40c0-9d31-c4d067b9d06e-kube-api-access-txvjz\" (UniqueName: \"kubernetes.io/projected/2abfa50d-b0bd-40c0-9d31-c4d067b9d06e-kube-api-access-txvjz\") pod \"2abfa50d-b0bd-40c0-9d31-c4d067b9d06e\" (UID: \"2abfa50d-b0bd-40c0-9d31-c4d067b9d06e\") " Nov 02 18:00:17.672622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:17.672599 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2abfa50d-b0bd-40c0-9d31-c4d067b9d06e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2abfa50d-b0bd-40c0-9d31-c4d067b9d06e-maas-api-service-ca\") pod \"2abfa50d-b0bd-40c0-9d31-c4d067b9d06e\" (UID: \"2abfa50d-b0bd-40c0-9d31-c4d067b9d06e\") " Nov 02 18:00:17.673001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:17.672940 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2abfa50d-b0bd-40c0-9d31-c4d067b9d06e-maas-api-service-ca" pod "2abfa50d-b0bd-40c0-9d31-c4d067b9d06e" (UID: "2abfa50d-b0bd-40c0-9d31-c4d067b9d06e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:17.673453 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:17.673431 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2abfa50d-b0bd-40c0-9d31-c4d067b9d06e-maas-api-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.674698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:17.674671 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abfa50d-b0bd-40c0-9d31-c4d067b9d06e-kube-api-access-txvjz" pod "2abfa50d-b0bd-40c0-9d31-c4d067b9d06e" (UID: "2abfa50d-b0bd-40c0-9d31-c4d067b9d06e"). InnerVolumeSpecName "kube-api-access-txvjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:17.775630 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:17.775592 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txvjz\" (UniqueName: \"kubernetes.io/projected/2abfa50d-b0bd-40c0-9d31-c4d067b9d06e-kube-api-access-txvjz\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.343003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:18.339778 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:18.385908 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:18.385870 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" Nov 02 18:00:18.386772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:18.386725 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm" event={"ID":"2abfa50d-b0bd-40c0-9d31-c4d067b9d06e","Type":"ContainerDied","Data":"ab23f6e48008d9442e151d8a37ca18bc53d2ca60ba13f29dd7bc534f87ff8362"} Nov 02 18:00:18.386940 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:18.386842 5549 scope.go:122] "RemoveContainer" containerID="fff3a05e71ecd7c0b636ee658d21ee93308ffa7ad711fb443f4929820e15febd" Nov 02 18:00:18.437909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:18.437877 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm"] Nov 02 18:00:18.443551 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:18.443523 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894040-dnksm"] Nov 02 18:00:19.115119 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:19.115082 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:19.578652 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:19.578609 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:19.867023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:19.859103 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:19.999008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:19.998972 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:20.324880 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:20.324848 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2abfa50d-b0bd-40c0-9d31-c4d067b9d06e" path="/var/lib/kubelet/pods/2abfa50d-b0bd-40c0-9d31-c4d067b9d06e/volumes" Nov 02 18:00:20.398209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:20.398178 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:20.615113 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:20.605419 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:21.082792 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:21.082754 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:21.239750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:21.232465 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:21.939354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:21.922192 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:22.724169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:22.724130 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:23.579254 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:23.579213 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:24.368774 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:24.368737 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:24.512977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:24.510515 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:25.386986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:25.386916 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:25.639043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:25.638910 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:25.793987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:25.791593 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:26.253790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:26.253708 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:26.449246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:26.449210 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:00:27.223786 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:27.223745 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:28.768093 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:28.768050 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:29.632670 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:29.611563 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:30.325272 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:30.325234 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:30.928938 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:30.928903 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:31.621639 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:31.621592 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:32.339004 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:32.330740 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:32.946246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:32.946193 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:33.631608 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:33.631530 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:34.314911 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:34.314870 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:34.986801 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:34.986758 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:35.607159 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:35.607124 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:35.741816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:35.740842 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:35.957448 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:35.957403 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:36.541979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:36.539830 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:37.238674 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:37.238636 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:37.937975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:37.937485 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:38.749979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:38.749550 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:39.488714 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:39.488677 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:40.272590 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:40.272520 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:40.925662 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:40.925596 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:41.737975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:41.732418 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:42.640745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:42.640668 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:43.367977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:43.358338 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:44.136584 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:44.136548 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:44.935609 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:44.935574 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:46.069015 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:46.060389 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:46.930873 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:46.930833 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:47.590544 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:00:47.590503 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:00:47.720981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:47.719001 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:48.528979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:48.526550 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:49.428990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:49.420980 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:49.493577 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:49.493541 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7341-31ce46"] Nov 02 18:00:49.495171 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:49.495139 5549 cpu_manager.go:414] "containerMap: removing container" podUID="2abfa50d-b0bd-40c0-9d31-c4d067b9d06e" containerName="cleanup" Nov 02 18:00:49.495345 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:49.495330 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2abfa50d-b0bd-40c0-9d31-c4d067b9d06e" containerName="cleanup" Nov 02 18:00:49.495460 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:49.495450 5549 cpu_manager.go:414] "containerMap: removing container" podUID="2abfa50d-b0bd-40c0-9d31-c4d067b9d06e" containerName="cleanup" Nov 02 18:00:49.495544 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:49.495535 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2abfa50d-b0bd-40c0-9d31-c4d067b9d06e" containerName="cleanup" Nov 02 18:00:49.495689 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:49.495680 5549 cpu_manager.go:414] "containerMap: removing container" podUID="2abfa50d-b0bd-40c0-9d31-c4d067b9d06e" containerName="cleanup" Nov 02 18:00:49.495765 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:49.495753 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2abfa50d-b0bd-40c0-9d31-c4d067b9d06e" containerName="cleanup" Nov 02 18:00:49.496064 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:49.496045 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="2abfa50d-b0bd-40c0-9d31-c4d067b9d06e" containerName="cleanup" Nov 02 18:00:49.499974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:49.496184 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="2abfa50d-b0bd-40c0-9d31-c4d067b9d06e" containerName="cleanup" Nov 02 18:00:49.504856 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:49.504827 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7341-31ce46"] Nov 02 18:00:49.505200 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:49.505179 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7341-31ce46" Nov 02 18:00:49.531664 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:49.531635 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zf9kl\"" Nov 02 18:00:49.628732 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:49.628700 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmw59\" (UniqueName: \"kubernetes.io/projected/eaa13839-adcd-4592-8fc8-730e761f7029-kube-api-access-hmw59\") pod \"test-curl-7341-31ce46\" (UID: \"eaa13839-adcd-4592-8fc8-730e761f7029\") " pod="opendatahub/test-curl-7341-31ce46" Nov 02 18:00:49.730701 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:49.730662 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmw59\" (UniqueName: \"kubernetes.io/projected/eaa13839-adcd-4592-8fc8-730e761f7029-kube-api-access-hmw59\") pod \"test-curl-7341-31ce46\" (UID: \"eaa13839-adcd-4592-8fc8-730e761f7029\") " pod="opendatahub/test-curl-7341-31ce46" Nov 02 18:00:49.739847 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:49.739812 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmw59\" (UniqueName: \"kubernetes.io/projected/eaa13839-adcd-4592-8fc8-730e761f7029-kube-api-access-hmw59\") pod \"test-curl-7341-31ce46\" (UID: \"eaa13839-adcd-4592-8fc8-730e761f7029\") " pod="opendatahub/test-curl-7341-31ce46" Nov 02 18:00:49.874515 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:49.874484 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7341-31ce46" Nov 02 18:00:50.160495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:50.160461 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7341-31ce46"] Nov 02 18:00:50.161526 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:00:50.161500 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeaa13839_adcd_4592_8fc8_730e761f7029.slice/crio-7dbe0f00fa4fcdd8b65fbb6445c81b4d5ea29c95e9c792755a9d5e6f51cac47b WatchSource:0}: Error finding container 7dbe0f00fa4fcdd8b65fbb6445c81b4d5ea29c95e9c792755a9d5e6f51cac47b: Status 404 returned error can't find the container with id 7dbe0f00fa4fcdd8b65fbb6445c81b4d5ea29c95e9c792755a9d5e6f51cac47b Nov 02 18:00:50.210347 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:50.210284 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:50.773503 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:50.773466 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7341-31ce46" event={"ID":"eaa13839-adcd-4592-8fc8-730e761f7029","Type":"ContainerStarted","Data":"28a79b642eeda9daab793678d74db41f5ee58017180282e7829261b95710b516"} Nov 02 18:00:50.774026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:50.773515 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7341-31ce46" event={"ID":"eaa13839-adcd-4592-8fc8-730e761f7029","Type":"ContainerStarted","Data":"7dbe0f00fa4fcdd8b65fbb6445c81b4d5ea29c95e9c792755a9d5e6f51cac47b"} Nov 02 18:00:50.792895 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:50.792836 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7341-31ce46" podStartSLOduration=1.792820344 podStartE2EDuration="1.792820344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:50.789420482 +0000 UTC m=+1622.577926013" watchObservedRunningTime="2026-11-02 18:00:50.792820344 +0000 UTC m=+1622.581325869" Nov 02 18:00:50.923889 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:50.923328 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:51.476124 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:51.476085 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7341-31ce46"] Nov 02 18:00:51.482655 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:51.482620 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7341-31ce46"] Nov 02 18:00:51.616784 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:51.616690 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:52.534033 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:52.519262 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:52.789106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:52.789019 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7341-31ce46" podUID="eaa13839-adcd-4592-8fc8-730e761f7029" containerName="test-curl-7341-31ce46" containerID="cri-o://28a79b642eeda9daab793678d74db41f5ee58017180282e7829261b95710b516" gracePeriod=2 Nov 02 18:00:52.969805 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:52.969764 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz"] Nov 02 18:00:53.058454 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:53.058330 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:53.395977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:53.395884 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:54.225991 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:54.225930 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:54.508770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:54.508675 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb"] Nov 02 18:00:54.582461 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:54.582426 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:54.738990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:54.738614 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:54.752982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:54.747001 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7350-c40f5f"] Nov 02 18:00:54.752982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:54.748401 5549 cpu_manager.go:414] "containerMap: removing container" podUID="eaa13839-adcd-4592-8fc8-730e761f7029" containerName="test-curl-7341-31ce46" Nov 02 18:00:54.752982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:54.748424 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa13839-adcd-4592-8fc8-730e761f7029" containerName="test-curl-7341-31ce46" Nov 02 18:00:54.752982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:54.748667 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa13839-adcd-4592-8fc8-730e761f7029" containerName="test-curl-7341-31ce46" Nov 02 18:00:54.752982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:54.748696 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="2abfa50d-b0bd-40c0-9d31-c4d067b9d06e" containerName="cleanup" Nov 02 18:00:54.821976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:54.818680 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7350-c40f5f"] Nov 02 18:00:54.821976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:54.818794 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-c40f5f" Nov 02 18:00:54.903112 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:54.886436 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mdgw\" (UniqueName: \"kubernetes.io/projected/91054785-1d71-4ea7-973e-8e383697a8b8-kube-api-access-8mdgw\") pod \"test-curl-7350-c40f5f\" (UID: \"91054785-1d71-4ea7-973e-8e383697a8b8\") " pod="opendatahub/test-curl-7350-c40f5f" Nov 02 18:00:55.025366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:55.025320 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mdgw\" (UniqueName: \"kubernetes.io/projected/91054785-1d71-4ea7-973e-8e383697a8b8-kube-api-access-8mdgw\") pod \"test-curl-7350-c40f5f\" (UID: \"91054785-1d71-4ea7-973e-8e383697a8b8\") " pod="opendatahub/test-curl-7350-c40f5f" Nov 02 18:00:55.035793 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:55.035761 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mdgw\" (UniqueName: \"kubernetes.io/projected/91054785-1d71-4ea7-973e-8e383697a8b8-kube-api-access-8mdgw\") pod \"test-curl-7350-c40f5f\" (UID: \"91054785-1d71-4ea7-973e-8e383697a8b8\") " pod="opendatahub/test-curl-7350-c40f5f" Nov 02 18:00:55.172621 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:55.160906 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:55.226933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:55.225609 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-c40f5f" Nov 02 18:00:55.304453 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:55.304420 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7341-31ce46" Nov 02 18:00:55.436179 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:55.436143 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa13839-adcd-4592-8fc8-730e761f7029-kube-api-access-hmw59\" (UniqueName: \"kubernetes.io/projected/eaa13839-adcd-4592-8fc8-730e761f7029-kube-api-access-hmw59\") pod \"eaa13839-adcd-4592-8fc8-730e761f7029\" (UID: \"eaa13839-adcd-4592-8fc8-730e761f7029\") " Nov 02 18:00:55.438399 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:55.438368 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa13839-adcd-4592-8fc8-730e761f7029-kube-api-access-hmw59" pod "eaa13839-adcd-4592-8fc8-730e761f7029" (UID: "eaa13839-adcd-4592-8fc8-730e761f7029"). InnerVolumeSpecName "kube-api-access-hmw59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:55.538445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:55.538411 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmw59\" (UniqueName: \"kubernetes.io/projected/eaa13839-adcd-4592-8fc8-730e761f7029-kube-api-access-hmw59\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:55.557841 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:55.557813 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7350-c40f5f"] Nov 02 18:00:55.558399 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:00:55.558375 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91054785_1d71_4ea7_973e_8e383697a8b8.slice/crio-43d4878de02bc9a5ac487b2fbbd6c40ea6213c4a8ade557c2c4875fa54e1187a WatchSource:0}: Error finding container 43d4878de02bc9a5ac487b2fbbd6c40ea6213c4a8ade557c2c4875fa54e1187a: Status 404 returned error can't find the container with id 43d4878de02bc9a5ac487b2fbbd6c40ea6213c4a8ade557c2c4875fa54e1187a Nov 02 18:00:55.847583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:55.847543 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7350-c40f5f" event={"ID":"91054785-1d71-4ea7-973e-8e383697a8b8","Type":"ContainerStarted","Data":"d610253693cbe65368d06f666e0bdb268aa4e88d6748541ae8680b97cb13d93d"} Nov 02 18:00:55.847797 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:55.847599 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7350-c40f5f" event={"ID":"91054785-1d71-4ea7-973e-8e383697a8b8","Type":"ContainerStarted","Data":"43d4878de02bc9a5ac487b2fbbd6c40ea6213c4a8ade557c2c4875fa54e1187a"} Nov 02 18:00:55.849406 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:55.849329 5549 generic.go:356] "Generic (PLEG): container finished" podID="eaa13839-adcd-4592-8fc8-730e761f7029" containerID="28a79b642eeda9daab793678d74db41f5ee58017180282e7829261b95710b516" exitCode=137 Nov 02 18:00:55.849406 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:55.849352 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7341-31ce46" Nov 02 18:00:55.849557 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:55.849380 5549 scope.go:122] "RemoveContainer" containerID="28a79b642eeda9daab793678d74db41f5ee58017180282e7829261b95710b516" Nov 02 18:00:55.861184 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:55.861154 5549 scope.go:122] "RemoveContainer" containerID="28a79b642eeda9daab793678d74db41f5ee58017180282e7829261b95710b516" Nov 02 18:00:55.861565 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:00:55.861541 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28a79b642eeda9daab793678d74db41f5ee58017180282e7829261b95710b516\": container with ID starting with 28a79b642eeda9daab793678d74db41f5ee58017180282e7829261b95710b516 not found: ID does not exist" containerID="28a79b642eeda9daab793678d74db41f5ee58017180282e7829261b95710b516" Nov 02 18:00:55.861618 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:55.861575 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a79b642eeda9daab793678d74db41f5ee58017180282e7829261b95710b516"} err="failed to get container status \"28a79b642eeda9daab793678d74db41f5ee58017180282e7829261b95710b516\": rpc error: code = NotFound desc = could not find container \"28a79b642eeda9daab793678d74db41f5ee58017180282e7829261b95710b516\": container with ID starting with 28a79b642eeda9daab793678d74db41f5ee58017180282e7829261b95710b516 not found: ID does not exist" Nov 02 18:00:55.880164 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:00:55.880119 5549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7341-31ce46\" is forbidden: User \"system:node:ip-10-0-142-133.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-133.us-east-2.compute.internal' and this object" podUID="eaa13839-adcd-4592-8fc8-730e761f7029" pod="opendatahub/test-curl-7341-31ce46" Nov 02 18:00:55.975389 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:55.941440 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:56.325509 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:56.325473 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa13839-adcd-4592-8fc8-730e761f7029" path="/var/lib/kubelet/pods/eaa13839-adcd-4592-8fc8-730e761f7029/volumes" Nov 02 18:00:56.511150 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:56.511096 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7350-c40f5f" podStartSLOduration=2.511078314 podStartE2EDuration="2.511078314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:55.877034283 +0000 UTC m=+1627.665539848" watchObservedRunningTime="2026-11-02 18:00:56.511078314 +0000 UTC m=+1628.299583853" Nov 02 18:00:56.516451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:56.516420 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7350-c40f5f"] Nov 02 18:00:56.521570 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:56.521539 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7350-c40f5f"] Nov 02 18:00:56.669465 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:56.669387 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7350-583613"] Nov 02 18:00:56.670260 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:56.670244 5549 cpu_manager.go:414] "containerMap: removing container" podUID="91054785-1d71-4ea7-973e-8e383697a8b8" containerName="test-curl-7350-c40f5f" Nov 02 18:00:56.670314 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:56.670261 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91054785-1d71-4ea7-973e-8e383697a8b8" containerName="test-curl-7350-c40f5f" Nov 02 18:00:56.670400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:56.670391 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="91054785-1d71-4ea7-973e-8e383697a8b8" containerName="test-curl-7350-c40f5f" Nov 02 18:00:56.675755 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:56.675731 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-583613" Nov 02 18:00:56.681549 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:56.681516 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7350-583613"] Nov 02 18:00:56.741974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:56.735590 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:56.884427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:56.884394 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:56.912741 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:56.912705 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9bt6\" (UniqueName: \"kubernetes.io/projected/a139472c-f9c9-4c5e-986d-4edd85c59531-kube-api-access-x9bt6\") pod \"test-curl-7350-583613\" (UID: \"a139472c-f9c9-4c5e-986d-4edd85c59531\") " pod="opendatahub/test-curl-7350-583613" Nov 02 18:00:57.013869 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:57.013830 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9bt6\" (UniqueName: \"kubernetes.io/projected/a139472c-f9c9-4c5e-986d-4edd85c59531-kube-api-access-x9bt6\") pod \"test-curl-7350-583613\" (UID: \"a139472c-f9c9-4c5e-986d-4edd85c59531\") " pod="opendatahub/test-curl-7350-583613" Nov 02 18:00:57.022557 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:57.022531 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9bt6\" (UniqueName: \"kubernetes.io/projected/a139472c-f9c9-4c5e-986d-4edd85c59531-kube-api-access-x9bt6\") pod \"test-curl-7350-583613\" (UID: \"a139472c-f9c9-4c5e-986d-4edd85c59531\") " pod="opendatahub/test-curl-7350-583613" Nov 02 18:00:57.291214 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:57.291133 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-583613" Nov 02 18:00:57.295086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:57.294567 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:57.786974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:57.760009 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:57.876988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:57.876936 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7350-c40f5f" podUID="91054785-1d71-4ea7-973e-8e383697a8b8" containerName="test-curl-7350-c40f5f" containerID="cri-o://d610253693cbe65368d06f666e0bdb268aa4e88d6748541ae8680b97cb13d93d" gracePeriod=2 Nov 02 18:00:57.879888 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:00:57.879856 5549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7350-c40f5f\" is forbidden: User \"system:node:ip-10-0-142-133.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-133.us-east-2.compute.internal' and this object" podUID="91054785-1d71-4ea7-973e-8e383697a8b8" pod="opendatahub/test-curl-7350-c40f5f" Nov 02 18:00:57.918977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:57.915509 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:58.016797 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:58.016766 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7350-583613"] Nov 02 18:00:58.017054 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:00:58.017031 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda139472c_f9c9_4c5e_986d_4edd85c59531.slice/crio-72a0a5a1bd9c024e067031b530ec5e8ceac31563bb48197f6120b5d249b1b3c0 WatchSource:0}: Error finding container 72a0a5a1bd9c024e067031b530ec5e8ceac31563bb48197f6120b5d249b1b3c0: Status 404 returned error can't find the container with id 72a0a5a1bd9c024e067031b530ec5e8ceac31563bb48197f6120b5d249b1b3c0 Nov 02 18:00:58.318627 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:00:58.318533 5549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7350-c40f5f\" is forbidden: User \"system:node:ip-10-0-142-133.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-133.us-east-2.compute.internal' and this object" podUID="91054785-1d71-4ea7-973e-8e383697a8b8" pod="opendatahub/test-curl-7350-c40f5f" Nov 02 18:00:58.756977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:58.737554 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:58.888422 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:58.888384 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7350-583613" event={"ID":"a139472c-f9c9-4c5e-986d-4edd85c59531","Type":"ContainerStarted","Data":"ba0d1f5357011bf6069e95a8bc28e701599cfa23ab27b1b3116a5227d9e0b637"} Nov 02 18:00:58.888815 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:58.888437 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7350-583613" event={"ID":"a139472c-f9c9-4c5e-986d-4edd85c59531","Type":"ContainerStarted","Data":"72a0a5a1bd9c024e067031b530ec5e8ceac31563bb48197f6120b5d249b1b3c0"} Nov 02 18:00:58.909449 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:58.909387 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7350-583613" podStartSLOduration=2.909366979 podStartE2EDuration="2.909366979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:58.9062612 +0000 UTC m=+1630.694766753" watchObservedRunningTime="2026-11-02 18:00:58.909366979 +0000 UTC m=+1630.697872512" Nov 02 18:00:59.254276 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:59.254241 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:59.567126 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:59.567033 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7350-583613"] Nov 02 18:00:59.571465 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:59.571431 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7350-583613"] Nov 02 18:00:59.663552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:59.650490 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:59.713429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:59.713396 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7350-4466f5"] Nov 02 18:00:59.714331 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:59.714307 5549 cpu_manager.go:414] "containerMap: removing container" podUID="a139472c-f9c9-4c5e-986d-4edd85c59531" containerName="test-curl-7350-583613" Nov 02 18:00:59.714331 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:59.714326 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a139472c-f9c9-4c5e-986d-4edd85c59531" containerName="test-curl-7350-583613" Nov 02 18:00:59.714617 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:59.714457 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a139472c-f9c9-4c5e-986d-4edd85c59531" containerName="test-curl-7350-583613" Nov 02 18:00:59.718973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:59.718844 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-4466f5" Nov 02 18:00:59.724415 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:59.724386 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7350-4466f5"] Nov 02 18:00:59.769319 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:59.769282 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ssr8\" (UniqueName: \"kubernetes.io/projected/caa7fb6d-2fb8-4056-887d-c892b75fdf76-kube-api-access-7ssr8\") pod \"test-curl-7350-4466f5\" (UID: \"caa7fb6d-2fb8-4056-887d-c892b75fdf76\") " pod="opendatahub/test-curl-7350-4466f5" Nov 02 18:00:59.870567 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:59.870470 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ssr8\" (UniqueName: \"kubernetes.io/projected/caa7fb6d-2fb8-4056-887d-c892b75fdf76-kube-api-access-7ssr8\") pod \"test-curl-7350-4466f5\" (UID: \"caa7fb6d-2fb8-4056-887d-c892b75fdf76\") " pod="opendatahub/test-curl-7350-4466f5" Nov 02 18:00:59.879477 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:00:59.879444 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ssr8\" (UniqueName: \"kubernetes.io/projected/caa7fb6d-2fb8-4056-887d-c892b75fdf76-kube-api-access-7ssr8\") pod \"test-curl-7350-4466f5\" (UID: \"caa7fb6d-2fb8-4056-887d-c892b75fdf76\") " pod="opendatahub/test-curl-7350-4466f5" Nov 02 18:01:00.037548 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:00.037502 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-4466f5" Nov 02 18:01:00.091638 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:00.091599 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-c40f5f" Nov 02 18:01:00.176275 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:00.176189 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91054785-1d71-4ea7-973e-8e383697a8b8-kube-api-access-8mdgw\" (UniqueName: \"kubernetes.io/projected/91054785-1d71-4ea7-973e-8e383697a8b8-kube-api-access-8mdgw\") pod \"91054785-1d71-4ea7-973e-8e383697a8b8\" (UID: \"91054785-1d71-4ea7-973e-8e383697a8b8\") " Nov 02 18:01:00.178426 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:00.178395 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91054785-1d71-4ea7-973e-8e383697a8b8-kube-api-access-8mdgw" pod "91054785-1d71-4ea7-973e-8e383697a8b8" (UID: "91054785-1d71-4ea7-973e-8e383697a8b8"). InnerVolumeSpecName "kube-api-access-8mdgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:00.278008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:00.277895 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mdgw\" (UniqueName: \"kubernetes.io/projected/91054785-1d71-4ea7-973e-8e383697a8b8-kube-api-access-8mdgw\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:00.323102 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:00.323071 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91054785-1d71-4ea7-973e-8e383697a8b8" path="/var/lib/kubelet/pods/91054785-1d71-4ea7-973e-8e383697a8b8/volumes" Nov 02 18:01:00.323960 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:00.323928 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7350-4466f5"] Nov 02 18:01:00.324196 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:01:00.324171 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcaa7fb6d_2fb8_4056_887d_c892b75fdf76.slice/crio-4f0dfff2dd1ba23e0a5cd6df3ba2c475e62ccee2091ccceeca3dbe29d6f80942 WatchSource:0}: Error finding container 4f0dfff2dd1ba23e0a5cd6df3ba2c475e62ccee2091ccceeca3dbe29d6f80942: Status 404 returned error can't find the container with id 4f0dfff2dd1ba23e0a5cd6df3ba2c475e62ccee2091ccceeca3dbe29d6f80942 Nov 02 18:01:00.645090 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:00.644067 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:00.906521 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:00.906430 5549 generic.go:356] "Generic (PLEG): container finished" podID="91054785-1d71-4ea7-973e-8e383697a8b8" containerID="d610253693cbe65368d06f666e0bdb268aa4e88d6748541ae8680b97cb13d93d" exitCode=137 Nov 02 18:01:00.906521 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:00.906472 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-c40f5f" Nov 02 18:01:00.906720 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:00.906536 5549 scope.go:122] "RemoveContainer" containerID="d610253693cbe65368d06f666e0bdb268aa4e88d6748541ae8680b97cb13d93d" Nov 02 18:01:00.908483 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:00.908420 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7350-4466f5" event={"ID":"caa7fb6d-2fb8-4056-887d-c892b75fdf76","Type":"ContainerStarted","Data":"6bbaf5aceb3e6eb25d0da535c08de439017a1504527574438d691446588a054e"} Nov 02 18:01:00.908609 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:00.908529 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7350-4466f5" event={"ID":"caa7fb6d-2fb8-4056-887d-c892b75fdf76","Type":"ContainerStarted","Data":"4f0dfff2dd1ba23e0a5cd6df3ba2c475e62ccee2091ccceeca3dbe29d6f80942"} Nov 02 18:01:00.908668 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:00.908638 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7350-583613" podUID="a139472c-f9c9-4c5e-986d-4edd85c59531" containerName="test-curl-7350-583613" containerID="cri-o://ba0d1f5357011bf6069e95a8bc28e701599cfa23ab27b1b3116a5227d9e0b637" gracePeriod=2 Nov 02 18:01:00.922001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:00.921945 5549 scope.go:122] "RemoveContainer" containerID="d610253693cbe65368d06f666e0bdb268aa4e88d6748541ae8680b97cb13d93d" Nov 02 18:01:00.922636 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:01:00.922606 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d610253693cbe65368d06f666e0bdb268aa4e88d6748541ae8680b97cb13d93d\": container with ID starting with d610253693cbe65368d06f666e0bdb268aa4e88d6748541ae8680b97cb13d93d not found: ID does not exist" containerID="d610253693cbe65368d06f666e0bdb268aa4e88d6748541ae8680b97cb13d93d" Nov 02 18:01:00.922756 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:00.922651 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d610253693cbe65368d06f666e0bdb268aa4e88d6748541ae8680b97cb13d93d"} err="failed to get container status \"d610253693cbe65368d06f666e0bdb268aa4e88d6748541ae8680b97cb13d93d\": rpc error: code = NotFound desc = could not find container \"d610253693cbe65368d06f666e0bdb268aa4e88d6748541ae8680b97cb13d93d\": container with ID starting with d610253693cbe65368d06f666e0bdb268aa4e88d6748541ae8680b97cb13d93d not found: ID does not exist" Nov 02 18:01:00.930566 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:00.930506 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7350-4466f5" podStartSLOduration=1.9304879640000001 podStartE2EDuration="1.930487964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:00.925692529 +0000 UTC m=+1632.714198072" watchObservedRunningTime="2026-11-02 18:01:00.930487964 +0000 UTC m=+1632.718993492" Nov 02 18:01:01.131715 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:01.131678 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:01.279289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:01.279258 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:01.557029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:01.553550 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:01.611321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:01.611250 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7350-4466f5"] Nov 02 18:01:01.616892 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:01.616793 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7350-4466f5"] Nov 02 18:01:01.758242 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:01.758204 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7350-046cae"] Nov 02 18:01:01.759270 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:01.759245 5549 cpu_manager.go:414] "containerMap: removing container" podUID="caa7fb6d-2fb8-4056-887d-c892b75fdf76" containerName="test-curl-7350-4466f5" Nov 02 18:01:01.759270 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:01.759268 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caa7fb6d-2fb8-4056-887d-c892b75fdf76" containerName="test-curl-7350-4466f5" Nov 02 18:01:01.759498 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:01.759484 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="caa7fb6d-2fb8-4056-887d-c892b75fdf76" containerName="test-curl-7350-4466f5" Nov 02 18:01:01.765129 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:01.765109 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-046cae" Nov 02 18:01:01.778275 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:01.778247 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7350-046cae"] Nov 02 18:01:01.817302 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:01.817205 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8kb\" (UniqueName: \"kubernetes.io/projected/05cbb4ed-a2bf-462a-80fd-a3985656d8ee-kube-api-access-ws8kb\") pod \"test-curl-7350-046cae\" (UID: \"05cbb4ed-a2bf-462a-80fd-a3985656d8ee\") " pod="opendatahub/test-curl-7350-046cae" Nov 02 18:01:01.918506 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:01.918471 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8kb\" (UniqueName: \"kubernetes.io/projected/05cbb4ed-a2bf-462a-80fd-a3985656d8ee-kube-api-access-ws8kb\") pod \"test-curl-7350-046cae\" (UID: \"05cbb4ed-a2bf-462a-80fd-a3985656d8ee\") " pod="opendatahub/test-curl-7350-046cae" Nov 02 18:01:01.927830 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:01.927802 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8kb\" (UniqueName: \"kubernetes.io/projected/05cbb4ed-a2bf-462a-80fd-a3985656d8ee-kube-api-access-ws8kb\") pod \"test-curl-7350-046cae\" (UID: \"05cbb4ed-a2bf-462a-80fd-a3985656d8ee\") " pod="opendatahub/test-curl-7350-046cae" Nov 02 18:01:02.080082 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:02.080003 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-046cae" Nov 02 18:01:02.347407 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:02.347331 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:02.380068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:02.380034 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7350-046cae"] Nov 02 18:01:02.382185 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:01:02.382149 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05cbb4ed_a2bf_462a_80fd_a3985656d8ee.slice/crio-f75d9690034e53d22652b34928be96ede4e45c0c244678ad6e8aec0edc7639d4 WatchSource:0}: Error finding container f75d9690034e53d22652b34928be96ede4e45c0c244678ad6e8aec0edc7639d4: Status 404 returned error can't find the container with id f75d9690034e53d22652b34928be96ede4e45c0c244678ad6e8aec0edc7639d4 Nov 02 18:01:02.503302 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:02.503264 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:02.669031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:02.664437 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:02.928904 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:02.928868 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7350-046cae" event={"ID":"05cbb4ed-a2bf-462a-80fd-a3985656d8ee","Type":"ContainerStarted","Data":"98a28d02dd33092ba430db4442ec0a6dce54054c7b07c0afd2216db2a5076491"} Nov 02 18:01:02.929046 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:02.928911 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7350-046cae" event={"ID":"05cbb4ed-a2bf-462a-80fd-a3985656d8ee","Type":"ContainerStarted","Data":"f75d9690034e53d22652b34928be96ede4e45c0c244678ad6e8aec0edc7639d4"} Nov 02 18:01:02.929046 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:02.929013 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7350-4466f5" podUID="caa7fb6d-2fb8-4056-887d-c892b75fdf76" containerName="test-curl-7350-4466f5" containerID="cri-o://6bbaf5aceb3e6eb25d0da535c08de439017a1504527574438d691446588a054e" gracePeriod=2 Nov 02 18:01:02.952027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:02.951969 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7350-046cae" podStartSLOduration=1.951939319 podStartE2EDuration="1.951939319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:02.947942473 +0000 UTC m=+1634.736448003" watchObservedRunningTime="2026-11-02 18:01:02.951939319 +0000 UTC m=+1634.740444847" Nov 02 18:01:03.078348 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:03.078321 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-583613" Nov 02 18:01:03.218836 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:03.218801 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a139472c-f9c9-4c5e-986d-4edd85c59531-kube-api-access-x9bt6\" (UniqueName: \"kubernetes.io/projected/a139472c-f9c9-4c5e-986d-4edd85c59531-kube-api-access-x9bt6\") pod \"a139472c-f9c9-4c5e-986d-4edd85c59531\" (UID: \"a139472c-f9c9-4c5e-986d-4edd85c59531\") " Nov 02 18:01:03.220929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:03.220901 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a139472c-f9c9-4c5e-986d-4edd85c59531-kube-api-access-x9bt6" pod "a139472c-f9c9-4c5e-986d-4edd85c59531" (UID: "a139472c-f9c9-4c5e-986d-4edd85c59531"). InnerVolumeSpecName "kube-api-access-x9bt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:03.321509 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:03.321475 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9bt6\" (UniqueName: \"kubernetes.io/projected/a139472c-f9c9-4c5e-986d-4edd85c59531-kube-api-access-x9bt6\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:03.502446 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:03.498400 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:03.715684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:03.715639 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7350-046cae"] Nov 02 18:01:03.720597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:03.720565 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7350-046cae"] Nov 02 18:01:03.939003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:03.938888 5549 generic.go:356] "Generic (PLEG): container finished" podID="a139472c-f9c9-4c5e-986d-4edd85c59531" containerID="ba0d1f5357011bf6069e95a8bc28e701599cfa23ab27b1b3116a5227d9e0b637" exitCode=137 Nov 02 18:01:03.939003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:03.938939 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-583613" Nov 02 18:01:03.939245 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:03.939023 5549 scope.go:122] "RemoveContainer" containerID="ba0d1f5357011bf6069e95a8bc28e701599cfa23ab27b1b3116a5227d9e0b637" Nov 02 18:01:03.950423 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:03.950402 5549 scope.go:122] "RemoveContainer" containerID="ba0d1f5357011bf6069e95a8bc28e701599cfa23ab27b1b3116a5227d9e0b637" Nov 02 18:01:03.950651 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:01:03.950629 5549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7350-583613\" is forbidden: User \"system:node:ip-10-0-142-133.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-133.us-east-2.compute.internal' and this object" podUID="a139472c-f9c9-4c5e-986d-4edd85c59531" pod="opendatahub/test-curl-7350-583613" Nov 02 18:01:03.950749 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:01:03.950661 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0d1f5357011bf6069e95a8bc28e701599cfa23ab27b1b3116a5227d9e0b637\": container with ID starting with ba0d1f5357011bf6069e95a8bc28e701599cfa23ab27b1b3116a5227d9e0b637 not found: ID does not exist" containerID="ba0d1f5357011bf6069e95a8bc28e701599cfa23ab27b1b3116a5227d9e0b637" Nov 02 18:01:03.950749 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:03.950691 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0d1f5357011bf6069e95a8bc28e701599cfa23ab27b1b3116a5227d9e0b637"} err="failed to get container status \"ba0d1f5357011bf6069e95a8bc28e701599cfa23ab27b1b3116a5227d9e0b637\": rpc error: code = NotFound desc = could not find container \"ba0d1f5357011bf6069e95a8bc28e701599cfa23ab27b1b3116a5227d9e0b637\": container with ID starting with ba0d1f5357011bf6069e95a8bc28e701599cfa23ab27b1b3116a5227d9e0b637 not found: ID does not exist" Nov 02 18:01:04.322760 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:04.322729 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a139472c-f9c9-4c5e-986d-4edd85c59531" path="/var/lib/kubelet/pods/a139472c-f9c9-4c5e-986d-4edd85c59531/volumes" Nov 02 18:01:04.385977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:04.385216 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:04.950306 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:04.950180 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7350-046cae" podUID="05cbb4ed-a2bf-462a-80fd-a3985656d8ee" containerName="test-curl-7350-046cae" containerID="cri-o://98a28d02dd33092ba430db4442ec0a6dce54054c7b07c0afd2216db2a5076491" gracePeriod=2 Nov 02 18:01:05.147336 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:05.147188 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:05.263021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:05.262989 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-4466f5" Nov 02 18:01:05.293905 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:05.293868 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:05.368209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:05.368179 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caa7fb6d-2fb8-4056-887d-c892b75fdf76-kube-api-access-7ssr8\" (UniqueName: \"kubernetes.io/projected/caa7fb6d-2fb8-4056-887d-c892b75fdf76-kube-api-access-7ssr8\") pod \"caa7fb6d-2fb8-4056-887d-c892b75fdf76\" (UID: \"caa7fb6d-2fb8-4056-887d-c892b75fdf76\") " Nov 02 18:01:05.370437 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:05.370410 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa7fb6d-2fb8-4056-887d-c892b75fdf76-kube-api-access-7ssr8" pod "caa7fb6d-2fb8-4056-887d-c892b75fdf76" (UID: "caa7fb6d-2fb8-4056-887d-c892b75fdf76"). InnerVolumeSpecName "kube-api-access-7ssr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:05.470929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:05.470872 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ssr8\" (UniqueName: \"kubernetes.io/projected/caa7fb6d-2fb8-4056-887d-c892b75fdf76-kube-api-access-7ssr8\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:05.710656 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:05.710619 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:05.861976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:05.861019 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:06.026030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:06.012361 5549 generic.go:356] "Generic (PLEG): container finished" podID="caa7fb6d-2fb8-4056-887d-c892b75fdf76" containerID="6bbaf5aceb3e6eb25d0da535c08de439017a1504527574438d691446588a054e" exitCode=137 Nov 02 18:01:06.026030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:06.012775 5549 scope.go:122] "RemoveContainer" containerID="6bbaf5aceb3e6eb25d0da535c08de439017a1504527574438d691446588a054e" Nov 02 18:01:06.026030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:06.012931 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-4466f5" Nov 02 18:01:06.082968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:06.080360 5549 scope.go:122] "RemoveContainer" containerID="6bbaf5aceb3e6eb25d0da535c08de439017a1504527574438d691446588a054e" Nov 02 18:01:06.082968 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:01:06.082202 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bbaf5aceb3e6eb25d0da535c08de439017a1504527574438d691446588a054e\": container with ID starting with 6bbaf5aceb3e6eb25d0da535c08de439017a1504527574438d691446588a054e not found: ID does not exist" containerID="6bbaf5aceb3e6eb25d0da535c08de439017a1504527574438d691446588a054e" Nov 02 18:01:06.082968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:06.082241 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bbaf5aceb3e6eb25d0da535c08de439017a1504527574438d691446588a054e"} err="failed to get container status \"6bbaf5aceb3e6eb25d0da535c08de439017a1504527574438d691446588a054e\": rpc error: code = NotFound desc = could not find container \"6bbaf5aceb3e6eb25d0da535c08de439017a1504527574438d691446588a054e\": container with ID starting with 6bbaf5aceb3e6eb25d0da535c08de439017a1504527574438d691446588a054e not found: ID does not exist" Nov 02 18:01:06.104977 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:01:06.096218 5549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7350-4466f5\" is forbidden: User \"system:node:ip-10-0-142-133.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-133.us-east-2.compute.internal' and this object" podUID="caa7fb6d-2fb8-4056-887d-c892b75fdf76" pod="opendatahub/test-curl-7350-4466f5" Nov 02 18:01:06.272046 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:06.268848 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:06.347034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:06.339040 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caa7fb6d-2fb8-4056-887d-c892b75fdf76" path="/var/lib/kubelet/pods/caa7fb6d-2fb8-4056-887d-c892b75fdf76/volumes" Nov 02 18:01:07.024368 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:07.024332 5549 generic.go:356] "Generic (PLEG): container finished" podID="05cbb4ed-a2bf-462a-80fd-a3985656d8ee" containerID="98a28d02dd33092ba430db4442ec0a6dce54054c7b07c0afd2216db2a5076491" exitCode=137 Nov 02 18:01:07.213687 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:07.213578 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:07.270562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:07.270527 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-046cae" Nov 02 18:01:07.367978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:07.365182 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05cbb4ed-a2bf-462a-80fd-a3985656d8ee-kube-api-access-ws8kb\" (UniqueName: \"kubernetes.io/projected/05cbb4ed-a2bf-462a-80fd-a3985656d8ee-kube-api-access-ws8kb\") pod \"05cbb4ed-a2bf-462a-80fd-a3985656d8ee\" (UID: \"05cbb4ed-a2bf-462a-80fd-a3985656d8ee\") " Nov 02 18:01:07.377818 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:07.377394 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05cbb4ed-a2bf-462a-80fd-a3985656d8ee-kube-api-access-ws8kb" pod "05cbb4ed-a2bf-462a-80fd-a3985656d8ee" (UID: "05cbb4ed-a2bf-462a-80fd-a3985656d8ee"). InnerVolumeSpecName "kube-api-access-ws8kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:07.467690 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:07.467608 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws8kb\" (UniqueName: \"kubernetes.io/projected/05cbb4ed-a2bf-462a-80fd-a3985656d8ee-kube-api-access-ws8kb\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:08.001999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.001944 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" podUID="8bc4054d-d2ce-4d98-8daa-a70690cf863e" containerName="main" containerID="cri-o://bc3f9f0d925afb77c8d668ce610b6a6c4b95f8510e8ca2f941d79331e35d3cd0" gracePeriod=45 Nov 02 18:01:08.038201 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.034381 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:08.104431 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.101826 5549 scope.go:122] "RemoveContainer" containerID="98a28d02dd33092ba430db4442ec0a6dce54054c7b07c0afd2216db2a5076491" Nov 02 18:01:08.104431 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.102038 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-046cae" Nov 02 18:01:08.117621 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:01:08.117577 5549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7350-046cae\" is forbidden: User \"system:node:ip-10-0-142-133.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-133.us-east-2.compute.internal' and this object" podUID="05cbb4ed-a2bf-462a-80fd-a3985656d8ee" pod="opendatahub/test-curl-7350-046cae" Nov 02 18:01:08.198821 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.198788 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:01:08.573575 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:01:08.573053 5549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7350-046cae\" is forbidden: User \"system:node:ip-10-0-142-133.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-133.us-east-2.compute.internal' and this object" podUID="05cbb4ed-a2bf-462a-80fd-a3985656d8ee" pod="opendatahub/test-curl-7350-046cae" Nov 02 18:01:08.580159 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.580131 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05cbb4ed-a2bf-462a-80fd-a3985656d8ee" path="/var/lib/kubelet/pods/05cbb4ed-a2bf-462a-80fd-a3985656d8ee/volumes" Nov 02 18:01:08.621451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.621426 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 18:01:08.761735 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.761694 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bc4054d-d2ce-4d98-8daa-a70690cf863e-tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc4054d-d2ce-4d98-8daa-a70690cf863e-tls-certs\") pod \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " Nov 02 18:01:08.761916 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.761816 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-tmp-dir\") pod \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " Nov 02 18:01:08.761916 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.761854 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-home\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-home\") pod \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " Nov 02 18:01:08.761916 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.761895 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc4054d-d2ce-4d98-8daa-a70690cf863e-kube-api-access-xnhvt\" (UniqueName: \"kubernetes.io/projected/8bc4054d-d2ce-4d98-8daa-a70690cf863e-kube-api-access-xnhvt\") pod \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " Nov 02 18:01:08.762142 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.761940 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-model-cache\") pod \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " Nov 02 18:01:08.762142 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.762043 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-dshm\") pod \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\" (UID: \"8bc4054d-d2ce-4d98-8daa-a70690cf863e\") " Nov 02 18:01:08.762256 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.762157 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-tmp-dir" pod "8bc4054d-d2ce-4d98-8daa-a70690cf863e" (UID: "8bc4054d-d2ce-4d98-8daa-a70690cf863e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:08.762256 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.762174 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-home" pod "8bc4054d-d2ce-4d98-8daa-a70690cf863e" (UID: "8bc4054d-d2ce-4d98-8daa-a70690cf863e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:08.762256 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.762240 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-model-cache" pod "8bc4054d-d2ce-4d98-8daa-a70690cf863e" (UID: "8bc4054d-d2ce-4d98-8daa-a70690cf863e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:08.762803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.762779 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:08.762803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.762800 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:08.762803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.762809 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:08.764253 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.764221 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-dshm" pod "8bc4054d-d2ce-4d98-8daa-a70690cf863e" (UID: "8bc4054d-d2ce-4d98-8daa-a70690cf863e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:08.764253 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.764236 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bc4054d-d2ce-4d98-8daa-a70690cf863e-tls-certs" pod "8bc4054d-d2ce-4d98-8daa-a70690cf863e" (UID: "8bc4054d-d2ce-4d98-8daa-a70690cf863e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:01:08.764420 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.764354 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc4054d-d2ce-4d98-8daa-a70690cf863e-kube-api-access-xnhvt" pod "8bc4054d-d2ce-4d98-8daa-a70690cf863e" (UID: "8bc4054d-d2ce-4d98-8daa-a70690cf863e"). InnerVolumeSpecName "kube-api-access-xnhvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:08.864181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.864099 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc4054d-d2ce-4d98-8daa-a70690cf863e-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:08.864181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.864129 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnhvt\" (UniqueName: \"kubernetes.io/projected/8bc4054d-d2ce-4d98-8daa-a70690cf863e-kube-api-access-xnhvt\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:08.864181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.864146 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc4054d-d2ce-4d98-8daa-a70690cf863e-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:08.889171 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.889138 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb"] Nov 02 18:01:08.889348 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.889325 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" podUID="a3822c1d-660b-4c73-9104-9bde5105e0c7" containerName="istio-proxy" containerID="cri-o://c9fe04711861759b38d73fb23296027177dfa6a449654b1330e47f5d0c49a7f2" gracePeriod=30 Nov 02 18:01:08.893451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:08.893423 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh"] Nov 02 18:01:09.111908 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:09.111875 5549 generic.go:356] "Generic (PLEG): container finished" podID="8bc4054d-d2ce-4d98-8daa-a70690cf863e" containerID="bc3f9f0d925afb77c8d668ce610b6a6c4b95f8510e8ca2f941d79331e35d3cd0" exitCode=0 Nov 02 18:01:09.112126 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:09.111922 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" Nov 02 18:01:09.112126 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:09.111974 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" event={"ID":"8bc4054d-d2ce-4d98-8daa-a70690cf863e","Type":"ContainerDied","Data":"bc3f9f0d925afb77c8d668ce610b6a6c4b95f8510e8ca2f941d79331e35d3cd0"} Nov 02 18:01:09.112126 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:09.112016 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz" event={"ID":"8bc4054d-d2ce-4d98-8daa-a70690cf863e","Type":"ContainerDied","Data":"c74e1f8883311fcfeebc43d6f53c81a97dec11bf14043e9e37587b3ba38fceae"} Nov 02 18:01:09.112126 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:09.112038 5549 scope.go:122] "RemoveContainer" containerID="bc3f9f0d925afb77c8d668ce610b6a6c4b95f8510e8ca2f941d79331e35d3cd0" Nov 02 18:01:09.112309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:09.112131 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" podUID="034a5399-ee48-4bfb-885c-0bcfd316985e" containerName="istio-proxy" containerID="cri-o://25f459a452a6a9c3c43cc2c233924826015ec6317989ce4df6ebf40fd9a58443" gracePeriod=30 Nov 02 18:01:09.123838 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:09.123820 5549 scope.go:122] "RemoveContainer" containerID="bc3f9f0d925afb77c8d668ce610b6a6c4b95f8510e8ca2f941d79331e35d3cd0" Nov 02 18:01:09.124145 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:01:09.124123 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3f9f0d925afb77c8d668ce610b6a6c4b95f8510e8ca2f941d79331e35d3cd0\": container with ID starting with bc3f9f0d925afb77c8d668ce610b6a6c4b95f8510e8ca2f941d79331e35d3cd0 not found: ID does not exist" containerID="bc3f9f0d925afb77c8d668ce610b6a6c4b95f8510e8ca2f941d79331e35d3cd0" Nov 02 18:01:09.124191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:09.124154 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3f9f0d925afb77c8d668ce610b6a6c4b95f8510e8ca2f941d79331e35d3cd0"} err="failed to get container status \"bc3f9f0d925afb77c8d668ce610b6a6c4b95f8510e8ca2f941d79331e35d3cd0\": rpc error: code = NotFound desc = could not find container \"bc3f9f0d925afb77c8d668ce610b6a6c4b95f8510e8ca2f941d79331e35d3cd0\": container with ID starting with bc3f9f0d925afb77c8d668ce610b6a6c4b95f8510e8ca2f941d79331e35d3cd0 not found: ID does not exist" Nov 02 18:01:09.157901 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:09.157870 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz"] Nov 02 18:01:09.169378 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:09.169351 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-47e1c2/auth-test-model-47e1c2-kserve-7b8fbf649c-xvsfz"] Nov 02 18:01:09.224849 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:09.224816 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:09.490682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:09.490650 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:09.537885 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:09.537848 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" podUID="7b70cca5-bca5-4aaa-97da-4cf46d011a44" containerName="main" containerID="cri-o://c8525140fa1af86e98e416909d5d8bf9b1a146809b4545169a24b37f8f2c5398" gracePeriod=45 Nov 02 18:01:09.641976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:09.638201 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:10.392085 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.391845 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bc4054d-d2ce-4d98-8daa-a70690cf863e" path="/var/lib/kubelet/pods/8bc4054d-d2ce-4d98-8daa-a70690cf863e/volumes" Nov 02 18:01:10.529108 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.526983 5549 generic.go:356] "Generic (PLEG): container finished" podID="7b70cca5-bca5-4aaa-97da-4cf46d011a44" containerID="c8525140fa1af86e98e416909d5d8bf9b1a146809b4545169a24b37f8f2c5398" exitCode=0 Nov 02 18:01:10.530214 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.528784 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:10.530436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.528981 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" event={"ID":"7b70cca5-bca5-4aaa-97da-4cf46d011a44","Type":"ContainerDied","Data":"c8525140fa1af86e98e416909d5d8bf9b1a146809b4545169a24b37f8f2c5398"} Nov 02 18:01:10.661053 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.658382 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 18:01:10.684665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.675459 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-home\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-home\") pod \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " Nov 02 18:01:10.684665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.675527 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-dshm\") pod \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " Nov 02 18:01:10.684665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.675813 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b70cca5-bca5-4aaa-97da-4cf46d011a44-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b70cca5-bca5-4aaa-97da-4cf46d011a44-tls-certs\") pod \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " Nov 02 18:01:10.684665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.675877 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-model-cache\") pod \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " Nov 02 18:01:10.684665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.675974 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-tmp-dir\") pod \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " Nov 02 18:01:10.684665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.676004 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b70cca5-bca5-4aaa-97da-4cf46d011a44-kube-api-access-cbtz9\" (UniqueName: \"kubernetes.io/projected/7b70cca5-bca5-4aaa-97da-4cf46d011a44-kube-api-access-cbtz9\") pod \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\" (UID: \"7b70cca5-bca5-4aaa-97da-4cf46d011a44\") " Nov 02 18:01:10.685793 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.685597 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-model-cache" pod "7b70cca5-bca5-4aaa-97da-4cf46d011a44" (UID: "7b70cca5-bca5-4aaa-97da-4cf46d011a44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:10.686315 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.686013 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-tmp-dir" pod "7b70cca5-bca5-4aaa-97da-4cf46d011a44" (UID: "7b70cca5-bca5-4aaa-97da-4cf46d011a44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:10.688163 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.688141 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-home" pod "7b70cca5-bca5-4aaa-97da-4cf46d011a44" (UID: "7b70cca5-bca5-4aaa-97da-4cf46d011a44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:10.705690 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.705474 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b70cca5-bca5-4aaa-97da-4cf46d011a44-tls-certs" pod "7b70cca5-bca5-4aaa-97da-4cf46d011a44" (UID: "7b70cca5-bca5-4aaa-97da-4cf46d011a44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:01:10.745242 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.738053 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b70cca5-bca5-4aaa-97da-4cf46d011a44-kube-api-access-cbtz9" pod "7b70cca5-bca5-4aaa-97da-4cf46d011a44" (UID: "7b70cca5-bca5-4aaa-97da-4cf46d011a44"). InnerVolumeSpecName "kube-api-access-cbtz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:10.745242 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.738861 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:10.745242 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.739218 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-dshm" pod "7b70cca5-bca5-4aaa-97da-4cf46d011a44" (UID: "7b70cca5-bca5-4aaa-97da-4cf46d011a44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:10.821023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.811723 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b70cca5-bca5-4aaa-97da-4cf46d011a44-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:10.821023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.811765 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:10.821023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.811779 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:10.821023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.811793 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbtz9\" (UniqueName: \"kubernetes.io/projected/7b70cca5-bca5-4aaa-97da-4cf46d011a44-kube-api-access-cbtz9\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:10.821023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.811807 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:10.821023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.811819 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b70cca5-bca5-4aaa-97da-4cf46d011a44-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:10.963620 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:10.963587 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7"] Nov 02 18:01:11.042257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:11.042219 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:11.075507 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:11.075456 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" podUID="7b70cca5-bca5-4aaa-97da-4cf46d011a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.189:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:01:11.548649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:11.548604 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" event={"ID":"7b70cca5-bca5-4aaa-97da-4cf46d011a44","Type":"ContainerDied","Data":"5bd1f29ee9790e1e6034e9923e508e8b7a630a5dc9281ca94de1132b4431ee8b"} Nov 02 18:01:11.548831 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:11.548672 5549 scope.go:122] "RemoveContainer" containerID="c8525140fa1af86e98e416909d5d8bf9b1a146809b4545169a24b37f8f2c5398" Nov 02 18:01:11.548890 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:11.548830 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb" Nov 02 18:01:11.593768 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:11.593734 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb"] Nov 02 18:01:11.604669 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:11.604637 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9165c3/auth-test-model-9165c3-kserve-5476ff7fd9-bwfnb"] Nov 02 18:01:11.657068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:11.657018 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:11.816561 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:11.816419 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:12.324313 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.324282 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b70cca5-bca5-4aaa-97da-4cf46d011a44" path="/var/lib/kubelet/pods/7b70cca5-bca5-4aaa-97da-4cf46d011a44/volumes" Nov 02 18:01:12.367688 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.367652 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:12.427613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.427577 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc"] Nov 02 18:01:12.428447 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.428430 5549 cpu_manager.go:414] "containerMap: removing container" podUID="8bc4054d-d2ce-4d98-8daa-a70690cf863e" containerName="main" Nov 02 18:01:12.428494 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.428449 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bc4054d-d2ce-4d98-8daa-a70690cf863e" containerName="main" Nov 02 18:01:12.428494 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.428481 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7b70cca5-bca5-4aaa-97da-4cf46d011a44" containerName="main" Nov 02 18:01:12.428494 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.428486 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b70cca5-bca5-4aaa-97da-4cf46d011a44" containerName="main" Nov 02 18:01:12.428593 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.428501 5549 cpu_manager.go:414] "containerMap: removing container" podUID="05cbb4ed-a2bf-462a-80fd-a3985656d8ee" containerName="test-curl-7350-046cae" Nov 02 18:01:12.428593 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.428507 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05cbb4ed-a2bf-462a-80fd-a3985656d8ee" containerName="test-curl-7350-046cae" Nov 02 18:01:12.428666 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.428629 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bc4054d-d2ce-4d98-8daa-a70690cf863e" containerName="main" Nov 02 18:01:12.428666 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.428642 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b70cca5-bca5-4aaa-97da-4cf46d011a44" containerName="main" Nov 02 18:01:12.428666 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.428651 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="05cbb4ed-a2bf-462a-80fd-a3985656d8ee" containerName="test-curl-7350-046cae" Nov 02 18:01:12.433732 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.433706 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" Nov 02 18:01:12.438238 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.438216 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-5ec436\"" Nov 02 18:01:12.443030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.443008 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc"] Nov 02 18:01:12.463763 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.463731 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7"] Nov 02 18:01:12.468744 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.468703 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" Nov 02 18:01:12.474584 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.473177 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-5ec436-dockercfg-2mdhk\"" Nov 02 18:01:12.476205 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.476177 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7"] Nov 02 18:01:12.522332 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.522299 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:12.576056 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.571335 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhpvw\" (UniqueName: \"kubernetes.io/projected/c2b8094f-68c0-461b-b5c9-c6a431267bf2-kube-api-access-nhpvw\") pod \"payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc\" (UID: \"c2b8094f-68c0-461b-b5c9-c6a431267bf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" Nov 02 18:01:12.576056 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.571660 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5700c62a-902c-4366-8a4e-8d8ca71b3eef-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7\" (UID: \"5700c62a-902c-4366-8a4e-8d8ca71b3eef\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" Nov 02 18:01:12.576056 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.571711 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5qwp\" (UniqueName: \"kubernetes.io/projected/5700c62a-902c-4366-8a4e-8d8ca71b3eef-kube-api-access-h5qwp\") pod \"payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7\" (UID: \"5700c62a-902c-4366-8a4e-8d8ca71b3eef\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" Nov 02 18:01:12.576056 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.571801 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b8094f-68c0-461b-b5c9-c6a431267bf2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc\" (UID: \"c2b8094f-68c0-461b-b5c9-c6a431267bf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" Nov 02 18:01:12.676988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.674098 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5700c62a-902c-4366-8a4e-8d8ca71b3eef-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7\" (UID: \"5700c62a-902c-4366-8a4e-8d8ca71b3eef\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" Nov 02 18:01:12.676988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.674166 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5qwp\" (UniqueName: \"kubernetes.io/projected/5700c62a-902c-4366-8a4e-8d8ca71b3eef-kube-api-access-h5qwp\") pod \"payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7\" (UID: \"5700c62a-902c-4366-8a4e-8d8ca71b3eef\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" Nov 02 18:01:12.676988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.674258 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b8094f-68c0-461b-b5c9-c6a431267bf2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc\" (UID: \"c2b8094f-68c0-461b-b5c9-c6a431267bf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" Nov 02 18:01:12.676988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.674380 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhpvw\" (UniqueName: \"kubernetes.io/projected/c2b8094f-68c0-461b-b5c9-c6a431267bf2-kube-api-access-nhpvw\") pod \"payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc\" (UID: \"c2b8094f-68c0-461b-b5c9-c6a431267bf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" Nov 02 18:01:12.685978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.684694 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b8094f-68c0-461b-b5c9-c6a431267bf2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc\" (UID: \"c2b8094f-68c0-461b-b5c9-c6a431267bf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" Nov 02 18:01:12.689976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.686644 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5700c62a-902c-4366-8a4e-8d8ca71b3eef-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7\" (UID: \"5700c62a-902c-4366-8a4e-8d8ca71b3eef\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" Nov 02 18:01:12.710976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.710465 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhpvw\" (UniqueName: \"kubernetes.io/projected/c2b8094f-68c0-461b-b5c9-c6a431267bf2-kube-api-access-nhpvw\") pod \"payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc\" (UID: \"c2b8094f-68c0-461b-b5c9-c6a431267bf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" Nov 02 18:01:12.727018 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.713066 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5qwp\" (UniqueName: \"kubernetes.io/projected/5700c62a-902c-4366-8a4e-8d8ca71b3eef-kube-api-access-h5qwp\") pod \"payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7\" (UID: \"5700c62a-902c-4366-8a4e-8d8ca71b3eef\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" Nov 02 18:01:12.747981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.743138 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:12.755980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.753299 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" Nov 02 18:01:12.855383 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.854680 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" Nov 02 18:01:12.955005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:12.954044 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:13.663029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:13.655822 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc"] Nov 02 18:01:13.669273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:13.664509 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:13.717670 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:13.717637 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7"] Nov 02 18:01:14.067977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.065597 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:14.251250 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.251220 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 18:01:14.271442 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.271408 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3822c1d-660b-4c73-9104-9bde5105e0c7-kube-api-access-xk95s\" (UniqueName: \"kubernetes.io/projected/a3822c1d-660b-4c73-9104-9bde5105e0c7-kube-api-access-xk95s\") pod \"a3822c1d-660b-4c73-9104-9bde5105e0c7\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " Nov 02 18:01:14.271577 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.271498 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-data\") pod \"a3822c1d-660b-4c73-9104-9bde5105e0c7\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " Nov 02 18:01:14.271577 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.271531 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-workload-socket\") pod \"a3822c1d-660b-4c73-9104-9bde5105e0c7\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " Nov 02 18:01:14.271727 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.271578 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-token\" (UniqueName: \"kubernetes.io/projected/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-token\") pod \"a3822c1d-660b-4c73-9104-9bde5105e0c7\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " Nov 02 18:01:14.271727 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.271650 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-envoy\") pod \"a3822c1d-660b-4c73-9104-9bde5105e0c7\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " Nov 02 18:01:14.271839 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.271777 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3822c1d-660b-4c73-9104-9bde5105e0c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3822c1d-660b-4c73-9104-9bde5105e0c7-istiod-ca-cert\") pod \"a3822c1d-660b-4c73-9104-9bde5105e0c7\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " Nov 02 18:01:14.271839 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.271805 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-data" pod "a3822c1d-660b-4c73-9104-9bde5105e0c7" (UID: "a3822c1d-660b-4c73-9104-9bde5105e0c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:14.271839 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.271823 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-workload-socket" pod "a3822c1d-660b-4c73-9104-9bde5105e0c7" (UID: "a3822c1d-660b-4c73-9104-9bde5105e0c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:14.272020 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.271974 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-podinfo\") pod \"a3822c1d-660b-4c73-9104-9bde5105e0c7\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " Nov 02 18:01:14.272114 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.272078 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-credential-socket\") pod \"a3822c1d-660b-4c73-9104-9bde5105e0c7\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " Nov 02 18:01:14.272183 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.272120 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-workload-certs\") pod \"a3822c1d-660b-4c73-9104-9bde5105e0c7\" (UID: \"a3822c1d-660b-4c73-9104-9bde5105e0c7\") " Nov 02 18:01:14.272183 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.272147 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3822c1d-660b-4c73-9104-9bde5105e0c7-istiod-ca-cert" pod "a3822c1d-660b-4c73-9104-9bde5105e0c7" (UID: "a3822c1d-660b-4c73-9104-9bde5105e0c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:01:14.272429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.272406 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-credential-socket" pod "a3822c1d-660b-4c73-9104-9bde5105e0c7" (UID: "a3822c1d-660b-4c73-9104-9bde5105e0c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:14.272564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.272542 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-workload-certs" pod "a3822c1d-660b-4c73-9104-9bde5105e0c7" (UID: "a3822c1d-660b-4c73-9104-9bde5105e0c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:14.272940 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.272919 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:14.273068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.272945 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:14.273068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.272999 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:14.273068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.273013 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:14.273068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.273028 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3822c1d-660b-4c73-9104-9bde5105e0c7-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:14.274253 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.274227 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-token" pod "a3822c1d-660b-4c73-9104-9bde5105e0c7" (UID: "a3822c1d-660b-4c73-9104-9bde5105e0c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:14.274253 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.274243 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-podinfo" pod "a3822c1d-660b-4c73-9104-9bde5105e0c7" (UID: "a3822c1d-660b-4c73-9104-9bde5105e0c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:01:14.274388 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.274334 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-envoy" pod "a3822c1d-660b-4c73-9104-9bde5105e0c7" (UID: "a3822c1d-660b-4c73-9104-9bde5105e0c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:14.274431 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.274397 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3822c1d-660b-4c73-9104-9bde5105e0c7-kube-api-access-xk95s" pod "a3822c1d-660b-4c73-9104-9bde5105e0c7" (UID: "a3822c1d-660b-4c73-9104-9bde5105e0c7"). InnerVolumeSpecName "kube-api-access-xk95s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:14.375374 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.375341 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:14.375374 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.375365 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:14.375374 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.375375 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk95s\" (UniqueName: \"kubernetes.io/projected/a3822c1d-660b-4c73-9104-9bde5105e0c7-kube-api-access-xk95s\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:14.375374 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.375385 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3822c1d-660b-4c73-9104-9bde5105e0c7-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:14.745457 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.745413 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" event={"ID":"5700c62a-902c-4366-8a4e-8d8ca71b3eef","Type":"ContainerStarted","Data":"ec5a433d80eaf44d6b9867c361bb0a735ea3738653bf41481a2b457bbe3cfbd8"} Nov 02 18:01:14.745655 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.745484 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" event={"ID":"5700c62a-902c-4366-8a4e-8d8ca71b3eef","Type":"ContainerStarted","Data":"ee0cbbd9386cf14c230fa0caad9a5ab02d7b2975c9a7a73fd7f1c0437b35a43c"} Nov 02 18:01:14.746821 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.746765 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" Nov 02 18:01:14.748893 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.748866 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" event={"ID":"c2b8094f-68c0-461b-b5c9-c6a431267bf2","Type":"ContainerStarted","Data":"2b5fb0f3dafc4c75c0bdf1f8b100567d3d389a28e16689f87424ec3413670cc8"} Nov 02 18:01:14.749086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.748925 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" event={"ID":"c2b8094f-68c0-461b-b5c9-c6a431267bf2","Type":"ContainerStarted","Data":"6be341bf8a03f0d607041a1c8e1fdb3ce0751f781b381927a3183a25b7ed853b"} Nov 02 18:01:14.749086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.748964 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" Nov 02 18:01:14.750445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.750421 5549 generic.go:356] "Generic (PLEG): container finished" podID="a3822c1d-660b-4c73-9104-9bde5105e0c7" containerID="c9fe04711861759b38d73fb23296027177dfa6a449654b1330e47f5d0c49a7f2" exitCode=0 Nov 02 18:01:14.750576 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.750466 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" Nov 02 18:01:14.750576 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.750510 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" event={"ID":"a3822c1d-660b-4c73-9104-9bde5105e0c7","Type":"ContainerDied","Data":"c9fe04711861759b38d73fb23296027177dfa6a449654b1330e47f5d0c49a7f2"} Nov 02 18:01:14.750576 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.750557 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb" event={"ID":"a3822c1d-660b-4c73-9104-9bde5105e0c7","Type":"ContainerDied","Data":"28ed7723c2498b1c21c966605ae2de7714f58b484b6403a478abc25e01b03538"} Nov 02 18:01:14.750753 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.750587 5549 scope.go:122] "RemoveContainer" containerID="c9fe04711861759b38d73fb23296027177dfa6a449654b1330e47f5d0c49a7f2" Nov 02 18:01:14.753734 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.752824 5549 generic.go:356] "Generic (PLEG): container finished" podID="034a5399-ee48-4bfb-885c-0bcfd316985e" containerID="25f459a452a6a9c3c43cc2c233924826015ec6317989ce4df6ebf40fd9a58443" exitCode=0 Nov 02 18:01:14.753734 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.752877 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" event={"ID":"034a5399-ee48-4bfb-885c-0bcfd316985e","Type":"ContainerDied","Data":"25f459a452a6a9c3c43cc2c233924826015ec6317989ce4df6ebf40fd9a58443"} Nov 02 18:01:14.765271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.765242 5549 scope.go:122] "RemoveContainer" containerID="c9fe04711861759b38d73fb23296027177dfa6a449654b1330e47f5d0c49a7f2" Nov 02 18:01:14.765595 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:01:14.765567 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9fe04711861759b38d73fb23296027177dfa6a449654b1330e47f5d0c49a7f2\": container with ID starting with c9fe04711861759b38d73fb23296027177dfa6a449654b1330e47f5d0c49a7f2 not found: ID does not exist" containerID="c9fe04711861759b38d73fb23296027177dfa6a449654b1330e47f5d0c49a7f2" Nov 02 18:01:14.765725 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.765605 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9fe04711861759b38d73fb23296027177dfa6a449654b1330e47f5d0c49a7f2"} err="failed to get container status \"c9fe04711861759b38d73fb23296027177dfa6a449654b1330e47f5d0c49a7f2\": rpc error: code = NotFound desc = could not find container \"c9fe04711861759b38d73fb23296027177dfa6a449654b1330e47f5d0c49a7f2\": container with ID starting with c9fe04711861759b38d73fb23296027177dfa6a449654b1330e47f5d0c49a7f2 not found: ID does not exist" Nov 02 18:01:14.783351 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.783303 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" podStartSLOduration=2.783289385 podStartE2EDuration="2.783289385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:14.779048912 +0000 UTC m=+1646.567554442" watchObservedRunningTime="2026-11-02 18:01:14.783289385 +0000 UTC m=+1646.571794912" Nov 02 18:01:14.811710 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.811672 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb"] Nov 02 18:01:14.820270 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.820239 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-gc2vb"] Nov 02 18:01:14.828293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.828272 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 18:01:14.851037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.850984 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" podStartSLOduration=2.850969895 podStartE2EDuration="2.850969895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:14.849178934 +0000 UTC m=+1646.637684463" watchObservedRunningTime="2026-11-02 18:01:14.850969895 +0000 UTC m=+1646.639475422" Nov 02 18:01:14.872905 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.872860 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:14.883083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.883062 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/034a5399-ee48-4bfb-885c-0bcfd316985e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/034a5399-ee48-4bfb-885c-0bcfd316985e-istiod-ca-cert\") pod \"034a5399-ee48-4bfb-885c-0bcfd316985e\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " Nov 02 18:01:14.883202 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.883106 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-data\") pod \"034a5399-ee48-4bfb-885c-0bcfd316985e\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " Nov 02 18:01:14.883202 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.883154 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-podinfo\") pod \"034a5399-ee48-4bfb-885c-0bcfd316985e\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " Nov 02 18:01:14.883319 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.883250 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-token\" (UniqueName: \"kubernetes.io/projected/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-token\") pod \"034a5399-ee48-4bfb-885c-0bcfd316985e\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " Nov 02 18:01:14.883319 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.883299 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-workload-socket\") pod \"034a5399-ee48-4bfb-885c-0bcfd316985e\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " Nov 02 18:01:14.883451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.883339 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/034a5399-ee48-4bfb-885c-0bcfd316985e-kube-api-access-4sw7z\" (UniqueName: \"kubernetes.io/projected/034a5399-ee48-4bfb-885c-0bcfd316985e-kube-api-access-4sw7z\") pod \"034a5399-ee48-4bfb-885c-0bcfd316985e\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " Nov 02 18:01:14.883510 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.883444 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-data" pod "034a5399-ee48-4bfb-885c-0bcfd316985e" (UID: "034a5399-ee48-4bfb-885c-0bcfd316985e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:14.883510 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.883484 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-credential-socket\") pod \"034a5399-ee48-4bfb-885c-0bcfd316985e\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " Nov 02 18:01:14.883622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.883526 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-workload-certs\") pod \"034a5399-ee48-4bfb-885c-0bcfd316985e\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " Nov 02 18:01:14.883622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.883579 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-workload-socket" pod "034a5399-ee48-4bfb-885c-0bcfd316985e" (UID: "034a5399-ee48-4bfb-885c-0bcfd316985e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:14.883622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.883494 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/034a5399-ee48-4bfb-885c-0bcfd316985e-istiod-ca-cert" pod "034a5399-ee48-4bfb-885c-0bcfd316985e" (UID: "034a5399-ee48-4bfb-885c-0bcfd316985e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:01:14.883622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.883597 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-envoy\") pod \"034a5399-ee48-4bfb-885c-0bcfd316985e\" (UID: \"034a5399-ee48-4bfb-885c-0bcfd316985e\") " Nov 02 18:01:14.883842 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.883752 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-credential-socket" pod "034a5399-ee48-4bfb-885c-0bcfd316985e" (UID: "034a5399-ee48-4bfb-885c-0bcfd316985e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:14.883842 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.883824 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-workload-certs" pod "034a5399-ee48-4bfb-885c-0bcfd316985e" (UID: "034a5399-ee48-4bfb-885c-0bcfd316985e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:14.884515 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.884488 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/034a5399-ee48-4bfb-885c-0bcfd316985e-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:14.884634 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.884520 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:14.884634 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.884539 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:14.884634 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.884554 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:14.884634 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.884569 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:14.891992 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.885865 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-podinfo" pod "034a5399-ee48-4bfb-885c-0bcfd316985e" (UID: "034a5399-ee48-4bfb-885c-0bcfd316985e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:01:14.891992 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.885872 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-envoy" pod "034a5399-ee48-4bfb-885c-0bcfd316985e" (UID: "034a5399-ee48-4bfb-885c-0bcfd316985e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:14.891992 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.885761 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-token" pod "034a5399-ee48-4bfb-885c-0bcfd316985e" (UID: "034a5399-ee48-4bfb-885c-0bcfd316985e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:14.891992 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.885933 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/034a5399-ee48-4bfb-885c-0bcfd316985e-kube-api-access-4sw7z" pod "034a5399-ee48-4bfb-885c-0bcfd316985e" (UID: "034a5399-ee48-4bfb-885c-0bcfd316985e"). InnerVolumeSpecName "kube-api-access-4sw7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:14.986094 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.986059 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:14.986094 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.986090 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:14.986094 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.986101 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sw7z\" (UniqueName: \"kubernetes.io/projected/034a5399-ee48-4bfb-885c-0bcfd316985e-kube-api-access-4sw7z\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:14.986354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:14.986110 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/034a5399-ee48-4bfb-885c-0bcfd316985e-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:15.630976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:15.619026 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:15.764301 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:15.764273 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" Nov 02 18:01:15.764475 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:15.764315 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh" event={"ID":"034a5399-ee48-4bfb-885c-0bcfd316985e","Type":"ContainerDied","Data":"8f60f99aca8d8f34ab7f6179f99a3fde7c25ed5c4e113361477775b1959d1483"} Nov 02 18:01:15.764475 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:15.764385 5549 scope.go:122] "RemoveContainer" containerID="25f459a452a6a9c3c43cc2c233924826015ec6317989ce4df6ebf40fd9a58443" Nov 02 18:01:15.791616 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:15.791574 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh"] Nov 02 18:01:15.795322 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:15.795298 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-tcxlh"] Nov 02 18:01:16.322906 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.322875 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="034a5399-ee48-4bfb-885c-0bcfd316985e" path="/var/lib/kubelet/pods/034a5399-ee48-4bfb-885c-0bcfd316985e/volumes" Nov 02 18:01:16.323502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.323486 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3822c1d-660b-4c73-9104-9bde5105e0c7" path="/var/lib/kubelet/pods/a3822c1d-660b-4c73-9104-9bde5105e0c7/volumes" Nov 02 18:01:16.323871 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.323853 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m"] Nov 02 18:01:16.324805 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.324784 5549 cpu_manager.go:414] "containerMap: removing container" podUID="034a5399-ee48-4bfb-885c-0bcfd316985e" containerName="istio-proxy" Nov 02 18:01:16.324805 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.324803 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="034a5399-ee48-4bfb-885c-0bcfd316985e" containerName="istio-proxy" Nov 02 18:01:16.325056 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.324823 5549 cpu_manager.go:414] "containerMap: removing container" podUID="a3822c1d-660b-4c73-9104-9bde5105e0c7" containerName="istio-proxy" Nov 02 18:01:16.325056 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.324829 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3822c1d-660b-4c73-9104-9bde5105e0c7" containerName="istio-proxy" Nov 02 18:01:16.325286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.324947 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="034a5399-ee48-4bfb-885c-0bcfd316985e" containerName="istio-proxy" Nov 02 18:01:16.325324 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.325296 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3822c1d-660b-4c73-9104-9bde5105e0c7" containerName="istio-proxy" Nov 02 18:01:16.330851 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.330831 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:01:16.338168 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.338142 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m"] Nov 02 18:01:16.338381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.338364 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-42d786-serving-cert\"" Nov 02 18:01:16.414531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.414496 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a5568c-0ad3-4713-92ab-1730ef3e0607-maas-api-tls\") pod \"maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m\" (UID: \"07a5568c-0ad3-4713-92ab-1730ef3e0607\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:01:16.414682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.414567 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a5568c-0ad3-4713-92ab-1730ef3e0607-metrics-certs\") pod \"maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m\" (UID: \"07a5568c-0ad3-4713-92ab-1730ef3e0607\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:01:16.414743 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.414696 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a5568c-0ad3-4713-92ab-1730ef3e0607-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m\" (UID: \"07a5568c-0ad3-4713-92ab-1730ef3e0607\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:01:16.414799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.414782 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhcgz\" (UniqueName: \"kubernetes.io/projected/07a5568c-0ad3-4713-92ab-1730ef3e0607-kube-api-access-nhcgz\") pod \"maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m\" (UID: \"07a5568c-0ad3-4713-92ab-1730ef3e0607\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:01:16.514646 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.514571 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:16.542975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.542752 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a5568c-0ad3-4713-92ab-1730ef3e0607-maas-api-tls\") pod \"maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m\" (UID: \"07a5568c-0ad3-4713-92ab-1730ef3e0607\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:01:16.542975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.542854 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a5568c-0ad3-4713-92ab-1730ef3e0607-metrics-certs\") pod \"maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m\" (UID: \"07a5568c-0ad3-4713-92ab-1730ef3e0607\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:01:16.542975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.542917 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a5568c-0ad3-4713-92ab-1730ef3e0607-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m\" (UID: \"07a5568c-0ad3-4713-92ab-1730ef3e0607\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:01:16.543323 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.543048 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhcgz\" (UniqueName: \"kubernetes.io/projected/07a5568c-0ad3-4713-92ab-1730ef3e0607-kube-api-access-nhcgz\") pod \"maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m\" (UID: \"07a5568c-0ad3-4713-92ab-1730ef3e0607\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:01:16.543645 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:01:16.543615 5549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-serving-cert: secret "maas-api-e2e-extmod-a-42d786-serving-cert" not found Nov 02 18:01:16.544282 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.544254 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a5568c-0ad3-4713-92ab-1730ef3e0607-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m\" (UID: \"07a5568c-0ad3-4713-92ab-1730ef3e0607\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:01:16.544476 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:01:16.544454 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a5568c-0ad3-4713-92ab-1730ef3e0607-maas-api-tls podName:07a5568c-0ad3-4713-92ab-1730ef3e0607 nodeName:}" failed. No retries permitted until 2026-11-02 18:01:17.044430056 +0000 UTC m=+1648.832935587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07a5568c-0ad3-4713-92ab-1730ef3e0607-maas-api-tls") pod "maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" (UID: "07a5568c-0ad3-4713-92ab-1730ef3e0607") : secret "maas-api-e2e-extmod-a-42d786-serving-cert" not found Nov 02 18:01:16.573667 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.573516 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhcgz\" (UniqueName: \"kubernetes.io/projected/07a5568c-0ad3-4713-92ab-1730ef3e0607-kube-api-access-nhcgz\") pod \"maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m\" (UID: \"07a5568c-0ad3-4713-92ab-1730ef3e0607\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:01:16.600075 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.600046 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a5568c-0ad3-4713-92ab-1730ef3e0607-metrics-certs\") pod \"maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m\" (UID: \"07a5568c-0ad3-4713-92ab-1730ef3e0607\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:01:16.663151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.663107 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:16.863683 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:16.863587 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:17.058978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:17.056467 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a5568c-0ad3-4713-92ab-1730ef3e0607-maas-api-tls\") pod \"maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m\" (UID: \"07a5568c-0ad3-4713-92ab-1730ef3e0607\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:01:17.077980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:17.062209 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a5568c-0ad3-4713-92ab-1730ef3e0607-maas-api-tls\") pod \"maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m\" (UID: \"07a5568c-0ad3-4713-92ab-1730ef3e0607\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:01:17.229982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:17.228251 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:17.251832 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:17.251009 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:01:17.438986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:17.430699 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:17.638110 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:01:17.629301 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:01:17.848481 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:17.848450 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m"] Nov 02 18:01:17.851030 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:01:17.851001 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a5568c_0ad3_4713_92ab_1730ef3e0607.slice/crio-b7f2c4b7fad846dd4fe12735078cb6e5ce4b86e25df04d03753f21925bb89c91 WatchSource:0}: Error finding container b7f2c4b7fad846dd4fe12735078cb6e5ce4b86e25df04d03753f21925bb89c91: Status 404 returned error can't find the container with id b7f2c4b7fad846dd4fe12735078cb6e5ce4b86e25df04d03753f21925bb89c91 Nov 02 18:01:17.852750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:17.852693 5549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:01:18.001583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:17.999625 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:18.451077 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.451048 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:18.481030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.480999 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck"] Nov 02 18:01:18.486598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.486574 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck" Nov 02 18:01:18.487023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.486997 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk"] Nov 02 18:01:18.494089 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.494065 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk" Nov 02 18:01:18.494810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.494787 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-4ea582-dockercfg-b9rzf\"" Nov 02 18:01:18.494810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.494809 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-4ea582\"" Nov 02 18:01:18.495889 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.495868 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck"] Nov 02 18:01:18.501253 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.501230 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk"] Nov 02 18:01:18.616503 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.610902 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:18.658761 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.658661 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hswkd\" (UniqueName: \"kubernetes.io/projected/b077a6d9-5d78-4d91-8223-89244863b8e5-kube-api-access-hswkd\") pod \"payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk\" (UID: \"b077a6d9-5d78-4d91-8223-89244863b8e5\") " pod="openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk" Nov 02 18:01:18.659074 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.659043 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b077a6d9-5d78-4d91-8223-89244863b8e5-plugins-config-volume\") pod \"payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk\" (UID: \"b077a6d9-5d78-4d91-8223-89244863b8e5\") " pod="openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk" Nov 02 18:01:18.659262 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.659243 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7639dacc-94de-425d-8ba4-3fa1baa67fb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck\" (UID: \"7639dacc-94de-425d-8ba4-3fa1baa67fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck" Nov 02 18:01:18.659607 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.659588 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb9dd\" (UniqueName: \"kubernetes.io/projected/7639dacc-94de-425d-8ba4-3fa1baa67fb1-kube-api-access-vb9dd\") pod \"payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck\" (UID: \"7639dacc-94de-425d-8ba4-3fa1baa67fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck" Nov 02 18:01:18.769019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.766174 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb9dd\" (UniqueName: \"kubernetes.io/projected/7639dacc-94de-425d-8ba4-3fa1baa67fb1-kube-api-access-vb9dd\") pod \"payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck\" (UID: \"7639dacc-94de-425d-8ba4-3fa1baa67fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck" Nov 02 18:01:18.769019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.766442 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hswkd\" (UniqueName: \"kubernetes.io/projected/b077a6d9-5d78-4d91-8223-89244863b8e5-kube-api-access-hswkd\") pod \"payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk\" (UID: \"b077a6d9-5d78-4d91-8223-89244863b8e5\") " pod="openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk" Nov 02 18:01:18.769019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.766484 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b077a6d9-5d78-4d91-8223-89244863b8e5-plugins-config-volume\") pod \"payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk\" (UID: \"b077a6d9-5d78-4d91-8223-89244863b8e5\") " pod="openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk" Nov 02 18:01:18.769019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.766512 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7639dacc-94de-425d-8ba4-3fa1baa67fb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck\" (UID: \"7639dacc-94de-425d-8ba4-3fa1baa67fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck" Nov 02 18:01:18.769019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.767530 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7639dacc-94de-425d-8ba4-3fa1baa67fb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck\" (UID: \"7639dacc-94de-425d-8ba4-3fa1baa67fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck" Nov 02 18:01:18.769019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.768288 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b077a6d9-5d78-4d91-8223-89244863b8e5-plugins-config-volume\") pod \"payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk\" (UID: \"b077a6d9-5d78-4d91-8223-89244863b8e5\") " pod="openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk" Nov 02 18:01:18.789846 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.785384 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb9dd\" (UniqueName: \"kubernetes.io/projected/7639dacc-94de-425d-8ba4-3fa1baa67fb1-kube-api-access-vb9dd\") pod \"payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck\" (UID: \"7639dacc-94de-425d-8ba4-3fa1baa67fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck" Nov 02 18:01:18.809974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.805677 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hswkd\" (UniqueName: \"kubernetes.io/projected/b077a6d9-5d78-4d91-8223-89244863b8e5-kube-api-access-hswkd\") pod \"payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk\" (UID: \"b077a6d9-5d78-4d91-8223-89244863b8e5\") " pod="openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk" Nov 02 18:01:18.829448 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.829410 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck" Nov 02 18:01:18.849660 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.844699 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk" Nov 02 18:01:18.871222 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.871185 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" event={"ID":"07a5568c-0ad3-4713-92ab-1730ef3e0607","Type":"ContainerStarted","Data":"0717b3d52f98f93f88b0183801d55d19284ff5a9dcf62b2ce9e4f869fe42400d"} Nov 02 18:01:18.871472 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.871449 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" event={"ID":"07a5568c-0ad3-4713-92ab-1730ef3e0607","Type":"ContainerStarted","Data":"b7f2c4b7fad846dd4fe12735078cb6e5ce4b86e25df04d03753f21925bb89c91"} Nov 02 18:01:18.873371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.872859 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:01:18.895522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:18.895445 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" podStartSLOduration=2.359051833 podStartE2EDuration="2.895424921s" totalImagesPullingTime="536.373088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:18.89209452 +0000 UTC m=+1650.680600048" watchObservedRunningTime="2026-11-02 18:01:18.895424921 +0000 UTC m=+1650.683930452" Nov 02 18:01:19.048997 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:19.039871 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:19.302243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:19.302212 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck"] Nov 02 18:01:19.302499 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:01:19.302476 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7639dacc_94de_425d_8ba4_3fa1baa67fb1.slice/crio-813e19c99f372f2741c246e91383546325ba63faee572bf10e57bdd757b922f3 WatchSource:0}: Error finding container 813e19c99f372f2741c246e91383546325ba63faee572bf10e57bdd757b922f3: Status 404 returned error can't find the container with id 813e19c99f372f2741c246e91383546325ba63faee572bf10e57bdd757b922f3 Nov 02 18:01:19.322659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:19.322631 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk"] Nov 02 18:01:19.324929 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:01:19.324901 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb077a6d9_5d78_4d91_8223_89244863b8e5.slice/crio-cb356b51dd9bc80d33182e4eefc722e15d6170661c745c79f705b5b306fc2293 WatchSource:0}: Error finding container cb356b51dd9bc80d33182e4eefc722e15d6170661c745c79f705b5b306fc2293: Status 404 returned error can't find the container with id cb356b51dd9bc80d33182e4eefc722e15d6170661c745c79f705b5b306fc2293 Nov 02 18:01:19.781072 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:19.780899 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:19.881224 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:19.881187 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk" event={"ID":"b077a6d9-5d78-4d91-8223-89244863b8e5","Type":"ContainerStarted","Data":"9f4d9a4eeb1cf97c9c23f0142ee94f9d02a7c1d74caedb3cfc737e79e86f3743"} Nov 02 18:01:19.881678 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:19.881290 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk" Nov 02 18:01:19.881678 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:19.881319 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk" event={"ID":"b077a6d9-5d78-4d91-8223-89244863b8e5","Type":"ContainerStarted","Data":"cb356b51dd9bc80d33182e4eefc722e15d6170661c745c79f705b5b306fc2293"} Nov 02 18:01:19.883395 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:19.883358 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck" event={"ID":"7639dacc-94de-425d-8ba4-3fa1baa67fb1","Type":"ContainerStarted","Data":"614b3669c8b417be4d96d3bbbabe9299cb281d105ad661882990db086e9d0de7"} Nov 02 18:01:19.883530 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:19.883423 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck" event={"ID":"7639dacc-94de-425d-8ba4-3fa1baa67fb1","Type":"ContainerStarted","Data":"813e19c99f372f2741c246e91383546325ba63faee572bf10e57bdd757b922f3"} Nov 02 18:01:19.883803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:19.883785 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck" Nov 02 18:01:19.902312 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:19.902233 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk" podStartSLOduration=1.902217699 podStartE2EDuration="1.902217699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:19.898028471 +0000 UTC m=+1651.686534000" watchObservedRunningTime="2026-11-02 18:01:19.902217699 +0000 UTC m=+1651.690723228" Nov 02 18:01:19.921786 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:19.921733 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck" podStartSLOduration=1.921715061 podStartE2EDuration="1.921715061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:19.918212788 +0000 UTC m=+1651.706718314" watchObservedRunningTime="2026-11-02 18:01:19.921715061 +0000 UTC m=+1651.710220592" Nov 02 18:01:20.531999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:20.530082 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:21.228009 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:21.219554 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:21.503980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:21.502482 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:22.250831 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:22.250666 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:22.308499 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:22.308464 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds"] Nov 02 18:01:22.411980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:22.411908 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:23.048978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.042258 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:23.201204 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.201167 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:23.470927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.470896 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g"] Nov 02 18:01:23.475984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.475939 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.483117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.483094 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a8e574-openshift-default-dockercfg-bpskn\"" Nov 02 18:01:23.495913 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.495885 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g"] Nov 02 18:01:23.513610 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.513570 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7669f8e2-f625-4276-9726-d260732ffa7e-istiod-ca-cert\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.513799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.513740 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-istio-data\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.513862 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.513800 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqgbh\" (UniqueName: \"kubernetes.io/projected/7669f8e2-f625-4276-9726-d260732ffa7e-kube-api-access-gqgbh\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.513914 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.513886 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-workload-certs\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.514097 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.514077 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-credential-socket\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.514143 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.514110 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7669f8e2-f625-4276-9726-d260732ffa7e-istio-token\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.514182 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.514149 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-istio-envoy\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.514231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.514217 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-workload-socket\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.514276 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.514262 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7669f8e2-f625-4276-9726-d260732ffa7e-istio-podinfo\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.618030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.617228 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:23.649969 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.649386 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7669f8e2-f625-4276-9726-d260732ffa7e-istiod-ca-cert\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.649969 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.649535 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-istio-data\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.649969 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.649598 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgbh\" (UniqueName: \"kubernetes.io/projected/7669f8e2-f625-4276-9726-d260732ffa7e-kube-api-access-gqgbh\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.649969 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.649632 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-workload-certs\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.649969 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.649839 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-credential-socket\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.649969 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.649866 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7669f8e2-f625-4276-9726-d260732ffa7e-istio-token\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.649969 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.649913 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-istio-envoy\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.650479 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.650011 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-workload-socket\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.650479 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.650059 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7669f8e2-f625-4276-9726-d260732ffa7e-istio-podinfo\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.683976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.659999 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-workload-certs\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.683976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.674335 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-istio-data\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.683976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.675131 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-credential-socket\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.683976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.678452 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7669f8e2-f625-4276-9726-d260732ffa7e-istiod-ca-cert\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.683976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.678706 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-workload-socket\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.748022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.736502 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-istio-envoy\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.748022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.745725 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqgbh\" (UniqueName: \"kubernetes.io/projected/7669f8e2-f625-4276-9726-d260732ffa7e-kube-api-access-gqgbh\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.759978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.749218 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7669f8e2-f625-4276-9726-d260732ffa7e-istio-token\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.768050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.765511 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7669f8e2-f625-4276-9726-d260732ffa7e-istio-podinfo\") pod \"e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:23.804408 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:23.803460 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:24.116361 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:24.116308 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:24.234879 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:01:24.234844 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7669f8e2_f625_4276_9726_d260732ffa7e.slice/crio-fda3921074babf1f88c494f7399cbf8bdf026a3c524aa8497168427b30286a3c WatchSource:0}: Error finding container fda3921074babf1f88c494f7399cbf8bdf026a3c524aa8497168427b30286a3c: Status 404 returned error can't find the container with id fda3921074babf1f88c494f7399cbf8bdf026a3c524aa8497168427b30286a3c Nov 02 18:01:24.235015 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:24.234874 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g"] Nov 02 18:01:24.237789 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:24.237750 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:01:24.237863 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:24.237815 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:01:24.237863 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:24.237847 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:01:24.338146 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:24.334613 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:24.958012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:24.957977 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" event={"ID":"7669f8e2-f625-4276-9726-d260732ffa7e","Type":"ContainerStarted","Data":"a9b74124eb89d95ab2ee6f2cc22707845716c1f2e8ab7abad140ea0ccf9bc514"} Nov 02 18:01:24.958446 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:24.958038 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" event={"ID":"7669f8e2-f625-4276-9726-d260732ffa7e","Type":"ContainerStarted","Data":"fda3921074babf1f88c494f7399cbf8bdf026a3c524aa8497168427b30286a3c"} Nov 02 18:01:24.990516 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:24.990464 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" podStartSLOduration=1.990446288 podStartE2EDuration="1.990446288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:24.985356876 +0000 UTC m=+1656.773862404" watchObservedRunningTime="2026-11-02 18:01:24.990446288 +0000 UTC m=+1656.778951823" Nov 02 18:01:25.146981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:25.146383 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:25.768183 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:25.768151 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" Nov 02 18:01:25.804521 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:25.804491 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:25.810535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:25.810506 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:25.896535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:25.896505 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:01:25.921081 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:25.921049 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:25.969692 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:25.969656 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:25.970870 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:25.970843 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:01:25.990575 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:25.990540 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" podUID="8eecd66e-2f57-4935-8eb4-000f22375a2a" containerName="main" containerID="cri-o://479400edc13e2eaa4b309a5c6e457808f7ca76efa9c09e5142d8a313598cac1f" gracePeriod=45 Nov 02 18:01:26.087807 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.087771 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:26.524980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.522218 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 18:01:26.602263 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.602170 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-model-cache\") pod \"8eecd66e-2f57-4935-8eb4-000f22375a2a\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " Nov 02 18:01:26.602447 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.602299 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-dshm\") pod \"8eecd66e-2f57-4935-8eb4-000f22375a2a\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " Nov 02 18:01:26.602523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.602464 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-model-cache" pod "8eecd66e-2f57-4935-8eb4-000f22375a2a" (UID: "8eecd66e-2f57-4935-8eb4-000f22375a2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:26.602636 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.602612 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-home\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-home\") pod \"8eecd66e-2f57-4935-8eb4-000f22375a2a\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " Nov 02 18:01:26.602703 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.602676 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-tmp-dir\") pod \"8eecd66e-2f57-4935-8eb4-000f22375a2a\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " Nov 02 18:01:26.602759 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.602720 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eecd66e-2f57-4935-8eb4-000f22375a2a-kube-api-access-ld97v\" (UniqueName: \"kubernetes.io/projected/8eecd66e-2f57-4935-8eb4-000f22375a2a-kube-api-access-ld97v\") pod \"8eecd66e-2f57-4935-8eb4-000f22375a2a\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " Nov 02 18:01:26.602809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.602766 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eecd66e-2f57-4935-8eb4-000f22375a2a-tls-certs\" (UniqueName: \"kubernetes.io/secret/8eecd66e-2f57-4935-8eb4-000f22375a2a-tls-certs\") pod \"8eecd66e-2f57-4935-8eb4-000f22375a2a\" (UID: \"8eecd66e-2f57-4935-8eb4-000f22375a2a\") " Nov 02 18:01:26.603265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.603110 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-tmp-dir" pod "8eecd66e-2f57-4935-8eb4-000f22375a2a" (UID: "8eecd66e-2f57-4935-8eb4-000f22375a2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:26.603265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.603171 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-home" pod "8eecd66e-2f57-4935-8eb4-000f22375a2a" (UID: "8eecd66e-2f57-4935-8eb4-000f22375a2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:26.603767 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.603743 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:26.603876 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.603772 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:26.603876 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.603788 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:26.604892 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.604861 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-dshm" pod "8eecd66e-2f57-4935-8eb4-000f22375a2a" (UID: "8eecd66e-2f57-4935-8eb4-000f22375a2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:26.605588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.605555 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eecd66e-2f57-4935-8eb4-000f22375a2a-kube-api-access-ld97v" pod "8eecd66e-2f57-4935-8eb4-000f22375a2a" (UID: "8eecd66e-2f57-4935-8eb4-000f22375a2a"). InnerVolumeSpecName "kube-api-access-ld97v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:26.605698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.605603 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eecd66e-2f57-4935-8eb4-000f22375a2a-tls-certs" pod "8eecd66e-2f57-4935-8eb4-000f22375a2a" (UID: "8eecd66e-2f57-4935-8eb4-000f22375a2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:01:26.715036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.709766 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eecd66e-2f57-4935-8eb4-000f22375a2a-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:26.715036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.709809 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld97v\" (UniqueName: \"kubernetes.io/projected/8eecd66e-2f57-4935-8eb4-000f22375a2a-kube-api-access-ld97v\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:26.715036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.709828 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eecd66e-2f57-4935-8eb4-000f22375a2a-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:26.808518 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:26.808465 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" Nov 02 18:01:27.054992 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:27.054931 5549 generic.go:356] "Generic (PLEG): container finished" podID="8eecd66e-2f57-4935-8eb4-000f22375a2a" containerID="479400edc13e2eaa4b309a5c6e457808f7ca76efa9c09e5142d8a313598cac1f" exitCode=0 Nov 02 18:01:27.056642 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:27.056615 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" Nov 02 18:01:27.056782 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:27.056735 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" event={"ID":"8eecd66e-2f57-4935-8eb4-000f22375a2a","Type":"ContainerDied","Data":"479400edc13e2eaa4b309a5c6e457808f7ca76efa9c09e5142d8a313598cac1f"} Nov 02 18:01:27.056868 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:27.056788 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7" event={"ID":"8eecd66e-2f57-4935-8eb4-000f22375a2a","Type":"ContainerDied","Data":"d5c535fffc720ce87a9a4da2fe9bd9784d37b5c4ce70ded983e325593e7c0fb3"} Nov 02 18:01:27.056868 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:27.056824 5549 scope.go:122] "RemoveContainer" containerID="479400edc13e2eaa4b309a5c6e457808f7ca76efa9c09e5142d8a313598cac1f" Nov 02 18:01:27.082351 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:27.082330 5549 scope.go:122] "RemoveContainer" containerID="479400edc13e2eaa4b309a5c6e457808f7ca76efa9c09e5142d8a313598cac1f" Nov 02 18:01:27.082774 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:01:27.082748 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"479400edc13e2eaa4b309a5c6e457808f7ca76efa9c09e5142d8a313598cac1f\": container with ID starting with 479400edc13e2eaa4b309a5c6e457808f7ca76efa9c09e5142d8a313598cac1f not found: ID does not exist" containerID="479400edc13e2eaa4b309a5c6e457808f7ca76efa9c09e5142d8a313598cac1f" Nov 02 18:01:27.082893 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:27.082779 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479400edc13e2eaa4b309a5c6e457808f7ca76efa9c09e5142d8a313598cac1f"} err="failed to get container status \"479400edc13e2eaa4b309a5c6e457808f7ca76efa9c09e5142d8a313598cac1f\": rpc error: code = NotFound desc = could not find container \"479400edc13e2eaa4b309a5c6e457808f7ca76efa9c09e5142d8a313598cac1f\": container with ID starting with 479400edc13e2eaa4b309a5c6e457808f7ca76efa9c09e5142d8a313598cac1f not found: ID does not exist" Nov 02 18:01:27.093943 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:27.093913 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7"] Nov 02 18:01:27.097876 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:27.097847 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1e2281/auth-test-model-1e2281-kserve-6bb4954bb9-bmgc7"] Nov 02 18:01:27.248198 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:27.244755 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:27.404848 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:27.404760 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:27.574532 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:27.574476 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:28.164997 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:28.164167 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:28.321980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:28.321933 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eecd66e-2f57-4935-8eb4-000f22375a2a" path="/var/lib/kubelet/pods/8eecd66e-2f57-4935-8eb4-000f22375a2a/volumes" Nov 02 18:01:28.596178 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:28.596134 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:28.663571 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:28.663531 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx"] Nov 02 18:01:28.664512 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:28.664490 5549 cpu_manager.go:414] "containerMap: removing container" podUID="8eecd66e-2f57-4935-8eb4-000f22375a2a" containerName="main" Nov 02 18:01:28.664512 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:28.664512 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eecd66e-2f57-4935-8eb4-000f22375a2a" containerName="main" Nov 02 18:01:28.664751 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:28.664737 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eecd66e-2f57-4935-8eb4-000f22375a2a" containerName="main" Nov 02 18:01:28.670311 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:28.670286 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" Nov 02 18:01:28.678913 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:28.678884 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx"] Nov 02 18:01:28.723726 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:28.717326 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a8e574-dockercfg-dlm2d\"" Nov 02 18:01:28.829975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:28.829177 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a8e574\"" Nov 02 18:01:28.850041 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:28.837006 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:28.907000 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:28.893384 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq6gm\" (UniqueName: \"kubernetes.io/projected/766b693c-dd34-4892-9eeb-3528e08a3147-kube-api-access-rq6gm\") pod \"payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx\" (UID: \"766b693c-dd34-4892-9eeb-3528e08a3147\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" Nov 02 18:01:28.907000 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:28.893544 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766b693c-dd34-4892-9eeb-3528e08a3147-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx\" (UID: \"766b693c-dd34-4892-9eeb-3528e08a3147\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" Nov 02 18:01:28.999977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:28.999238 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq6gm\" (UniqueName: \"kubernetes.io/projected/766b693c-dd34-4892-9eeb-3528e08a3147-kube-api-access-rq6gm\") pod \"payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx\" (UID: \"766b693c-dd34-4892-9eeb-3528e08a3147\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" Nov 02 18:01:28.999977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:28.999440 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766b693c-dd34-4892-9eeb-3528e08a3147-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx\" (UID: \"766b693c-dd34-4892-9eeb-3528e08a3147\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" Nov 02 18:01:29.023977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:29.003610 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766b693c-dd34-4892-9eeb-3528e08a3147-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx\" (UID: \"766b693c-dd34-4892-9eeb-3528e08a3147\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" Nov 02 18:01:29.051040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:29.049675 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq6gm\" (UniqueName: \"kubernetes.io/projected/766b693c-dd34-4892-9eeb-3528e08a3147-kube-api-access-rq6gm\") pod \"payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx\" (UID: \"766b693c-dd34-4892-9eeb-3528e08a3147\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" Nov 02 18:01:29.201369 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:29.201335 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" Nov 02 18:01:29.428985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:29.421527 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:29.526774 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:29.526746 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx"] Nov 02 18:01:29.527597 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:01:29.527568 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766b693c_dd34_4892_9eeb_3528e08a3147.slice/crio-e5cf3b25308f2cf8df6b73f9b28a9fdca83ed97ebf7c9747d73e8f9f3c8a89ec WatchSource:0}: Error finding container e5cf3b25308f2cf8df6b73f9b28a9fdca83ed97ebf7c9747d73e8f9f3c8a89ec: Status 404 returned error can't find the container with id e5cf3b25308f2cf8df6b73f9b28a9fdca83ed97ebf7c9747d73e8f9f3c8a89ec Nov 02 18:01:30.119127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:30.119077 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" event={"ID":"766b693c-dd34-4892-9eeb-3528e08a3147","Type":"ContainerStarted","Data":"e4a7ba3036f8bebbbf8f6efae196aab43c58866bc33d4584b0c246ad16abb892"} Nov 02 18:01:30.119313 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:30.119194 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" event={"ID":"766b693c-dd34-4892-9eeb-3528e08a3147","Type":"ContainerStarted","Data":"e5cf3b25308f2cf8df6b73f9b28a9fdca83ed97ebf7c9747d73e8f9f3c8a89ec"} Nov 02 18:01:30.119812 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:30.119781 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" Nov 02 18:01:30.141270 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:30.141221 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" podStartSLOduration=2.141207221 podStartE2EDuration="2.141207221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:30.138469172 +0000 UTC m=+1661.926974761" watchObservedRunningTime="2026-11-02 18:01:30.141207221 +0000 UTC m=+1661.929712748" Nov 02 18:01:30.202832 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:30.202783 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:30.894518 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:30.894481 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk" Nov 02 18:01:30.894694 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:30.894570 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck" Nov 02 18:01:31.004006 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:31.003972 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:31.517425 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:31.517381 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh"] Nov 02 18:01:31.607974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:31.607591 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:32.034988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:32.020531 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:32.641976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:32.636991 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:33.061234 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.061199 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:33.327649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.327571 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j"] Nov 02 18:01:33.333571 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.333546 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" Nov 02 18:01:33.340414 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.340383 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j"] Nov 02 18:01:33.343612 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.343584 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-4ea582-serving-cert\"" Nov 02 18:01:33.409679 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.409639 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94ssg\" (UniqueName: \"kubernetes.io/projected/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-kube-api-access-94ssg\") pod \"maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j\" (UID: \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" Nov 02 18:01:33.410071 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.410022 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-openshift-service-ca\") pod \"maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j\" (UID: \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" Nov 02 18:01:33.410362 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.410340 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-metrics-certs\") pod \"maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j\" (UID: \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" Nov 02 18:01:33.410569 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.410550 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-maas-api-tls\") pod \"maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j\" (UID: \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" Nov 02 18:01:33.434669 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.434636 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh"] Nov 02 18:01:33.441422 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.441390 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.442825 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.442796 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:33.445770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.445735 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-807c08-openshift-default-dockercfg-bdd5k\"" Nov 02 18:01:33.459772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.459744 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh"] Nov 02 18:01:33.511885 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.511843 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-maas-api-tls\") pod \"maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j\" (UID: \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" Nov 02 18:01:33.512071 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.511937 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4xv\" (UniqueName: \"kubernetes.io/projected/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-kube-api-access-mp4xv\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.512121 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.512093 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istiod-ca-cert\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.512190 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.512176 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94ssg\" (UniqueName: \"kubernetes.io/projected/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-kube-api-access-94ssg\") pod \"maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j\" (UID: \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" Nov 02 18:01:33.512232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.512212 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-credential-socket\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.512268 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.512233 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-podinfo\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.512301 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.512278 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-openshift-service-ca\") pod \"maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j\" (UID: \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" Nov 02 18:01:33.512356 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.512341 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-workload-certs\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.512772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.512738 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-data\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.512925 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.512786 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-token\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.512925 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.512819 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-workload-socket\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.512925 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.512846 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-envoy\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.512925 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.512908 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-metrics-certs\") pod \"maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j\" (UID: \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" Nov 02 18:01:33.513194 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.513094 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-openshift-service-ca\") pod \"maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j\" (UID: \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" Nov 02 18:01:33.519014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.515457 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-maas-api-tls\") pod \"maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j\" (UID: \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" Nov 02 18:01:33.519014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.515628 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-metrics-certs\") pod \"maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j\" (UID: \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" Nov 02 18:01:33.526214 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.524008 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94ssg\" (UniqueName: \"kubernetes.io/projected/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-kube-api-access-94ssg\") pod \"maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j\" (UID: \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" Nov 02 18:01:33.593219 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.593137 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:33.615049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.615005 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-credential-socket\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.615205 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.615062 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-podinfo\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.615249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.615208 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-workload-certs\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.615300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.615273 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-data\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.615353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.615304 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-token\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.615353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.615335 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-workload-socket\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.615465 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.615359 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-envoy\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.615525 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.615486 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-credential-socket\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.615525 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.615518 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4xv\" (UniqueName: \"kubernetes.io/projected/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-kube-api-access-mp4xv\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.615642 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.615599 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istiod-ca-cert\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.615642 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.615598 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-workload-certs\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.615815 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.615774 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-workload-socket\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.615988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.615887 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-data\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.616383 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.616358 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istiod-ca-cert\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.617678 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.617657 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-envoy\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.617778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.617744 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-podinfo\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.626646 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.626619 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-token\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.626726 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.626711 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4xv\" (UniqueName: \"kubernetes.io/projected/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-kube-api-access-mp4xv\") pod \"e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.663902 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.663499 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" Nov 02 18:01:33.792981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.782437 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:33.803157 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:33.785440 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:34.447733 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:34.447696 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh"] Nov 02 18:01:34.449380 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:01:34.449350 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e4e1d9_7229_43d7_8e25_50d6dc6d1b23.slice/crio-7dd145f8c0c5150416d72679e898ac57fa0b56f64e29f96f2bb0d50aa80d4eb2 WatchSource:0}: Error finding container 7dd145f8c0c5150416d72679e898ac57fa0b56f64e29f96f2bb0d50aa80d4eb2: Status 404 returned error can't find the container with id 7dd145f8c0c5150416d72679e898ac57fa0b56f64e29f96f2bb0d50aa80d4eb2 Nov 02 18:01:34.461778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:34.461592 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:01:34.461778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:34.461726 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:01:34.461919 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:34.461792 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:01:34.467568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:34.467539 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j"] Nov 02 18:01:34.720737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:34.720696 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:35.186683 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:35.186645 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" event={"ID":"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9","Type":"ContainerStarted","Data":"5691e1470e5f546fc3848aef2a8e042e78374d05926a73bcb3cd63b4b7396e3d"} Nov 02 18:01:35.186855 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:35.186700 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" event={"ID":"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9","Type":"ContainerStarted","Data":"ee19324a4dafb53d2f792ce388922b53a807aa710a1cd17a9fb445ea2d38f122"} Nov 02 18:01:35.186855 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:35.186728 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" Nov 02 18:01:35.188423 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:35.188396 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" event={"ID":"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23","Type":"ContainerStarted","Data":"f4829e725c79e563c30f2e2294bd3d0ffc65d6cdb1e34e4bebd3c1da6cf10254"} Nov 02 18:01:35.188549 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:35.188442 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" event={"ID":"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23","Type":"ContainerStarted","Data":"7dd145f8c0c5150416d72679e898ac57fa0b56f64e29f96f2bb0d50aa80d4eb2"} Nov 02 18:01:35.218970 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:35.218908 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" podStartSLOduration=1.672481961 podStartE2EDuration="2.218893608s" totalImagesPullingTime="546.411647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:35.215366054 +0000 UTC m=+1667.003871582" watchObservedRunningTime="2026-11-02 18:01:35.218893608 +0000 UTC m=+1667.007399136" Nov 02 18:01:35.244850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:35.244791 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" podStartSLOduration=2.244778017 podStartE2EDuration="2.244778017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:35.243094483 +0000 UTC m=+1667.031600011" watchObservedRunningTime="2026-11-02 18:01:35.244778017 +0000 UTC m=+1667.033283619" Nov 02 18:01:35.792423 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:35.792391 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:35.822775 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:35.822736 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:36.087026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:36.086643 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:36.198288 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:36.198253 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:36.199140 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:36.199116 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:01:36.609965 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:36.609899 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:37.023759 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.023724 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:37.337603 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.337514 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" podUID="d16de1e2-a7a7-4046-90e8-85763b266321" containerName="main" containerID="cri-o://1a98c7d99b4d73b6b387f30afe43f487bd37713dd4be6d63b31186578ba3d08c" gracePeriod=45 Nov 02 18:01:37.477293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.477244 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" podUID="d16de1e2-a7a7-4046-90e8-85763b266321" containerName="main" probeResult="failure" output="Get \"https://10.133.0.188:8000/ready\": dial tcp 10.133.0.188:8000: connect: connection refused" Nov 02 18:01:37.542743 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.542705 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:37.659070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.659041 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 18:01:37.685134 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.685101 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:37.782903 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.782870 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16de1e2-a7a7-4046-90e8-85763b266321-kube-api-access-qh996\" (UniqueName: \"kubernetes.io/projected/d16de1e2-a7a7-4046-90e8-85763b266321-kube-api-access-qh996\") pod \"d16de1e2-a7a7-4046-90e8-85763b266321\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " Nov 02 18:01:37.783112 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.783094 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-dshm\") pod \"d16de1e2-a7a7-4046-90e8-85763b266321\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " Nov 02 18:01:37.783212 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.783196 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-model-cache\") pod \"d16de1e2-a7a7-4046-90e8-85763b266321\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " Nov 02 18:01:37.783256 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.783237 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-tmp-dir\") pod \"d16de1e2-a7a7-4046-90e8-85763b266321\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " Nov 02 18:01:37.783451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.783428 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-home\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-home\") pod \"d16de1e2-a7a7-4046-90e8-85763b266321\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " Nov 02 18:01:37.783553 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.783543 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d16de1e2-a7a7-4046-90e8-85763b266321-tls-certs\" (UniqueName: \"kubernetes.io/secret/d16de1e2-a7a7-4046-90e8-85763b266321-tls-certs\") pod \"d16de1e2-a7a7-4046-90e8-85763b266321\" (UID: \"d16de1e2-a7a7-4046-90e8-85763b266321\") " Nov 02 18:01:37.783686 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.783634 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-model-cache" pod "d16de1e2-a7a7-4046-90e8-85763b266321" (UID: "d16de1e2-a7a7-4046-90e8-85763b266321"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:37.783827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.783785 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-home" pod "d16de1e2-a7a7-4046-90e8-85763b266321" (UID: "d16de1e2-a7a7-4046-90e8-85763b266321"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:37.784034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.784012 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-tmp-dir" pod "d16de1e2-a7a7-4046-90e8-85763b266321" (UID: "d16de1e2-a7a7-4046-90e8-85763b266321"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:37.784387 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.784361 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:37.784490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.784391 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:37.784490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.784407 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:37.785776 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.785746 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16de1e2-a7a7-4046-90e8-85763b266321-kube-api-access-qh996" pod "d16de1e2-a7a7-4046-90e8-85763b266321" (UID: "d16de1e2-a7a7-4046-90e8-85763b266321"). InnerVolumeSpecName "kube-api-access-qh996". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:37.785876 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.785851 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d16de1e2-a7a7-4046-90e8-85763b266321-tls-certs" pod "d16de1e2-a7a7-4046-90e8-85763b266321" (UID: "d16de1e2-a7a7-4046-90e8-85763b266321"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:01:37.786062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.786036 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-dshm" pod "d16de1e2-a7a7-4046-90e8-85763b266321" (UID: "d16de1e2-a7a7-4046-90e8-85763b266321"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:37.848663 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.848631 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:37.885885 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.885802 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d16de1e2-a7a7-4046-90e8-85763b266321-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:37.885885 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.885835 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh996\" (UniqueName: \"kubernetes.io/projected/d16de1e2-a7a7-4046-90e8-85763b266321-kube-api-access-qh996\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:37.885885 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:37.885846 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d16de1e2-a7a7-4046-90e8-85763b266321-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:38.013978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.012902 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:38.218926 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.218891 5549 generic.go:356] "Generic (PLEG): container finished" podID="d16de1e2-a7a7-4046-90e8-85763b266321" containerID="1a98c7d99b4d73b6b387f30afe43f487bd37713dd4be6d63b31186578ba3d08c" exitCode=0 Nov 02 18:01:38.219470 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.218973 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" Nov 02 18:01:38.219470 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.218986 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" event={"ID":"d16de1e2-a7a7-4046-90e8-85763b266321","Type":"ContainerDied","Data":"1a98c7d99b4d73b6b387f30afe43f487bd37713dd4be6d63b31186578ba3d08c"} Nov 02 18:01:38.219470 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.219028 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds" event={"ID":"d16de1e2-a7a7-4046-90e8-85763b266321","Type":"ContainerDied","Data":"9d5a35bebcd222b8724c6c564ef0d65beee72d9618c67d42a327386d98f7cf17"} Nov 02 18:01:38.219470 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.219049 5549 scope.go:122] "RemoveContainer" containerID="1a98c7d99b4d73b6b387f30afe43f487bd37713dd4be6d63b31186578ba3d08c" Nov 02 18:01:38.232962 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.232921 5549 scope.go:122] "RemoveContainer" containerID="1a98c7d99b4d73b6b387f30afe43f487bd37713dd4be6d63b31186578ba3d08c" Nov 02 18:01:38.233278 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:01:38.233249 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a98c7d99b4d73b6b387f30afe43f487bd37713dd4be6d63b31186578ba3d08c\": container with ID starting with 1a98c7d99b4d73b6b387f30afe43f487bd37713dd4be6d63b31186578ba3d08c not found: ID does not exist" containerID="1a98c7d99b4d73b6b387f30afe43f487bd37713dd4be6d63b31186578ba3d08c" Nov 02 18:01:38.233373 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.233289 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a98c7d99b4d73b6b387f30afe43f487bd37713dd4be6d63b31186578ba3d08c"} err="failed to get container status \"1a98c7d99b4d73b6b387f30afe43f487bd37713dd4be6d63b31186578ba3d08c\": rpc error: code = NotFound desc = could not find container \"1a98c7d99b4d73b6b387f30afe43f487bd37713dd4be6d63b31186578ba3d08c\": container with ID starting with 1a98c7d99b4d73b6b387f30afe43f487bd37713dd4be6d63b31186578ba3d08c not found: ID does not exist" Nov 02 18:01:38.248652 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.248622 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds"] Nov 02 18:01:38.254848 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.254821 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b6b6f4/auth-test-model-b6b6f4-kserve-5c566bdd9c-jg4ds"] Nov 02 18:01:38.272982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.272923 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:38.322379 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.322350 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d16de1e2-a7a7-4046-90e8-85763b266321" path="/var/lib/kubelet/pods/d16de1e2-a7a7-4046-90e8-85763b266321/volumes" Nov 02 18:01:38.427979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.422861 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:38.930233 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.930198 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww"] Nov 02 18:01:38.931103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.931086 5549 cpu_manager.go:414] "containerMap: removing container" podUID="d16de1e2-a7a7-4046-90e8-85763b266321" containerName="main" Nov 02 18:01:38.931178 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.931105 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16de1e2-a7a7-4046-90e8-85763b266321" containerName="main" Nov 02 18:01:38.931293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.931283 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16de1e2-a7a7-4046-90e8-85763b266321" containerName="main" Nov 02 18:01:38.936812 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.936785 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" Nov 02 18:01:38.945644 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.945596 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww"] Nov 02 18:01:38.948761 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.948734 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-807c08\"" Nov 02 18:01:38.961382 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.961355 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:38.970663 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.970621 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b"] Nov 02 18:01:38.979212 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.979187 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" Nov 02 18:01:38.983568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.983539 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b"] Nov 02 18:01:38.994464 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:38.994439 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-807c08-dockercfg-qwgjm\"" Nov 02 18:01:39.026851 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.026816 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078222c5-8861-4bb3-9969-5aca397e9195-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww\" (UID: \"078222c5-8861-4bb3-9969-5aca397e9195\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" Nov 02 18:01:39.027098 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.027077 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58xw6\" (UniqueName: \"kubernetes.io/projected/52c33649-7da9-4772-817d-d924c1013123-kube-api-access-58xw6\") pod \"payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b\" (UID: \"52c33649-7da9-4772-817d-d924c1013123\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" Nov 02 18:01:39.027154 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.027135 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c33649-7da9-4772-817d-d924c1013123-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b\" (UID: \"52c33649-7da9-4772-817d-d924c1013123\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" Nov 02 18:01:39.027227 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.027211 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k76xr\" (UniqueName: \"kubernetes.io/projected/078222c5-8861-4bb3-9969-5aca397e9195-kube-api-access-k76xr\") pod \"payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww\" (UID: \"078222c5-8861-4bb3-9969-5aca397e9195\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" Nov 02 18:01:39.102145 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.102111 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:39.130142 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.130092 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c33649-7da9-4772-817d-d924c1013123-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b\" (UID: \"52c33649-7da9-4772-817d-d924c1013123\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" Nov 02 18:01:39.131600 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.131535 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k76xr\" (UniqueName: \"kubernetes.io/projected/078222c5-8861-4bb3-9969-5aca397e9195-kube-api-access-k76xr\") pod \"payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww\" (UID: \"078222c5-8861-4bb3-9969-5aca397e9195\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" Nov 02 18:01:39.131760 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.131008 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c33649-7da9-4772-817d-d924c1013123-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b\" (UID: \"52c33649-7da9-4772-817d-d924c1013123\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" Nov 02 18:01:39.132276 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.132221 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078222c5-8861-4bb3-9969-5aca397e9195-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww\" (UID: \"078222c5-8861-4bb3-9969-5aca397e9195\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" Nov 02 18:01:39.132437 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.132415 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58xw6\" (UniqueName: \"kubernetes.io/projected/52c33649-7da9-4772-817d-d924c1013123-kube-api-access-58xw6\") pod \"payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b\" (UID: \"52c33649-7da9-4772-817d-d924c1013123\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" Nov 02 18:01:39.134907 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.133508 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078222c5-8861-4bb3-9969-5aca397e9195-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww\" (UID: \"078222c5-8861-4bb3-9969-5aca397e9195\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" Nov 02 18:01:39.176988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.173102 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58xw6\" (UniqueName: \"kubernetes.io/projected/52c33649-7da9-4772-817d-d924c1013123-kube-api-access-58xw6\") pod \"payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b\" (UID: \"52c33649-7da9-4772-817d-d924c1013123\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" Nov 02 18:01:39.204436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.204398 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k76xr\" (UniqueName: \"kubernetes.io/projected/078222c5-8861-4bb3-9969-5aca397e9195-kube-api-access-k76xr\") pod \"payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww\" (UID: \"078222c5-8861-4bb3-9969-5aca397e9195\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" Nov 02 18:01:39.309976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.293441 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" Nov 02 18:01:39.309976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.296492 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:39.309976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.305359 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" Nov 02 18:01:39.503038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.458599 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:39.896682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.896650 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww"] Nov 02 18:01:39.897087 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:01:39.897058 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078222c5_8861_4bb3_9969_5aca397e9195.slice/crio-12ce20a61cb9c475952a2539c4e1dce0bc5f8fd57ede5ef260276e1ecfadf171 WatchSource:0}: Error finding container 12ce20a61cb9c475952a2539c4e1dce0bc5f8fd57ede5ef260276e1ecfadf171: Status 404 returned error can't find the container with id 12ce20a61cb9c475952a2539c4e1dce0bc5f8fd57ede5ef260276e1ecfadf171 Nov 02 18:01:39.983901 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:39.983870 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b"] Nov 02 18:01:39.984808 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:01:39.984782 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c33649_7da9_4772_817d_d924c1013123.slice/crio-800c2e7fc21a0d5e320a19bd7a24451a839407ddd89e1062f83298606428899f WatchSource:0}: Error finding container 800c2e7fc21a0d5e320a19bd7a24451a839407ddd89e1062f83298606428899f: Status 404 returned error can't find the container with id 800c2e7fc21a0d5e320a19bd7a24451a839407ddd89e1062f83298606428899f Nov 02 18:01:40.003040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:40.002731 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:40.527451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:40.527405 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" event={"ID":"52c33649-7da9-4772-817d-d924c1013123","Type":"ContainerStarted","Data":"52ba1de1ba738949b873f2471d44302a45f5b8df42d1a5f619cde9d1a1cd7870"} Nov 02 18:01:40.527888 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:40.527475 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" event={"ID":"52c33649-7da9-4772-817d-d924c1013123","Type":"ContainerStarted","Data":"800c2e7fc21a0d5e320a19bd7a24451a839407ddd89e1062f83298606428899f"} Nov 02 18:01:40.527888 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:40.527508 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" Nov 02 18:01:40.529184 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:40.529158 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" event={"ID":"078222c5-8861-4bb3-9969-5aca397e9195","Type":"ContainerStarted","Data":"ec208a09f5b6d8644a1a2bcaf1a63fddd42b8d7f0d32833f5eaf7a2bd81c8ce5"} Nov 02 18:01:40.529311 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:40.529208 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" event={"ID":"078222c5-8861-4bb3-9969-5aca397e9195","Type":"ContainerStarted","Data":"12ce20a61cb9c475952a2539c4e1dce0bc5f8fd57ede5ef260276e1ecfadf171"} Nov 02 18:01:40.529311 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:40.529252 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" Nov 02 18:01:40.549021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:40.548965 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" podStartSLOduration=2.548932187 podStartE2EDuration="2.548932187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:40.545710036 +0000 UTC m=+1672.334215566" watchObservedRunningTime="2026-11-02 18:01:40.548932187 +0000 UTC m=+1672.337437717" Nov 02 18:01:40.567091 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:40.567041 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" podStartSLOduration=2.567027998 podStartE2EDuration="2.567027998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:40.561657831 +0000 UTC m=+1672.350163371" watchObservedRunningTime="2026-11-02 18:01:40.567027998 +0000 UTC m=+1672.355533526" Nov 02 18:01:41.020975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:41.017208 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:41.148273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:41.148235 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" Nov 02 18:01:41.205375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:41.205345 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" Nov 02 18:01:41.661396 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:41.661362 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:41.825054 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:41.825020 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:41.979932 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:41.979899 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:42.842493 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:42.842371 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:42.984536 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:42.984501 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:43.065831 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.065798 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh"] Nov 02 18:01:43.073064 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.070992 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" Nov 02 18:01:43.077387 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.077355 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh"] Nov 02 18:01:43.078517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.078492 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-327608\"" Nov 02 18:01:43.078517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.078499 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-327608-dockercfg-lqrf6\"" Nov 02 18:01:43.440974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.440476 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwfkl\" (UniqueName: \"kubernetes.io/projected/9baba3f3-ce78-4167-8f82-0e24b38a5cda-kube-api-access-nwfkl\") pod \"payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh\" (UID: \"9baba3f3-ce78-4167-8f82-0e24b38a5cda\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" Nov 02 18:01:43.441165 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.441046 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9baba3f3-ce78-4167-8f82-0e24b38a5cda-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh\" (UID: \"9baba3f3-ce78-4167-8f82-0e24b38a5cda\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" Nov 02 18:01:43.473996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.471942 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:43.487987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.478035 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27"] Nov 02 18:01:43.500071 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.496149 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27"] Nov 02 18:01:43.500071 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.496273 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" Nov 02 18:01:43.542340 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.542307 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwfkl\" (UniqueName: \"kubernetes.io/projected/9baba3f3-ce78-4167-8f82-0e24b38a5cda-kube-api-access-nwfkl\") pod \"payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh\" (UID: \"9baba3f3-ce78-4167-8f82-0e24b38a5cda\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" Nov 02 18:01:43.542658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.542641 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9baba3f3-ce78-4167-8f82-0e24b38a5cda-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh\" (UID: \"9baba3f3-ce78-4167-8f82-0e24b38a5cda\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" Nov 02 18:01:43.543712 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.543681 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9baba3f3-ce78-4167-8f82-0e24b38a5cda-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh\" (UID: \"9baba3f3-ce78-4167-8f82-0e24b38a5cda\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" Nov 02 18:01:43.551563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.551538 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwfkl\" (UniqueName: \"kubernetes.io/projected/9baba3f3-ce78-4167-8f82-0e24b38a5cda-kube-api-access-nwfkl\") pod \"payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh\" (UID: \"9baba3f3-ce78-4167-8f82-0e24b38a5cda\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" Nov 02 18:01:43.644350 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.644314 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcmst\" (UniqueName: \"kubernetes.io/projected/88ee8230-6e65-4a89-b9bd-1f7100b03218-kube-api-access-jcmst\") pod \"payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27\" (UID: \"88ee8230-6e65-4a89-b9bd-1f7100b03218\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" Nov 02 18:01:43.644532 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.644371 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88ee8230-6e65-4a89-b9bd-1f7100b03218-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27\" (UID: \"88ee8230-6e65-4a89-b9bd-1f7100b03218\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" Nov 02 18:01:43.690030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.689947 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" Nov 02 18:01:43.746094 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.746052 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcmst\" (UniqueName: \"kubernetes.io/projected/88ee8230-6e65-4a89-b9bd-1f7100b03218-kube-api-access-jcmst\") pod \"payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27\" (UID: \"88ee8230-6e65-4a89-b9bd-1f7100b03218\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" Nov 02 18:01:43.746292 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.746118 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88ee8230-6e65-4a89-b9bd-1f7100b03218-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27\" (UID: \"88ee8230-6e65-4a89-b9bd-1f7100b03218\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" Nov 02 18:01:43.746924 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.746896 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88ee8230-6e65-4a89-b9bd-1f7100b03218-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27\" (UID: \"88ee8230-6e65-4a89-b9bd-1f7100b03218\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" Nov 02 18:01:43.758466 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.758438 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcmst\" (UniqueName: \"kubernetes.io/projected/88ee8230-6e65-4a89-b9bd-1f7100b03218-kube-api-access-jcmst\") pod \"payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27\" (UID: \"88ee8230-6e65-4a89-b9bd-1f7100b03218\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" Nov 02 18:01:43.868297 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.867715 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:43.909154 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.908897 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" Nov 02 18:01:43.979067 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:43.979031 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh"] Nov 02 18:01:43.979388 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:01:43.979355 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9baba3f3_ce78_4167_8f82_0e24b38a5cda.slice/crio-8728de1d36073acfeddd1c2dfbe0ae3d2ee9a4d97756ef564d5442354d16e1b7 WatchSource:0}: Error finding container 8728de1d36073acfeddd1c2dfbe0ae3d2ee9a4d97756ef564d5442354d16e1b7: Status 404 returned error can't find the container with id 8728de1d36073acfeddd1c2dfbe0ae3d2ee9a4d97756ef564d5442354d16e1b7 Nov 02 18:01:44.217709 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:01:44.217675 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ee8230_6e65_4a89_b9bd_1f7100b03218.slice/crio-3fa58c038d6298ff73d01b46ea499fb0a73d34ceaedd87ac2d9108024713a23d WatchSource:0}: Error finding container 3fa58c038d6298ff73d01b46ea499fb0a73d34ceaedd87ac2d9108024713a23d: Status 404 returned error can't find the container with id 3fa58c038d6298ff73d01b46ea499fb0a73d34ceaedd87ac2d9108024713a23d Nov 02 18:01:44.217709 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:44.217695 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27"] Nov 02 18:01:44.526116 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:44.525995 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:44.609395 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:44.608493 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" event={"ID":"88ee8230-6e65-4a89-b9bd-1f7100b03218","Type":"ContainerStarted","Data":"2c8fcd8bcc9d19d7465aa82957f7b2a66d3db5690c79ca44f8442d2cf9a57ab7"} Nov 02 18:01:44.609395 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:44.608569 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" event={"ID":"88ee8230-6e65-4a89-b9bd-1f7100b03218","Type":"ContainerStarted","Data":"3fa58c038d6298ff73d01b46ea499fb0a73d34ceaedd87ac2d9108024713a23d"} Nov 02 18:01:44.614055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:44.609928 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" Nov 02 18:01:44.628750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:44.628711 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" event={"ID":"9baba3f3-ce78-4167-8f82-0e24b38a5cda","Type":"ContainerStarted","Data":"d2eb79253c87269fc3bb921f14602077d89fedc597159de40def37e2d7a0978f"} Nov 02 18:01:44.628932 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:44.628806 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" event={"ID":"9baba3f3-ce78-4167-8f82-0e24b38a5cda","Type":"ContainerStarted","Data":"8728de1d36073acfeddd1c2dfbe0ae3d2ee9a4d97756ef564d5442354d16e1b7"} Nov 02 18:01:44.628932 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:44.628848 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" Nov 02 18:01:44.635830 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:44.635782 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" podStartSLOduration=1.6357682759999999 podStartE2EDuration="1.635768276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:44.635282958 +0000 UTC m=+1676.423788525" watchObservedRunningTime="2026-11-02 18:01:44.635768276 +0000 UTC m=+1676.424273800" Nov 02 18:01:44.654051 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:44.653998 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" podStartSLOduration=1.653948351 podStartE2EDuration="1.653948351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:44.653419263 +0000 UTC m=+1676.441924794" watchObservedRunningTime="2026-11-02 18:01:44.653948351 +0000 UTC m=+1676.442453879" Nov 02 18:01:45.402650 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:45.402613 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:45.547635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:45.547595 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:45.789785 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:45.789740 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:46.547604 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:46.547566 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" podUID="3a75b540-c8a0-4d10-ac48-5e645c73d0e4" containerName="main" containerID="cri-o://f5d954c0f6f5a55b071fcd85ef66e7bb59b948a750ba39555ac4ae9d81a0f1c7" gracePeriod=45 Nov 02 18:01:46.640974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:46.628813 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:46.869545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:46.869519 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 18:01:46.926607 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:46.926568 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-home\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-home\") pod \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " Nov 02 18:01:46.926804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:46.926660 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-dshm\") pod \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " Nov 02 18:01:46.926874 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:46.926821 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-tmp-dir\") pod \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " Nov 02 18:01:46.926874 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:46.926857 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-home" pod "3a75b540-c8a0-4d10-ac48-5e645c73d0e4" (UID: "3a75b540-c8a0-4d10-ac48-5e645c73d0e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:46.927041 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:46.926913 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-model-cache\") pod \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " Nov 02 18:01:46.927103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:46.927082 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-tls-certs\") pod \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " Nov 02 18:01:46.927162 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:46.927126 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-tmp-dir" pod "3a75b540-c8a0-4d10-ac48-5e645c73d0e4" (UID: "3a75b540-c8a0-4d10-ac48-5e645c73d0e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:46.927218 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:46.927154 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-kube-api-access-tclm9\" (UniqueName: \"kubernetes.io/projected/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-kube-api-access-tclm9\") pod \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\" (UID: \"3a75b540-c8a0-4d10-ac48-5e645c73d0e4\") " Nov 02 18:01:46.927218 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:46.927184 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-model-cache" pod "3a75b540-c8a0-4d10-ac48-5e645c73d0e4" (UID: "3a75b540-c8a0-4d10-ac48-5e645c73d0e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:46.928165 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:46.928138 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:46.928273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:46.928171 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:46.928273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:46.928186 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:46.929896 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:46.929845 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-kube-api-access-tclm9" pod "3a75b540-c8a0-4d10-ac48-5e645c73d0e4" (UID: "3a75b540-c8a0-4d10-ac48-5e645c73d0e4"). InnerVolumeSpecName "kube-api-access-tclm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:46.930080 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:46.929936 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-tls-certs" pod "3a75b540-c8a0-4d10-ac48-5e645c73d0e4" (UID: "3a75b540-c8a0-4d10-ac48-5e645c73d0e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:01:46.930230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:46.930209 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-dshm" pod "3a75b540-c8a0-4d10-ac48-5e645c73d0e4" (UID: "3a75b540-c8a0-4d10-ac48-5e645c73d0e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:47.029777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:47.029741 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:47.029777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:47.029773 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:47.029777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:47.029783 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tclm9\" (UniqueName: \"kubernetes.io/projected/3a75b540-c8a0-4d10-ac48-5e645c73d0e4-kube-api-access-tclm9\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:47.379059 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:47.371310 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:47.659288 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:01:47.655154 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:01:47.681743 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:47.681710 5549 generic.go:356] "Generic (PLEG): container finished" podID="3a75b540-c8a0-4d10-ac48-5e645c73d0e4" containerID="f5d954c0f6f5a55b071fcd85ef66e7bb59b948a750ba39555ac4ae9d81a0f1c7" exitCode=0 Nov 02 18:01:47.681970 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:47.681786 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" Nov 02 18:01:47.681970 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:47.681800 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" event={"ID":"3a75b540-c8a0-4d10-ac48-5e645c73d0e4","Type":"ContainerDied","Data":"f5d954c0f6f5a55b071fcd85ef66e7bb59b948a750ba39555ac4ae9d81a0f1c7"} Nov 02 18:01:47.681970 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:47.681844 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh" event={"ID":"3a75b540-c8a0-4d10-ac48-5e645c73d0e4","Type":"ContainerDied","Data":"53a6131f6a3ca04b9fb07bac29e3e223105f726c474005ac8010f101e65c5a50"} Nov 02 18:01:47.681970 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:47.681863 5549 scope.go:122] "RemoveContainer" containerID="f5d954c0f6f5a55b071fcd85ef66e7bb59b948a750ba39555ac4ae9d81a0f1c7" Nov 02 18:01:47.694064 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:47.694046 5549 scope.go:122] "RemoveContainer" containerID="f5d954c0f6f5a55b071fcd85ef66e7bb59b948a750ba39555ac4ae9d81a0f1c7" Nov 02 18:01:47.694440 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:01:47.694412 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5d954c0f6f5a55b071fcd85ef66e7bb59b948a750ba39555ac4ae9d81a0f1c7\": container with ID starting with f5d954c0f6f5a55b071fcd85ef66e7bb59b948a750ba39555ac4ae9d81a0f1c7 not found: ID does not exist" containerID="f5d954c0f6f5a55b071fcd85ef66e7bb59b948a750ba39555ac4ae9d81a0f1c7" Nov 02 18:01:47.694485 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:47.694455 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d954c0f6f5a55b071fcd85ef66e7bb59b948a750ba39555ac4ae9d81a0f1c7"} err="failed to get container status \"f5d954c0f6f5a55b071fcd85ef66e7bb59b948a750ba39555ac4ae9d81a0f1c7\": rpc error: code = NotFound desc = could not find container \"f5d954c0f6f5a55b071fcd85ef66e7bb59b948a750ba39555ac4ae9d81a0f1c7\": container with ID starting with f5d954c0f6f5a55b071fcd85ef66e7bb59b948a750ba39555ac4ae9d81a0f1c7 not found: ID does not exist" Nov 02 18:01:47.706395 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:47.706363 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh"] Nov 02 18:01:47.710792 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:47.710763 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-2961d5/auth-test-model-2961d5-kserve-9bb54dc6f-b89lh"] Nov 02 18:01:48.118457 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:48.118418 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:48.322464 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:48.322432 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a75b540-c8a0-4d10-ac48-5e645c73d0e4" path="/var/lib/kubelet/pods/3a75b540-c8a0-4d10-ac48-5e645c73d0e4/volumes" Nov 02 18:01:49.105638 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:49.105606 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:50.020892 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:50.020857 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:50.493011 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:50.488582 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:51.355475 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.355388 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:51.647276 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.639977 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" Nov 02 18:01:51.647276 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.643535 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" Nov 02 18:01:51.693536 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.692070 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89"] Nov 02 18:01:51.726494 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.712789 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:51.726494 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.714394 5549 cpu_manager.go:414] "containerMap: removing container" podUID="3a75b540-c8a0-4d10-ac48-5e645c73d0e4" containerName="main" Nov 02 18:01:51.726494 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.714421 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a75b540-c8a0-4d10-ac48-5e645c73d0e4" containerName="main" Nov 02 18:01:51.726494 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.714690 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a75b540-c8a0-4d10-ac48-5e645c73d0e4" containerName="main" Nov 02 18:01:51.726494 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.724538 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.731057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.730388 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-1ef96f-openshift-default-dockercfg-rk97p\"" Nov 02 18:01:51.738974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.734912 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89"] Nov 02 18:01:51.825710 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.825683 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istiod-ca-cert\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.825818 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.825744 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-credential-socket\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.825874 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.825835 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-token\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.825933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.825898 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb2vn\" (UniqueName: \"kubernetes.io/projected/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-kube-api-access-tb2vn\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.826033 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.826005 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-envoy\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.826098 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.826078 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-workload-certs\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.826153 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.826124 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-workload-socket\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.826194 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.826151 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-data\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.826239 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.826199 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-podinfo\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.891925 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.888934 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:51.942979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.942138 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istiod-ca-cert\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.948613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.944029 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istiod-ca-cert\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.948613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.944152 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-credential-socket\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.948613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.944219 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-token\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.948613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.944308 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb2vn\" (UniqueName: \"kubernetes.io/projected/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-kube-api-access-tb2vn\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.948613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.944349 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-envoy\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.948613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.944416 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-workload-certs\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.948613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.944468 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-workload-socket\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.948613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.944492 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-data\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.948613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.944559 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-podinfo\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.948613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.947617 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-credential-socket\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.948613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.948299 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-workload-certs\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.962235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.961338 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-data\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:51.967810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:51.967773 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-workload-socket\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:52.004057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:52.004017 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-podinfo\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:52.007436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:52.007379 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-token\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:52.015453 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:52.015426 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-envoy\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:52.021986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:52.020599 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb2vn\" (UniqueName: \"kubernetes.io/projected/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-kube-api-access-tb2vn\") pod \"e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:52.075971 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:52.073324 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:52.516859 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:01:52.516820 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9fbbb3_23a8_4f3f_a954_085d739a4ca2.slice/crio-4e5e11e5d0c165507c57fcfcb9808d73de42a43fb23b816c08455b082a16937a WatchSource:0}: Error finding container 4e5e11e5d0c165507c57fcfcb9808d73de42a43fb23b816c08455b082a16937a: Status 404 returned error can't find the container with id 4e5e11e5d0c165507c57fcfcb9808d73de42a43fb23b816c08455b082a16937a Nov 02 18:01:52.517038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:52.516864 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89"] Nov 02 18:01:52.520316 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:52.520279 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:01:52.520434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:52.520366 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:01:52.520434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:52.520414 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:01:52.639759 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:52.639640 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:52.799126 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:52.799043 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" event={"ID":"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2","Type":"ContainerStarted","Data":"4e035c6af637a2468e6356bfbfe9ed42c99f034770c74211741644eed8f4f062"} Nov 02 18:01:52.799126 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:52.799105 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" event={"ID":"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2","Type":"ContainerStarted","Data":"4e5e11e5d0c165507c57fcfcb9808d73de42a43fb23b816c08455b082a16937a"} Nov 02 18:01:52.826649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:52.826578 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" podStartSLOduration=1.826560247 podStartE2EDuration="1.826560247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:52.821229953 +0000 UTC m=+1684.609735482" watchObservedRunningTime="2026-11-02 18:01:52.826560247 +0000 UTC m=+1684.615065775" Nov 02 18:01:52.904920 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:52.904891 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:53.079065 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:53.078909 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:53.079249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:53.079094 5549 patch_prober.go:29] interesting pod/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 02 18:01:53.079249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:53.079146 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" podUID="3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 02 18:01:53.879995 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:53.874642 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t"] Nov 02 18:01:53.879995 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:53.874868 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" podUID="a543a25d-8467-4bac-98c4-e0cf38dd2c76" containerName="istio-proxy" containerID="cri-o://d045a6238809325661f83110e0d0c36c21a94caa5e351759728f082824c2837d" gracePeriod=30 Nov 02 18:01:53.879995 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:53.875648 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:54.078851 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:54.078817 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:54.816638 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:54.816603 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:54.817726 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:54.817701 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:01:54.927346 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:54.927314 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:55.187674 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:55.187590 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:55.327145 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:55.327113 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:55.641366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:55.641339 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" Nov 02 18:01:55.949029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:55.944456 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:56.385231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.385146 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:56.412359 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.412326 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t"] Nov 02 18:01:56.419060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.419037 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" Nov 02 18:01:56.425080 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.424685 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-1ef96f\"" Nov 02 18:01:56.425080 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.424746 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-1ef96f-dockercfg-r76s6\"" Nov 02 18:01:56.426919 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.426896 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t"] Nov 02 18:01:56.471214 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.471181 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d"] Nov 02 18:01:56.475994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.475933 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdsk6\" (UniqueName: \"kubernetes.io/projected/821f0afb-87af-4690-a871-6244febd580d-kube-api-access-xdsk6\") pod \"payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t\" (UID: \"821f0afb-87af-4690-a871-6244febd580d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" Nov 02 18:01:56.476145 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.476060 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/821f0afb-87af-4690-a871-6244febd580d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t\" (UID: \"821f0afb-87af-4690-a871-6244febd580d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" Nov 02 18:01:56.476488 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.476473 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" Nov 02 18:01:56.484384 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.484353 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d"] Nov 02 18:01:56.569976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.564898 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:56.610985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.585762 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8062774b-5045-48ff-9846-c20c1ea18860-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d\" (UID: \"8062774b-5045-48ff-9846-c20c1ea18860\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" Nov 02 18:01:56.610985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.586023 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftnpc\" (UniqueName: \"kubernetes.io/projected/8062774b-5045-48ff-9846-c20c1ea18860-kube-api-access-ftnpc\") pod \"payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d\" (UID: \"8062774b-5045-48ff-9846-c20c1ea18860\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" Nov 02 18:01:56.610985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.586105 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdsk6\" (UniqueName: \"kubernetes.io/projected/821f0afb-87af-4690-a871-6244febd580d-kube-api-access-xdsk6\") pod \"payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t\" (UID: \"821f0afb-87af-4690-a871-6244febd580d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" Nov 02 18:01:56.610985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.586158 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/821f0afb-87af-4690-a871-6244febd580d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t\" (UID: \"821f0afb-87af-4690-a871-6244febd580d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" Nov 02 18:01:56.611910 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.611801 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/821f0afb-87af-4690-a871-6244febd580d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t\" (UID: \"821f0afb-87af-4690-a871-6244febd580d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" Nov 02 18:01:56.727974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.720296 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8062774b-5045-48ff-9846-c20c1ea18860-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d\" (UID: \"8062774b-5045-48ff-9846-c20c1ea18860\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" Nov 02 18:01:56.727974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.720483 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftnpc\" (UniqueName: \"kubernetes.io/projected/8062774b-5045-48ff-9846-c20c1ea18860-kube-api-access-ftnpc\") pod \"payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d\" (UID: \"8062774b-5045-48ff-9846-c20c1ea18860\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" Nov 02 18:01:56.727974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.723505 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8062774b-5045-48ff-9846-c20c1ea18860-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d\" (UID: \"8062774b-5045-48ff-9846-c20c1ea18860\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" Nov 02 18:01:56.834980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.834859 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftnpc\" (UniqueName: \"kubernetes.io/projected/8062774b-5045-48ff-9846-c20c1ea18860-kube-api-access-ftnpc\") pod \"payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d\" (UID: \"8062774b-5045-48ff-9846-c20c1ea18860\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" Nov 02 18:01:56.838660 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.836271 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" Nov 02 18:01:56.860689 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:56.859355 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdsk6\" (UniqueName: \"kubernetes.io/projected/821f0afb-87af-4690-a871-6244febd580d-kube-api-access-xdsk6\") pod \"payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t\" (UID: \"821f0afb-87af-4690-a871-6244febd580d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" Nov 02 18:01:57.038167 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:57.038082 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" Nov 02 18:01:57.078278 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:57.078241 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:57.115976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:57.105069 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" Nov 02 18:01:57.235072 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:57.235035 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:57.403808 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:57.403771 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t"] Nov 02 18:01:57.404807 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:01:57.404776 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821f0afb_87af_4690_a871_6244febd580d.slice/crio-9a8fd17d7999daf36358dce62b703a3c89ebeb34b3532d389ae0f228185e2466 WatchSource:0}: Error finding container 9a8fd17d7999daf36358dce62b703a3c89ebeb34b3532d389ae0f228185e2466: Status 404 returned error can't find the container with id 9a8fd17d7999daf36358dce62b703a3c89ebeb34b3532d389ae0f228185e2466 Nov 02 18:01:57.579691 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:57.579662 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d"] Nov 02 18:01:57.580132 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:01:57.580103 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8062774b_5045_48ff_9846_c20c1ea18860.slice/crio-6fa21fdc5fe6a30d137c2229ccb935b48d8c30b2b79f6f540ec09c251ed67041 WatchSource:0}: Error finding container 6fa21fdc5fe6a30d137c2229ccb935b48d8c30b2b79f6f540ec09c251ed67041: Status 404 returned error can't find the container with id 6fa21fdc5fe6a30d137c2229ccb935b48d8c30b2b79f6f540ec09c251ed67041 Nov 02 18:01:57.856485 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:57.856454 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:57.870010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:57.867203 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" event={"ID":"8062774b-5045-48ff-9846-c20c1ea18860","Type":"ContainerStarted","Data":"7f5f6ef98cc6d2f8b8a177411874a92b32208376e68d17626ac9c2fc93bbeffc"} Nov 02 18:01:57.870010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:57.867274 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" event={"ID":"8062774b-5045-48ff-9846-c20c1ea18860","Type":"ContainerStarted","Data":"6fa21fdc5fe6a30d137c2229ccb935b48d8c30b2b79f6f540ec09c251ed67041"} Nov 02 18:01:57.870010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:57.868553 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" Nov 02 18:01:57.873358 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:57.873328 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" event={"ID":"821f0afb-87af-4690-a871-6244febd580d","Type":"ContainerStarted","Data":"baf60d7c581ff4c8b17acfd82f114983491f16adcfcf71f72efbf60b347ca8c9"} Nov 02 18:01:57.873502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:57.873411 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" Nov 02 18:01:57.873502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:57.873435 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" event={"ID":"821f0afb-87af-4690-a871-6244febd580d","Type":"ContainerStarted","Data":"9a8fd17d7999daf36358dce62b703a3c89ebeb34b3532d389ae0f228185e2466"} Nov 02 18:01:57.889228 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:57.889135 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" podStartSLOduration=1.88912032 podStartE2EDuration="1.88912032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:57.885197056 +0000 UTC m=+1689.673702596" watchObservedRunningTime="2026-11-02 18:01:57.88912032 +0000 UTC m=+1689.677625847" Nov 02 18:01:57.904622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:57.904574 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" podStartSLOduration=1.9045606739999998 podStartE2EDuration="1.904560674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:57.902151579 +0000 UTC m=+1689.690657108" watchObservedRunningTime="2026-11-02 18:01:57.904560674 +0000 UTC m=+1689.693066201" Nov 02 18:01:58.042189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:58.042157 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:58.184181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:58.184085 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:59.031178 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.019344 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:59.387291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.387174 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:59.611659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.611631 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 18:01:59.707849 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.707809 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-workload-socket\") pod \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " Nov 02 18:01:59.708044 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.707969 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-envoy\") pod \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " Nov 02 18:01:59.708140 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.708084 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-workload-certs\") pod \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " Nov 02 18:01:59.708140 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.708120 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istiod-ca-cert\") pod \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " Nov 02 18:01:59.708252 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.708174 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-workload-socket" pod "a543a25d-8467-4bac-98c4-e0cf38dd2c76" (UID: "a543a25d-8467-4bac-98c4-e0cf38dd2c76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:59.708308 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.708247 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a543a25d-8467-4bac-98c4-e0cf38dd2c76-kube-api-access-lbtst\" (UniqueName: \"kubernetes.io/projected/a543a25d-8467-4bac-98c4-e0cf38dd2c76-kube-api-access-lbtst\") pod \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " Nov 02 18:01:59.708402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.708317 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-data\") pod \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " Nov 02 18:01:59.708402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.708384 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-podinfo\") pod \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " Nov 02 18:01:59.708507 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.708489 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-workload-certs" pod "a543a25d-8467-4bac-98c4-e0cf38dd2c76" (UID: "a543a25d-8467-4bac-98c4-e0cf38dd2c76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:59.708570 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.708504 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-token\" (UniqueName: \"kubernetes.io/projected/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-token\") pod \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " Nov 02 18:01:59.708570 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.708547 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-credential-socket\") pod \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\" (UID: \"a543a25d-8467-4bac-98c4-e0cf38dd2c76\") " Nov 02 18:01:59.708681 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.708566 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istiod-ca-cert" pod "a543a25d-8467-4bac-98c4-e0cf38dd2c76" (UID: "a543a25d-8467-4bac-98c4-e0cf38dd2c76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:01:59.708744 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.708688 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-data" pod "a543a25d-8467-4bac-98c4-e0cf38dd2c76" (UID: "a543a25d-8467-4bac-98c4-e0cf38dd2c76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:59.708813 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.708792 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-credential-socket" pod "a543a25d-8467-4bac-98c4-e0cf38dd2c76" (UID: "a543a25d-8467-4bac-98c4-e0cf38dd2c76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:59.709641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.709611 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:59.709790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.709644 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:59.709790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.709660 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:59.709790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.709677 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:59.709790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.709692 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:59.710756 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.710732 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a543a25d-8467-4bac-98c4-e0cf38dd2c76-kube-api-access-lbtst" pod "a543a25d-8467-4bac-98c4-e0cf38dd2c76" (UID: "a543a25d-8467-4bac-98c4-e0cf38dd2c76"). InnerVolumeSpecName "kube-api-access-lbtst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:59.710878 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.710863 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-envoy" pod "a543a25d-8467-4bac-98c4-e0cf38dd2c76" (UID: "a543a25d-8467-4bac-98c4-e0cf38dd2c76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:59.710984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.710925 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-token" pod "a543a25d-8467-4bac-98c4-e0cf38dd2c76" (UID: "a543a25d-8467-4bac-98c4-e0cf38dd2c76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:59.711062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.711046 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-podinfo" pod "a543a25d-8467-4bac-98c4-e0cf38dd2c76" (UID: "a543a25d-8467-4bac-98c4-e0cf38dd2c76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:01:59.811107 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.811074 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:59.811107 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.811100 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:59.811107 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.811111 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbtst\" (UniqueName: \"kubernetes.io/projected/a543a25d-8467-4bac-98c4-e0cf38dd2c76-kube-api-access-lbtst\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:59.811367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.811122 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a543a25d-8467-4bac-98c4-e0cf38dd2c76-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:59.876541 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.874029 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:59.944130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.938819 5549 generic.go:356] "Generic (PLEG): container finished" podID="a543a25d-8467-4bac-98c4-e0cf38dd2c76" containerID="d045a6238809325661f83110e0d0c36c21a94caa5e351759728f082824c2837d" exitCode=0 Nov 02 18:01:59.944130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.940071 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" Nov 02 18:01:59.944130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.940398 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" event={"ID":"a543a25d-8467-4bac-98c4-e0cf38dd2c76","Type":"ContainerDied","Data":"d045a6238809325661f83110e0d0c36c21a94caa5e351759728f082824c2837d"} Nov 02 18:01:59.944130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.940449 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t" event={"ID":"a543a25d-8467-4bac-98c4-e0cf38dd2c76","Type":"ContainerDied","Data":"bd27c916de1fd3b0aaab405ba5b41cd6a85820a409c473c1516b3fd4b1580142"} Nov 02 18:01:59.944130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.940478 5549 scope.go:122] "RemoveContainer" containerID="d045a6238809325661f83110e0d0c36c21a94caa5e351759728f082824c2837d" Nov 02 18:01:59.963828 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.963800 5549 scope.go:122] "RemoveContainer" containerID="d045a6238809325661f83110e0d0c36c21a94caa5e351759728f082824c2837d" Nov 02 18:01:59.964296 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:01:59.964265 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d045a6238809325661f83110e0d0c36c21a94caa5e351759728f082824c2837d\": container with ID starting with d045a6238809325661f83110e0d0c36c21a94caa5e351759728f082824c2837d not found: ID does not exist" containerID="d045a6238809325661f83110e0d0c36c21a94caa5e351759728f082824c2837d" Nov 02 18:01:59.964386 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.964305 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d045a6238809325661f83110e0d0c36c21a94caa5e351759728f082824c2837d"} err="failed to get container status \"d045a6238809325661f83110e0d0c36c21a94caa5e351759728f082824c2837d\": rpc error: code = NotFound desc = could not find container \"d045a6238809325661f83110e0d0c36c21a94caa5e351759728f082824c2837d\": container with ID starting with d045a6238809325661f83110e0d0c36c21a94caa5e351759728f082824c2837d not found: ID does not exist" Nov 02 18:01:59.978304 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.978279 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t"] Nov 02 18:01:59.985193 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:01:59.985166 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-t5p9t"] Nov 02 18:02:00.076781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.076742 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn"] Nov 02 18:02:00.078271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.078248 5549 cpu_manager.go:414] "containerMap: removing container" podUID="a543a25d-8467-4bac-98c4-e0cf38dd2c76" containerName="istio-proxy" Nov 02 18:02:00.078359 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.078274 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a543a25d-8467-4bac-98c4-e0cf38dd2c76" containerName="istio-proxy" Nov 02 18:02:00.078630 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.078607 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a543a25d-8467-4bac-98c4-e0cf38dd2c76" containerName="istio-proxy" Nov 02 18:02:00.085974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.085932 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.087580 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.087549 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn"] Nov 02 18:02:00.096527 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.095944 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-4ea582\"/\"autores-model-4ea582-kserve-self-signed-certs\"" Nov 02 18:02:00.096527 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.096146 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-4ea582\"/\"default-dockercfg-npfdx\"" Nov 02 18:02:00.096527 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.096217 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-4ea582\"/\"kube-root-ca.crt\"" Nov 02 18:02:00.097136 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.097110 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-4ea582\"/\"openshift-service-ca.crt\"" Nov 02 18:02:00.159938 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.159907 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:00.259030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.242964 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-home\") pod \"autores-model-4ea582-kserve-7fbd76c949-cklqn\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.259030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.243133 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-tmp-dir\") pod \"autores-model-4ea582-kserve-7fbd76c949-cklqn\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.259030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.243226 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjt2k\" (UniqueName: \"kubernetes.io/projected/501cea11-004b-4161-b64f-552baabdd8bd-kube-api-access-rjt2k\") pod \"autores-model-4ea582-kserve-7fbd76c949-cklqn\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.259030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.243289 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501cea11-004b-4161-b64f-552baabdd8bd-tls-certs\") pod \"autores-model-4ea582-kserve-7fbd76c949-cklqn\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.259030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.243378 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-dshm\") pod \"autores-model-4ea582-kserve-7fbd76c949-cklqn\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.259030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.243408 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-model-cache\") pod \"autores-model-4ea582-kserve-7fbd76c949-cklqn\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.348977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.348544 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-home\") pod \"autores-model-4ea582-kserve-7fbd76c949-cklqn\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.348977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.348681 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-tmp-dir\") pod \"autores-model-4ea582-kserve-7fbd76c949-cklqn\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.348977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.348758 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjt2k\" (UniqueName: \"kubernetes.io/projected/501cea11-004b-4161-b64f-552baabdd8bd-kube-api-access-rjt2k\") pod \"autores-model-4ea582-kserve-7fbd76c949-cklqn\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.348977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.348808 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501cea11-004b-4161-b64f-552baabdd8bd-tls-certs\") pod \"autores-model-4ea582-kserve-7fbd76c949-cklqn\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.348977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.348883 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-dshm\") pod \"autores-model-4ea582-kserve-7fbd76c949-cklqn\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.348977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.348909 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-model-cache\") pod \"autores-model-4ea582-kserve-7fbd76c949-cklqn\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.349809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.349610 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-home\") pod \"autores-model-4ea582-kserve-7fbd76c949-cklqn\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.350041 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.350011 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-tmp-dir\") pod \"autores-model-4ea582-kserve-7fbd76c949-cklqn\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.353258 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.353233 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501cea11-004b-4161-b64f-552baabdd8bd-tls-certs\") pod \"autores-model-4ea582-kserve-7fbd76c949-cklqn\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.358973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.356490 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-model-cache\") pod \"autores-model-4ea582-kserve-7fbd76c949-cklqn\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.367818 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.367778 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjt2k\" (UniqueName: \"kubernetes.io/projected/501cea11-004b-4161-b64f-552baabdd8bd-kube-api-access-rjt2k\") pod \"autores-model-4ea582-kserve-7fbd76c949-cklqn\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.375267 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.375236 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-dshm\") pod \"autores-model-4ea582-kserve-7fbd76c949-cklqn\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.381057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.381025 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a543a25d-8467-4bac-98c4-e0cf38dd2c76" path="/var/lib/kubelet/pods/a543a25d-8467-4bac-98c4-e0cf38dd2c76/volumes" Nov 02 18:02:00.410353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.410314 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:00.839040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.833096 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:00.875593 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.874321 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn"] Nov 02 18:02:00.878681 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:02:00.878650 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod501cea11_004b_4161_b64f_552baabdd8bd.slice/crio-beee01123e1759f07606eac4b61a1be41afd027b5c564b2c8ecc50925abd0d9f WatchSource:0}: Error finding container beee01123e1759f07606eac4b61a1be41afd027b5c564b2c8ecc50925abd0d9f: Status 404 returned error can't find the container with id beee01123e1759f07606eac4b61a1be41afd027b5c564b2c8ecc50925abd0d9f Nov 02 18:02:00.951885 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:00.951856 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" event={"ID":"501cea11-004b-4161-b64f-552baabdd8bd","Type":"ContainerStarted","Data":"beee01123e1759f07606eac4b61a1be41afd027b5c564b2c8ecc50925abd0d9f"} Nov 02 18:02:01.455386 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:01.455347 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:01.589357 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:01.589321 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:01.900045 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:01.899942 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:01.963174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:01.963135 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" event={"ID":"501cea11-004b-4161-b64f-552baabdd8bd","Type":"ContainerStarted","Data":"9e557c1a1ba927f2b0e5ff4dab799775ffaef0b9ea34d7cb55721148dd3a9aa9"} Nov 02 18:02:01.984610 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:01.984552 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" podStartSLOduration=1.984538717 podStartE2EDuration="1.984538717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:01.981829705 +0000 UTC m=+1693.770335234" watchObservedRunningTime="2026-11-02 18:02:01.984538717 +0000 UTC m=+1693.773044242" Nov 02 18:02:02.411199 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:02.411164 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:02.601917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:02.596390 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:03.414979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:03.403529 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:04.145976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:04.139177 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:04.876287 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:04.876248 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:05.640978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:05.635591 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:06.435975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:06.435217 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:07.520366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:07.520329 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k"] Nov 02 18:02:07.520779 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:07.520613 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" podUID="9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" containerName="istio-proxy" containerID="cri-o://7996a3bda2218120283bac2912d2893c97793bd0ef20db36a8f2d87c9408a899" gracePeriod=30 Nov 02 18:02:07.525177 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:07.525149 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b"] Nov 02 18:02:07.525375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:07.525344 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" podUID="4952505b-a6ee-469c-b3ad-39f897429fb4" containerName="istio-proxy" containerID="cri-o://d61b233561e01a4e4acb17d924b420f0737c1dc82571c10afc1195ee03918b33" gracePeriod=30 Nov 02 18:02:07.578385 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:07.578348 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:08.002983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:08.000984 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:08.437044 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:08.417657 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:08.884275 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:08.884243 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" Nov 02 18:02:09.285199 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:09.285163 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:09.417434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:09.417403 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:09.705038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:09.704993 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:09.850548 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:09.850509 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:09.977032 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:09.968374 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" Nov 02 18:02:10.414315 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:10.414237 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:10.462979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:10.455118 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:10.488869 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:10.488829 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:11.204128 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:11.204096 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:11.261418 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:11.258935 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:11.942843 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:11.942798 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:12.623991 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:12.623551 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:12.846808 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:12.846691 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl"] Nov 02 18:02:12.873697 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:12.872297 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:12.882048 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:12.880457 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl"] Nov 02 18:02:12.939195 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:12.939163 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-4ea582-openshift-default-dockercfg-4stc9\"" Nov 02 18:02:12.975927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:12.975895 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 18:02:12.980349 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:12.980323 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 18:02:13.048384 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.048346 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-token\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.048584 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.048412 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-data\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.048584 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.048510 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-podinfo\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.048719 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.048700 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-credential-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.048765 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.048735 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-workload-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.048848 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.048835 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmtds\" (UniqueName: \"kubernetes.io/projected/fa70aff2-1e86-4d0c-a95a-57cc50537c65-kube-api-access-hmtds\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.048944 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.048926 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-workload-certs\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.049073 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.049017 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istiod-ca-cert\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.049121 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.049068 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-envoy\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.150419 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.150337 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-token\" (UniqueName: \"kubernetes.io/projected/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-token\") pod \"4952505b-a6ee-469c-b3ad-39f897429fb4\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " Nov 02 18:02:13.150419 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.150374 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-podinfo\") pod \"4952505b-a6ee-469c-b3ad-39f897429fb4\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " Nov 02 18:02:13.150679 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.150479 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4952505b-a6ee-469c-b3ad-39f897429fb4-kube-api-access-67nfr\" (UniqueName: \"kubernetes.io/projected/4952505b-a6ee-469c-b3ad-39f897429fb4-kube-api-access-67nfr\") pod \"4952505b-a6ee-469c-b3ad-39f897429fb4\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " Nov 02 18:02:13.150679 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.150500 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-workload-certs\") pod \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " Nov 02 18:02:13.150679 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.150528 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-data\") pod \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " Nov 02 18:02:13.150679 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.150570 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4952505b-a6ee-469c-b3ad-39f897429fb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4952505b-a6ee-469c-b3ad-39f897429fb4-istiod-ca-cert\") pod \"4952505b-a6ee-469c-b3ad-39f897429fb4\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " Nov 02 18:02:13.150679 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.150605 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-envoy\") pod \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " Nov 02 18:02:13.150679 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.150633 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-workload-socket\") pod \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " Nov 02 18:02:13.151040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.150710 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-kube-api-access-q7kj5\" (UniqueName: \"kubernetes.io/projected/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-kube-api-access-q7kj5\") pod \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " Nov 02 18:02:13.151040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.150817 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istiod-ca-cert\") pod \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " Nov 02 18:02:13.151040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.150850 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-data" pod "9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" (UID: "9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:13.151040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.150871 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-credential-socket\") pod \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " Nov 02 18:02:13.151040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.150983 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-credential-socket\") pod \"4952505b-a6ee-469c-b3ad-39f897429fb4\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " Nov 02 18:02:13.151040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.151019 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-envoy\") pod \"4952505b-a6ee-469c-b3ad-39f897429fb4\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " Nov 02 18:02:13.151040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.151020 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4952505b-a6ee-469c-b3ad-39f897429fb4-istiod-ca-cert" pod "4952505b-a6ee-469c-b3ad-39f897429fb4" (UID: "4952505b-a6ee-469c-b3ad-39f897429fb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:02:13.151400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.151045 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-token\" (UniqueName: \"kubernetes.io/projected/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-token\") pod \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " Nov 02 18:02:13.151400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.151094 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-data\") pod \"4952505b-a6ee-469c-b3ad-39f897429fb4\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " Nov 02 18:02:13.151400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.151138 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-workload-certs\") pod \"4952505b-a6ee-469c-b3ad-39f897429fb4\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " Nov 02 18:02:13.151400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.151294 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-workload-socket\") pod \"4952505b-a6ee-469c-b3ad-39f897429fb4\" (UID: \"4952505b-a6ee-469c-b3ad-39f897429fb4\") " Nov 02 18:02:13.151400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.151337 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-podinfo\") pod \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\" (UID: \"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5\") " Nov 02 18:02:13.151647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.151598 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-credential-socket" pod "4952505b-a6ee-469c-b3ad-39f897429fb4" (UID: "4952505b-a6ee-469c-b3ad-39f897429fb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:13.151744 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.151722 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-workload-socket" pod "9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" (UID: "9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:13.152326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.151874 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-credential-socket" pod "9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" (UID: "9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:13.152326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.151922 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-workload-certs" pod "4952505b-a6ee-469c-b3ad-39f897429fb4" (UID: "4952505b-a6ee-469c-b3ad-39f897429fb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:13.152326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.152018 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-workload-certs" pod "9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" (UID: "9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:13.152326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.152055 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istiod-ca-cert" pod "9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" (UID: "9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:02:13.152326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.152143 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-data" pod "4952505b-a6ee-469c-b3ad-39f897429fb4" (UID: "4952505b-a6ee-469c-b3ad-39f897429fb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:13.152326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.152283 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-workload-socket" pod "4952505b-a6ee-469c-b3ad-39f897429fb4" (UID: "4952505b-a6ee-469c-b3ad-39f897429fb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:13.152727 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.152486 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istiod-ca-cert\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.152727 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.152536 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-envoy\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.152727 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.152644 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-token\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.152727 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.152674 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-data\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.152727 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.152699 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-podinfo\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.153083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.152812 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-credential-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.153083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.152835 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-workload-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.153083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.152920 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtds\" (UniqueName: \"kubernetes.io/projected/fa70aff2-1e86-4d0c-a95a-57cc50537c65-kube-api-access-hmtds\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.153083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.153042 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-workload-certs\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.153300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.153128 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:13.153300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.153138 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:13.153300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.153147 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4952505b-a6ee-469c-b3ad-39f897429fb4-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:13.153300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.153157 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:13.153300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.153170 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:13.153300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.153189 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:13.153300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.153201 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:13.153300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.153211 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:13.153300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.153227 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:13.153300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.153236 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:13.153684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.153592 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-workload-certs\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.153742 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.153688 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-data\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.154031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.154003 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-kube-api-access-q7kj5" pod "9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" (UID: "9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5"). InnerVolumeSpecName "kube-api-access-q7kj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:13.154185 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.154083 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-envoy" pod "9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" (UID: "9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:13.154322 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.154295 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-workload-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.154452 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.154433 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istiod-ca-cert\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.154531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.154488 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-credential-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.155084 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.155063 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4952505b-a6ee-469c-b3ad-39f897429fb4-kube-api-access-67nfr" pod "4952505b-a6ee-469c-b3ad-39f897429fb4" (UID: "4952505b-a6ee-469c-b3ad-39f897429fb4"). InnerVolumeSpecName "kube-api-access-67nfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:13.155171 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.155138 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-token" pod "9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" (UID: "9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:13.155299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.155277 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-podinfo" pod "4952505b-a6ee-469c-b3ad-39f897429fb4" (UID: "4952505b-a6ee-469c-b3ad-39f897429fb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:02:13.155299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.155290 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-token" pod "4952505b-a6ee-469c-b3ad-39f897429fb4" (UID: "4952505b-a6ee-469c-b3ad-39f897429fb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:13.156757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.156728 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-envoy" pod "4952505b-a6ee-469c-b3ad-39f897429fb4" (UID: "4952505b-a6ee-469c-b3ad-39f897429fb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:13.156843 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.156825 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-podinfo\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.156843 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.156834 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-envoy\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.156921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.156879 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-podinfo" pod "9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" (UID: "9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:02:13.163337 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.163310 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmtds\" (UniqueName: \"kubernetes.io/projected/fa70aff2-1e86-4d0c-a95a-57cc50537c65-kube-api-access-hmtds\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.163418 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.163404 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-token\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.209609 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.209575 5549 generic.go:356] "Generic (PLEG): container finished" podID="9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" containerID="7996a3bda2218120283bac2912d2893c97793bd0ef20db36a8f2d87c9408a899" exitCode=0 Nov 02 18:02:13.209920 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.209625 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" Nov 02 18:02:13.209920 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.209660 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" event={"ID":"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5","Type":"ContainerDied","Data":"7996a3bda2218120283bac2912d2893c97793bd0ef20db36a8f2d87c9408a899"} Nov 02 18:02:13.209920 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.209715 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k" event={"ID":"9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5","Type":"ContainerDied","Data":"f8972be8c22c4fbfc1ed2c9bb33c93325357b79b385daf18195e0fdc59bc3fcb"} Nov 02 18:02:13.209920 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.209745 5549 scope.go:122] "RemoveContainer" containerID="7996a3bda2218120283bac2912d2893c97793bd0ef20db36a8f2d87c9408a899" Nov 02 18:02:13.211370 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.211345 5549 generic.go:356] "Generic (PLEG): container finished" podID="4952505b-a6ee-469c-b3ad-39f897429fb4" containerID="d61b233561e01a4e4acb17d924b420f0737c1dc82571c10afc1195ee03918b33" exitCode=0 Nov 02 18:02:13.211547 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.211397 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" Nov 02 18:02:13.211547 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.211424 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" event={"ID":"4952505b-a6ee-469c-b3ad-39f897429fb4","Type":"ContainerDied","Data":"d61b233561e01a4e4acb17d924b420f0737c1dc82571c10afc1195ee03918b33"} Nov 02 18:02:13.211547 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.211486 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b" event={"ID":"4952505b-a6ee-469c-b3ad-39f897429fb4","Type":"ContainerDied","Data":"bdb581fc99ca0fe31958f71dc16a911e98c4a90f473354c81daefc80c0ba2abf"} Nov 02 18:02:13.223091 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.223071 5549 scope.go:122] "RemoveContainer" containerID="7996a3bda2218120283bac2912d2893c97793bd0ef20db36a8f2d87c9408a899" Nov 02 18:02:13.223366 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:02:13.223347 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7996a3bda2218120283bac2912d2893c97793bd0ef20db36a8f2d87c9408a899\": container with ID starting with 7996a3bda2218120283bac2912d2893c97793bd0ef20db36a8f2d87c9408a899 not found: ID does not exist" containerID="7996a3bda2218120283bac2912d2893c97793bd0ef20db36a8f2d87c9408a899" Nov 02 18:02:13.223444 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.223378 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7996a3bda2218120283bac2912d2893c97793bd0ef20db36a8f2d87c9408a899"} err="failed to get container status \"7996a3bda2218120283bac2912d2893c97793bd0ef20db36a8f2d87c9408a899\": rpc error: code = NotFound desc = could not find container \"7996a3bda2218120283bac2912d2893c97793bd0ef20db36a8f2d87c9408a899\": container with ID starting with 7996a3bda2218120283bac2912d2893c97793bd0ef20db36a8f2d87c9408a899 not found: ID does not exist" Nov 02 18:02:13.223444 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.223404 5549 scope.go:122] "RemoveContainer" containerID="d61b233561e01a4e4acb17d924b420f0737c1dc82571c10afc1195ee03918b33" Nov 02 18:02:13.234702 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.234685 5549 scope.go:122] "RemoveContainer" containerID="d61b233561e01a4e4acb17d924b420f0737c1dc82571c10afc1195ee03918b33" Nov 02 18:02:13.235011 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:02:13.234986 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61b233561e01a4e4acb17d924b420f0737c1dc82571c10afc1195ee03918b33\": container with ID starting with d61b233561e01a4e4acb17d924b420f0737c1dc82571c10afc1195ee03918b33 not found: ID does not exist" containerID="d61b233561e01a4e4acb17d924b420f0737c1dc82571c10afc1195ee03918b33" Nov 02 18:02:13.235060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.235022 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61b233561e01a4e4acb17d924b420f0737c1dc82571c10afc1195ee03918b33"} err="failed to get container status \"d61b233561e01a4e4acb17d924b420f0737c1dc82571c10afc1195ee03918b33\": rpc error: code = NotFound desc = could not find container \"d61b233561e01a4e4acb17d924b420f0737c1dc82571c10afc1195ee03918b33\": container with ID starting with d61b233561e01a4e4acb17d924b420f0737c1dc82571c10afc1195ee03918b33 not found: ID does not exist" Nov 02 18:02:13.243410 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.243388 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k"] Nov 02 18:02:13.254503 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.254474 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67nfr\" (UniqueName: \"kubernetes.io/projected/4952505b-a6ee-469c-b3ad-39f897429fb4-kube-api-access-67nfr\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:13.254503 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.254504 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:13.254669 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.254523 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7kj5\" (UniqueName: \"kubernetes.io/projected/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-kube-api-access-q7kj5\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:13.254669 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.254536 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:13.254669 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.254549 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:13.254669 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.254562 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:13.254669 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.254575 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:13.254669 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.254588 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4952505b-a6ee-469c-b3ad-39f897429fb4-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:13.254997 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.254974 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-4gg9k"] Nov 02 18:02:13.274557 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.274530 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:13.274860 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.274840 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b"] Nov 02 18:02:13.282061 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.282037 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-vrb5b"] Nov 02 18:02:13.485252 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.480452 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:13.568480 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:02:13.568447 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa70aff2_1e86_4d0c_a95a_57cc50537c65.slice/crio-6d9435f17f38ea4e1f2940ebbcbab0b7268f4435ee34419b7dcd4a12f118774a WatchSource:0}: Error finding container 6d9435f17f38ea4e1f2940ebbcbab0b7268f4435ee34419b7dcd4a12f118774a: Status 404 returned error can't find the container with id 6d9435f17f38ea4e1f2940ebbcbab0b7268f4435ee34419b7dcd4a12f118774a Nov 02 18:02:13.570359 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.570334 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl"] Nov 02 18:02:13.572249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.572211 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:13.572332 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.572295 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:13.572382 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:13.572340 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:14.106622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.106587 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv"] Nov 02 18:02:14.108103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.108075 5549 cpu_manager.go:414] "containerMap: removing container" podUID="9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" containerName="istio-proxy" Nov 02 18:02:14.108103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.108100 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" containerName="istio-proxy" Nov 02 18:02:14.108317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.108126 5549 cpu_manager.go:414] "containerMap: removing container" podUID="4952505b-a6ee-469c-b3ad-39f897429fb4" containerName="istio-proxy" Nov 02 18:02:14.108317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.108134 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4952505b-a6ee-469c-b3ad-39f897429fb4" containerName="istio-proxy" Nov 02 18:02:14.108517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.108326 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" containerName="istio-proxy" Nov 02 18:02:14.108517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.108340 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="4952505b-a6ee-469c-b3ad-39f897429fb4" containerName="istio-proxy" Nov 02 18:02:14.113275 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.113251 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" Nov 02 18:02:14.123443 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.121060 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-1ef96f-serving-cert\"" Nov 02 18:02:14.125702 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.125680 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv"] Nov 02 18:02:14.211305 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.211267 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7wm2\" (UniqueName: \"kubernetes.io/projected/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-kube-api-access-d7wm2\") pod \"maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv\" (UID: \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" Nov 02 18:02:14.211514 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.211418 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv\" (UID: \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" Nov 02 18:02:14.211610 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.211593 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-metrics-certs\") pod \"maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv\" (UID: \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" Nov 02 18:02:14.211743 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.211716 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv\" (UID: \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" Nov 02 18:02:14.221741 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.221714 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:14.223231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.223202 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" event={"ID":"fa70aff2-1e86-4d0c-a95a-57cc50537c65","Type":"ContainerStarted","Data":"fa2a441db385a2d0f3e298f7bf6b5254630aa859c097399e5ef3eb0ff560d7b2"} Nov 02 18:02:14.224899 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.224875 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" event={"ID":"fa70aff2-1e86-4d0c-a95a-57cc50537c65","Type":"ContainerStarted","Data":"6d9435f17f38ea4e1f2940ebbcbab0b7268f4435ee34419b7dcd4a12f118774a"} Nov 02 18:02:14.249490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.249427 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" podStartSLOduration=2.249408412 podStartE2EDuration="2.249408412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:14.245623222 +0000 UTC m=+1706.034128752" watchObservedRunningTime="2026-11-02 18:02:14.249408412 +0000 UTC m=+1706.037913941" Nov 02 18:02:14.275339 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.275310 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:14.276749 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.276720 5549 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Nov 02 18:02:14.276889 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.276778 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" podUID="fa70aff2-1e86-4d0c-a95a-57cc50537c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Nov 02 18:02:14.313389 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.313357 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv\" (UID: \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" Nov 02 18:02:14.313551 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.313523 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-metrics-certs\") pod \"maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv\" (UID: \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" Nov 02 18:02:14.313666 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.313646 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv\" (UID: \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" Nov 02 18:02:14.313887 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.313864 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7wm2\" (UniqueName: \"kubernetes.io/projected/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-kube-api-access-d7wm2\") pod \"maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv\" (UID: \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" Nov 02 18:02:14.314583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.314557 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv\" (UID: \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" Nov 02 18:02:14.316275 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.316252 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-metrics-certs\") pod \"maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv\" (UID: \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" Nov 02 18:02:14.316411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.316388 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv\" (UID: \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" Nov 02 18:02:14.322194 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.322174 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7wm2\" (UniqueName: \"kubernetes.io/projected/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-kube-api-access-d7wm2\") pod \"maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv\" (UID: \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" Nov 02 18:02:14.326475 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.326449 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4952505b-a6ee-469c-b3ad-39f897429fb4" path="/var/lib/kubelet/pods/4952505b-a6ee-469c-b3ad-39f897429fb4/volumes" Nov 02 18:02:14.327514 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.327492 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5" path="/var/lib/kubelet/pods/9e748d4a-ed12-40fc-bbd0-7a85ef51b0b5/volumes" Nov 02 18:02:14.414135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.414045 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:14.441309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.441273 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" Nov 02 18:02:14.864284 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:14.853185 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv"] Nov 02 18:02:14.880000 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:02:14.877176 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca8565f_3d0f_4154_ba3f_9e99a38af3fe.slice/crio-2886fddf46de6ebdc86ecd027a1ccbae933ab73b24e24211364be0853dd71b46 WatchSource:0}: Error finding container 2886fddf46de6ebdc86ecd027a1ccbae933ab73b24e24211364be0853dd71b46: Status 404 returned error can't find the container with id 2886fddf46de6ebdc86ecd027a1ccbae933ab73b24e24211364be0853dd71b46 Nov 02 18:02:15.230731 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:15.230687 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn"] Nov 02 18:02:15.256572 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:15.256539 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" event={"ID":"dca8565f-3d0f-4154-ba3f-9e99a38af3fe","Type":"ContainerStarted","Data":"2886fddf46de6ebdc86ecd027a1ccbae933ab73b24e24211364be0853dd71b46"} Nov 02 18:02:15.274913 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:15.274868 5549 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Nov 02 18:02:15.275125 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:15.274992 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" podUID="fa70aff2-1e86-4d0c-a95a-57cc50537c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Nov 02 18:02:15.339003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:15.316880 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:15.497190 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:15.496931 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:15.806080 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:15.765715 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:16.233002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:16.229765 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:16.276986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:16.275267 5549 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Nov 02 18:02:16.276986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:16.275333 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" podUID="fa70aff2-1e86-4d0c-a95a-57cc50537c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Nov 02 18:02:16.325043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:16.322687 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" event={"ID":"dca8565f-3d0f-4154-ba3f-9e99a38af3fe","Type":"ContainerStarted","Data":"55377d9e1845e0ddbad7b2b80fa68b60ad756d9d6ce052b3075681f735b3361a"} Nov 02 18:02:16.325043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:16.322770 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" Nov 02 18:02:16.364158 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:16.364093 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" podStartSLOduration=1.841551234 podStartE2EDuration="2.364071379s" totalImagesPullingTime="522.520145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:16.357895429 +0000 UTC m=+1708.146400969" watchObservedRunningTime="2026-11-02 18:02:16.364071379 +0000 UTC m=+1708.152576910" Nov 02 18:02:17.275023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:17.274985 5549 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Nov 02 18:02:17.275621 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:17.275038 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" podUID="fa70aff2-1e86-4d0c-a95a-57cc50537c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Nov 02 18:02:17.432976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:17.427325 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:17.619105 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:02:17.619016 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:02:17.845256 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:17.845221 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:18.005388 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:18.005335 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:18.285085 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:18.279648 5549 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Nov 02 18:02:18.285085 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:18.279746 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" podUID="fa70aff2-1e86-4d0c-a95a-57cc50537c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Nov 02 18:02:18.431669 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:18.431618 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:19.235199 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:19.235161 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:19.275559 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:19.275523 5549 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Nov 02 18:02:19.275752 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:19.275590 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" podUID="fa70aff2-1e86-4d0c-a95a-57cc50537c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Nov 02 18:02:19.606914 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:19.606814 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:20.017760 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:20.013261 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:20.275768 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:20.275672 5549 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Nov 02 18:02:20.275768 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:20.275735 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" podUID="fa70aff2-1e86-4d0c-a95a-57cc50537c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Nov 02 18:02:20.790938 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:20.790902 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:20.814283 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:20.814242 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8"] Nov 02 18:02:20.823508 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:20.823477 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8"] Nov 02 18:02:20.823771 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:20.823613 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:20.904974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:20.901304 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-5ec436-openshift-default-dockercfg-wqbgt\"" Nov 02 18:02:20.939985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:20.931847 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:20.940201 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:20.940170 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:20.940277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:20.940214 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468e1807-5ca2-4bef-8674-0879f2b3a77e-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:20.940340 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:20.940315 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:20.940398 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:20.940351 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:20.940448 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:20.940393 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:20.940537 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:20.940518 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:20.940679 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:20.940660 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:20.940983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:20.940909 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svrpj\" (UniqueName: \"kubernetes.io/projected/468e1807-5ca2-4bef-8674-0879f2b3a77e-kube-api-access-svrpj\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:20.965674 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:20.965604 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:21.043382 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.043282 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.043382 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.043356 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468e1807-5ca2-4bef-8674-0879f2b3a77e-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.043649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.043445 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.043649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.043478 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.043649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.043512 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.043649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.043617 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.043649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.043640 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.043924 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.043745 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svrpj\" (UniqueName: \"kubernetes.io/projected/468e1807-5ca2-4bef-8674-0879f2b3a77e-kube-api-access-svrpj\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.043924 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.043776 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.043924 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.043814 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.044122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.043946 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.044122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.044074 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.044233 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.044201 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468e1807-5ca2-4bef-8674-0879f2b3a77e-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.044410 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.044385 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.046637 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.046607 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.047364 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.047340 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.052048 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.052027 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.052621 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.052598 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svrpj\" (UniqueName: \"kubernetes.io/projected/468e1807-5ca2-4bef-8674-0879f2b3a77e-kube-api-access-svrpj\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.142385 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.142345 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:21.336704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.336566 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:21.337275 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.337253 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:21.338276 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.338252 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:02:21.447646 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.447613 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8"] Nov 02 18:02:21.448115 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:02:21.448084 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468e1807_5ca2_4bef_8674_0879f2b3a77e.slice/crio-eef08e048e9fb49e3fc36555ac68a09db02c9d904254664117f884908dc62551 WatchSource:0}: Error finding container eef08e048e9fb49e3fc36555ac68a09db02c9d904254664117f884908dc62551: Status 404 returned error can't find the container with id eef08e048e9fb49e3fc36555ac68a09db02c9d904254664117f884908dc62551 Nov 02 18:02:21.455589 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.455550 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:21.455695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.455628 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:21.455695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.455660 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:21.658606 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.658511 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" event={"ID":"468e1807-5ca2-4bef-8674-0879f2b3a77e","Type":"ContainerStarted","Data":"48eeaee54e05500ad1e69249059ca6c5f245d875d0a0165926f2f4f87332e7f8"} Nov 02 18:02:21.658606 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.658578 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" event={"ID":"468e1807-5ca2-4bef-8674-0879f2b3a77e","Type":"ContainerStarted","Data":"eef08e048e9fb49e3fc36555ac68a09db02c9d904254664117f884908dc62551"} Nov 02 18:02:21.683583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.683518 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" podStartSLOduration=1.683501006 podStartE2EDuration="1.683501006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:21.681780755 +0000 UTC m=+1713.470286296" watchObservedRunningTime="2026-11-02 18:02:21.683501006 +0000 UTC m=+1713.472006536" Nov 02 18:02:21.845985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:21.845641 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:22.142546 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:22.142503 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:22.144167 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:22.144145 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Nov 02 18:02:22.144243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:22.144194 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" podUID="468e1807-5ca2-4bef-8674-0879f2b3a77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Nov 02 18:02:22.617974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:22.606152 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:23.150991 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:23.143447 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Nov 02 18:02:23.150991 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:23.143515 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" podUID="468e1807-5ca2-4bef-8674-0879f2b3a77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Nov 02 18:02:23.316977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:23.316919 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:23.547978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:23.546225 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" Nov 02 18:02:23.621985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:23.614845 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:24.143620 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:24.143586 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Nov 02 18:02:24.143801 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:24.143638 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" podUID="468e1807-5ca2-4bef-8674-0879f2b3a77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Nov 02 18:02:24.559537 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:24.559500 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:24.715378 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:24.715343 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:25.018398 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:25.018364 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:25.183981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:25.158820 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Nov 02 18:02:25.183981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:25.162035 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" podUID="468e1807-5ca2-4bef-8674-0879f2b3a77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Nov 02 18:02:25.781973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:25.778505 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:26.012973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:26.012065 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:26.142888 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:26.142799 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Nov 02 18:02:26.142888 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:26.142854 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" podUID="468e1807-5ca2-4bef-8674-0879f2b3a77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Nov 02 18:02:26.593241 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:26.593187 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:27.120109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:27.120073 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:02:27.143051 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:27.143014 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Nov 02 18:02:27.143238 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:27.143077 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" podUID="468e1807-5ca2-4bef-8674-0879f2b3a77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Nov 02 18:02:27.390132 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:27.390019 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:27.712776 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:27.712738 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:28.148026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:28.147502 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Nov 02 18:02:28.148026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:28.147577 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" podUID="468e1807-5ca2-4bef-8674-0879f2b3a77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Nov 02 18:02:28.420309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:28.420188 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:28.575151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:28.575109 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:28.729007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:28.726881 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:28.996021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:28.994676 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:29.148181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:29.148146 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:29.366676 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:29.366590 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:29.391982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:29.375249 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:29.414120 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:29.414088 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:02:30.301068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:30.288420 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:30.324974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:30.318570 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" podUID="501cea11-004b-4161-b64f-552baabdd8bd" containerName="main" containerID="cri-o://9e557c1a1ba927f2b0e5ff4dab799775ffaef0b9ea34d7cb55721148dd3a9aa9" gracePeriod=45 Nov 02 18:02:30.905132 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:30.904212 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:30.991434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:30.991385 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:02:31.238489 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:31.230765 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" podUID="501cea11-004b-4161-b64f-552baabdd8bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.224:8000/ready\": EOF" Nov 02 18:02:31.238489 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:31.231743 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" podUID="501cea11-004b-4161-b64f-552baabdd8bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.224:8000/ready\": dial tcp 10.133.0.224:8000: connect: connection refused" Nov 02 18:02:31.238489 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:31.231840 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:31.272220 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:31.272179 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:02:31.317127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:31.293477 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:02:31.849217 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:31.842094 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5"] Nov 02 18:02:32.121055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.099380 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" Nov 02 18:02:32.141061 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.125018 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5"] Nov 02 18:02:32.186451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.186414 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:32.239034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.187013 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"routing-overlay\"" Nov 02 18:02:32.239034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.187774 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"payload-processing-external-model-e2e--69602eb2-dockercfg-ffvnd\"" Nov 02 18:02:32.239034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.187827 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"agc-cred-d1a3311a2190\"" Nov 02 18:02:32.239034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.187898 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"payload-processing-external-model-plugins-e2e-extmod-b-327608\"" Nov 02 18:02:32.239034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.208923 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90faafa6-7183-4a82-86cc-17721fcf2368-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5\" (UID: \"90faafa6-7183-4a82-86cc-17721fcf2368\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" Nov 02 18:02:32.239034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.222343 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90faafa6-7183-4a82-86cc-17721fcf2368-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5\" (UID: \"90faafa6-7183-4a82-86cc-17721fcf2368\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" Nov 02 18:02:32.239034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.222520 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5plx\" (UniqueName: \"kubernetes.io/projected/90faafa6-7183-4a82-86cc-17721fcf2368-kube-api-access-t5plx\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5\" (UID: \"90faafa6-7183-4a82-86cc-17721fcf2368\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" Nov 02 18:02:32.239034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.222641 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90faafa6-7183-4a82-86cc-17721fcf2368-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5\" (UID: \"90faafa6-7183-4a82-86cc-17721fcf2368\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" Nov 02 18:02:32.282111 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.282070 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"openshift-service-ca.crt\"" Nov 02 18:02:32.282519 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.282501 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"kube-root-ca.crt\"" Nov 02 18:02:32.282763 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.282739 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" podUID="501cea11-004b-4161-b64f-552baabdd8bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.224:8000/ready\": dial tcp 10.133.0.224:8000: connect: connection refused" Nov 02 18:02:32.443207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.421492 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90faafa6-7183-4a82-86cc-17721fcf2368-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5\" (UID: \"90faafa6-7183-4a82-86cc-17721fcf2368\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" Nov 02 18:02:32.443207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.421692 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5plx\" (UniqueName: \"kubernetes.io/projected/90faafa6-7183-4a82-86cc-17721fcf2368-kube-api-access-t5plx\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5\" (UID: \"90faafa6-7183-4a82-86cc-17721fcf2368\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" Nov 02 18:02:32.443207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.421843 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90faafa6-7183-4a82-86cc-17721fcf2368-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5\" (UID: \"90faafa6-7183-4a82-86cc-17721fcf2368\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" Nov 02 18:02:32.443207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.422193 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90faafa6-7183-4a82-86cc-17721fcf2368-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5\" (UID: \"90faafa6-7183-4a82-86cc-17721fcf2368\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" Nov 02 18:02:32.443207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.423250 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90faafa6-7183-4a82-86cc-17721fcf2368-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5\" (UID: \"90faafa6-7183-4a82-86cc-17721fcf2368\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" Nov 02 18:02:32.443207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.424109 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90faafa6-7183-4a82-86cc-17721fcf2368-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5\" (UID: \"90faafa6-7183-4a82-86cc-17721fcf2368\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" Nov 02 18:02:32.443207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.430271 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90faafa6-7183-4a82-86cc-17721fcf2368-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5\" (UID: \"90faafa6-7183-4a82-86cc-17721fcf2368\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" Nov 02 18:02:32.443207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.433425 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5plx\" (UniqueName: \"kubernetes.io/projected/90faafa6-7183-4a82-86cc-17721fcf2368-kube-api-access-t5plx\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5\" (UID: \"90faafa6-7183-4a82-86cc-17721fcf2368\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" Nov 02 18:02:32.636296 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.635181 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:32.674076 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.674047 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" Nov 02 18:02:33.009024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:32.970015 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:33.484256 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:33.484222 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z"] Nov 02 18:02:33.565333 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:33.565276 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:33.601978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:33.574190 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z"] Nov 02 18:02:33.601978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:33.574416 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" Nov 02 18:02:33.601978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:33.595402 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726"] Nov 02 18:02:33.609249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:33.609216 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726"] Nov 02 18:02:33.609564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:33.609549 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726" Nov 02 18:02:33.621176 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:33.621149 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-bee8f6-dockercfg-lvn4q\"" Nov 02 18:02:33.621445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:33.621426 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-bee8f6\"" Nov 02 18:02:33.659428 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:33.659339 5549 generic.go:356] "Generic (PLEG): container finished" podID="501cea11-004b-4161-b64f-552baabdd8bd" containerID="9e557c1a1ba927f2b0e5ff4dab799775ffaef0b9ea34d7cb55721148dd3a9aa9" exitCode=0 Nov 02 18:02:33.659687 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:33.659662 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" event={"ID":"501cea11-004b-4161-b64f-552baabdd8bd","Type":"ContainerDied","Data":"9e557c1a1ba927f2b0e5ff4dab799775ffaef0b9ea34d7cb55721148dd3a9aa9"} Nov 02 18:02:33.659902 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:33.659747 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:33.806265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:33.803901 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5"] Nov 02 18:02:33.848974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:33.807180 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-model-cache\") pod \"501cea11-004b-4161-b64f-552baabdd8bd\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " Nov 02 18:02:34.075978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.075417 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-home\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-home\") pod \"501cea11-004b-4161-b64f-552baabdd8bd\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " Nov 02 18:02:34.075978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.075528 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-tmp-dir\") pod \"501cea11-004b-4161-b64f-552baabdd8bd\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " Nov 02 18:02:34.075978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.075613 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/501cea11-004b-4161-b64f-552baabdd8bd-kube-api-access-rjt2k\" (UniqueName: \"kubernetes.io/projected/501cea11-004b-4161-b64f-552baabdd8bd-kube-api-access-rjt2k\") pod \"501cea11-004b-4161-b64f-552baabdd8bd\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " Nov 02 18:02:34.075978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.075791 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/501cea11-004b-4161-b64f-552baabdd8bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/501cea11-004b-4161-b64f-552baabdd8bd-tls-certs\") pod \"501cea11-004b-4161-b64f-552baabdd8bd\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " Nov 02 18:02:34.075978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.011892 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:34.076293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.076008 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-dshm\") pod \"501cea11-004b-4161-b64f-552baabdd8bd\" (UID: \"501cea11-004b-4161-b64f-552baabdd8bd\") " Nov 02 18:02:34.076293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:33.856248 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-model-cache" pod "501cea11-004b-4161-b64f-552baabdd8bd" (UID: "501cea11-004b-4161-b64f-552baabdd8bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:34.076362 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.076326 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-home" pod "501cea11-004b-4161-b64f-552baabdd8bd" (UID: "501cea11-004b-4161-b64f-552baabdd8bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:34.076792 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.076756 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdfbd7a-1637-47c2-b84e-abfb6319483a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z\" (UID: \"afdfbd7a-1637-47c2-b84e-abfb6319483a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" Nov 02 18:02:34.077161 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.077136 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kshg\" (UniqueName: \"kubernetes.io/projected/bbffe25b-3628-4d4a-a7a8-af2eb246aaba-kube-api-access-7kshg\") pod \"payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726\" (UID: \"bbffe25b-3628-4d4a-a7a8-af2eb246aaba\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726" Nov 02 18:02:34.077254 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.077235 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbffe25b-3628-4d4a-a7a8-af2eb246aaba-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726\" (UID: \"bbffe25b-3628-4d4a-a7a8-af2eb246aaba\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726" Nov 02 18:02:34.077385 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.077365 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f85qc\" (UniqueName: \"kubernetes.io/projected/afdfbd7a-1637-47c2-b84e-abfb6319483a-kube-api-access-f85qc\") pod \"payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z\" (UID: \"afdfbd7a-1637-47c2-b84e-abfb6319483a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" Nov 02 18:02:34.077692 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.077665 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:34.077748 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.077701 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:34.130138 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.091431 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-tmp-dir" pod "501cea11-004b-4161-b64f-552baabdd8bd" (UID: "501cea11-004b-4161-b64f-552baabdd8bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:34.130138 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.114691 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/501cea11-004b-4161-b64f-552baabdd8bd-tls-certs" pod "501cea11-004b-4161-b64f-552baabdd8bd" (UID: "501cea11-004b-4161-b64f-552baabdd8bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:02:34.130138 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.114808 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-dshm" pod "501cea11-004b-4161-b64f-552baabdd8bd" (UID: "501cea11-004b-4161-b64f-552baabdd8bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:34.130138 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.115126 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501cea11-004b-4161-b64f-552baabdd8bd-kube-api-access-rjt2k" pod "501cea11-004b-4161-b64f-552baabdd8bd" (UID: "501cea11-004b-4161-b64f-552baabdd8bd"). InnerVolumeSpecName "kube-api-access-rjt2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:34.189979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.188115 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbffe25b-3628-4d4a-a7a8-af2eb246aaba-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726\" (UID: \"bbffe25b-3628-4d4a-a7a8-af2eb246aaba\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726" Nov 02 18:02:34.189979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.188275 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f85qc\" (UniqueName: \"kubernetes.io/projected/afdfbd7a-1637-47c2-b84e-abfb6319483a-kube-api-access-f85qc\") pod \"payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z\" (UID: \"afdfbd7a-1637-47c2-b84e-abfb6319483a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" Nov 02 18:02:34.189979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.188423 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdfbd7a-1637-47c2-b84e-abfb6319483a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z\" (UID: \"afdfbd7a-1637-47c2-b84e-abfb6319483a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" Nov 02 18:02:34.189979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.188742 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kshg\" (UniqueName: \"kubernetes.io/projected/bbffe25b-3628-4d4a-a7a8-af2eb246aaba-kube-api-access-7kshg\") pod \"payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726\" (UID: \"bbffe25b-3628-4d4a-a7a8-af2eb246aaba\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726" Nov 02 18:02:34.189979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.188848 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:34.189979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.188868 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjt2k\" (UniqueName: \"kubernetes.io/projected/501cea11-004b-4161-b64f-552baabdd8bd-kube-api-access-rjt2k\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:34.189979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.188886 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501cea11-004b-4161-b64f-552baabdd8bd-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:34.189979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.188901 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/501cea11-004b-4161-b64f-552baabdd8bd-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:34.213001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.211268 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdfbd7a-1637-47c2-b84e-abfb6319483a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z\" (UID: \"afdfbd7a-1637-47c2-b84e-abfb6319483a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" Nov 02 18:02:34.234905 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.218478 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbffe25b-3628-4d4a-a7a8-af2eb246aaba-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726\" (UID: \"bbffe25b-3628-4d4a-a7a8-af2eb246aaba\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726" Nov 02 18:02:34.250220 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.250186 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f85qc\" (UniqueName: \"kubernetes.io/projected/afdfbd7a-1637-47c2-b84e-abfb6319483a-kube-api-access-f85qc\") pod \"payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z\" (UID: \"afdfbd7a-1637-47c2-b84e-abfb6319483a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" Nov 02 18:02:34.255362 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.254370 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kshg\" (UniqueName: \"kubernetes.io/projected/bbffe25b-3628-4d4a-a7a8-af2eb246aaba-kube-api-access-7kshg\") pod \"payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726\" (UID: \"bbffe25b-3628-4d4a-a7a8-af2eb246aaba\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726" Nov 02 18:02:34.333439 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.333396 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" Nov 02 18:02:34.376403 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.376383 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726" Nov 02 18:02:34.674443 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.674403 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" event={"ID":"90faafa6-7183-4a82-86cc-17721fcf2368","Type":"ContainerStarted","Data":"de8e87f17050edae24a761da66bc431590f89bf67047cc33dae7afacd924cbf1"} Nov 02 18:02:34.675338 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.674458 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" event={"ID":"90faafa6-7183-4a82-86cc-17721fcf2368","Type":"ContainerStarted","Data":"e43e054fd33acf079c05d6ed9e9e8dac5b4346e69c65b956867aba3ba6971789"} Nov 02 18:02:34.675527 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.675508 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" Nov 02 18:02:34.677087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.677063 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" event={"ID":"501cea11-004b-4161-b64f-552baabdd8bd","Type":"ContainerDied","Data":"beee01123e1759f07606eac4b61a1be41afd027b5c564b2c8ecc50925abd0d9f"} Nov 02 18:02:34.677218 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.677101 5549 scope.go:122] "RemoveContainer" containerID="9e557c1a1ba927f2b0e5ff4dab799775ffaef0b9ea34d7cb55721148dd3a9aa9" Nov 02 18:02:34.677293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.677228 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn" Nov 02 18:02:34.760964 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.760901 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" podStartSLOduration=3.760886502 podStartE2EDuration="3.760886502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:34.732226066 +0000 UTC m=+1726.520731595" watchObservedRunningTime="2026-11-02 18:02:34.760886502 +0000 UTC m=+1726.549392030" Nov 02 18:02:34.762103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.762082 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn"] Nov 02 18:02:34.770807 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.770744 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-4ea582/autores-model-4ea582-kserve-7fbd76c949-cklqn"] Nov 02 18:02:34.892710 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.892677 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:34.907181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:34.902221 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z"] Nov 02 18:02:35.057398 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:35.057363 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726"] Nov 02 18:02:35.062886 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:02:35.062258 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffe25b_3628_4d4a_a7a8_af2eb246aaba.slice/crio-c4072f4dd453dff1eb8bde0b680c1a4088a88231768c5ea488fabc1e613e9371 WatchSource:0}: Error finding container c4072f4dd453dff1eb8bde0b680c1a4088a88231768c5ea488fabc1e613e9371: Status 404 returned error can't find the container with id c4072f4dd453dff1eb8bde0b680c1a4088a88231768c5ea488fabc1e613e9371 Nov 02 18:02:35.728042 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:35.727403 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726" event={"ID":"bbffe25b-3628-4d4a-a7a8-af2eb246aaba","Type":"ContainerStarted","Data":"f07c81a75f3d62189fc5639d54c901615ddab87c4bf91f2bb6b4ad2807336e0a"} Nov 02 18:02:35.728042 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:35.727474 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726" event={"ID":"bbffe25b-3628-4d4a-a7a8-af2eb246aaba","Type":"ContainerStarted","Data":"c4072f4dd453dff1eb8bde0b680c1a4088a88231768c5ea488fabc1e613e9371"} Nov 02 18:02:35.728769 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:35.728713 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726" Nov 02 18:02:35.733936 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:35.733905 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" event={"ID":"afdfbd7a-1637-47c2-b84e-abfb6319483a","Type":"ContainerStarted","Data":"108926dafb2221310167071da2e527eb7c2a3da5e04508c301b365f808f802a3"} Nov 02 18:02:35.734168 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:35.733998 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" Nov 02 18:02:35.734168 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:35.734027 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" event={"ID":"afdfbd7a-1637-47c2-b84e-abfb6319483a","Type":"ContainerStarted","Data":"00d55ac3e660ca056b7bab2e32db99952598fe414fdb89fa66ff7c1ff786a866"} Nov 02 18:02:35.758014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:35.757966 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726" podStartSLOduration=3.757943089 podStartE2EDuration="3.757943089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:35.75165522 +0000 UTC m=+1727.540160747" watchObservedRunningTime="2026-11-02 18:02:35.757943089 +0000 UTC m=+1727.546448617" Nov 02 18:02:35.778979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:35.778894 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" podStartSLOduration=3.778881263 podStartE2EDuration="3.778881263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:35.773871354 +0000 UTC m=+1727.562376883" watchObservedRunningTime="2026-11-02 18:02:35.778881263 +0000 UTC m=+1727.567386790" Nov 02 18:02:36.006008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:36.005895 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:36.346482 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:36.346392 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="501cea11-004b-4161-b64f-552baabdd8bd" path="/var/lib/kubelet/pods/501cea11-004b-4161-b64f-552baabdd8bd/volumes" Nov 02 18:02:36.911360 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:36.911277 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:37.612038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:37.607765 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:38.214613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:38.214576 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:38.911553 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:38.909825 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:39.741973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:39.741466 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:40.696530 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:40.696490 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:41.554983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:41.546569 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:41.810402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:41.808298 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": read tcp 10.133.0.2:50048->10.133.0.213:15021: read: connection reset by peer" start-of-body= Nov 02 18:02:41.810402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:41.808358 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" podUID="7669f8e2-f625-4276-9726-d260732ffa7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": read tcp 10.133.0.2:50048->10.133.0.213:15021: read: connection reset by peer" Nov 02 18:02:41.842369 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:41.842338 5549 generic.go:356] "Generic (PLEG): container finished" podID="7669f8e2-f625-4276-9726-d260732ffa7e" containerID="a9b74124eb89d95ab2ee6f2cc22707845716c1f2e8ab7abad140ea0ccf9bc514" exitCode=0 Nov 02 18:02:41.842531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:41.842410 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" event={"ID":"7669f8e2-f625-4276-9726-d260732ffa7e","Type":"ContainerDied","Data":"a9b74124eb89d95ab2ee6f2cc22707845716c1f2e8ab7abad140ea0ccf9bc514"} Nov 02 18:02:41.842923 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:41.842907 5549 scope.go:122] "RemoveContainer" containerID="a9b74124eb89d95ab2ee6f2cc22707845716c1f2e8ab7abad140ea0ccf9bc514" Nov 02 18:02:41.845087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:41.845054 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:41.845168 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:41.845133 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:41.845211 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:41.845172 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:42.285602 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:42.285565 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:42.431837 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:42.431700 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:42.876999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:42.876920 5549 generic.go:356] "Generic (PLEG): container finished" podID="d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" containerID="f4829e725c79e563c30f2e2294bd3d0ffc65d6cdb1e34e4bebd3c1da6cf10254" exitCode=0 Nov 02 18:02:42.877837 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:42.877615 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" event={"ID":"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23","Type":"ContainerDied","Data":"f4829e725c79e563c30f2e2294bd3d0ffc65d6cdb1e34e4bebd3c1da6cf10254"} Nov 02 18:02:42.897373 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:42.896331 5549 scope.go:122] "RemoveContainer" containerID="f4829e725c79e563c30f2e2294bd3d0ffc65d6cdb1e34e4bebd3c1da6cf10254" Nov 02 18:02:42.922975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:42.900324 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:42.922975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:42.914288 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:42.922975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:42.914391 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:43.052988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:43.038397 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" event={"ID":"7669f8e2-f625-4276-9726-d260732ffa7e","Type":"ContainerStarted","Data":"4517d8051e757f479ac49d66444b0fd5c9ce4d7dec3c5f86ad0a84cf84824ff5"} Nov 02 18:02:43.653609 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:43.653522 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:43.809391 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:43.809352 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:02:43.809744 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:43.809722 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Nov 02 18:02:43.809889 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:43.809866 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" podUID="7669f8e2-f625-4276-9726-d260732ffa7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Nov 02 18:02:44.067231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:44.067184 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" event={"ID":"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23","Type":"ContainerStarted","Data":"a33d40a5260c77ad0c8ff115e6beeb25f6faec27c35092cb0509ac125985e055"} Nov 02 18:02:44.603218 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:44.603179 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:44.788418 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:44.788383 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:02:44.788968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:44.788824 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Nov 02 18:02:44.789271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:44.789245 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" podUID="d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Nov 02 18:02:44.807421 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:44.807374 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Nov 02 18:02:44.807718 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:44.807688 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" podUID="7669f8e2-f625-4276-9726-d260732ffa7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Nov 02 18:02:45.372592 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:45.372543 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:45.794398 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:45.794356 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Nov 02 18:02:45.796300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:45.796260 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" podUID="d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Nov 02 18:02:45.805400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:45.805364 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Nov 02 18:02:45.805653 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:45.805623 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" podUID="7669f8e2-f625-4276-9726-d260732ffa7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Nov 02 18:02:46.031836 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:46.031790 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:46.204030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:46.200483 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:46.747215 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:46.747177 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" Nov 02 18:02:46.747627 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:46.747289 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" Nov 02 18:02:46.790372 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:46.790335 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Nov 02 18:02:46.790654 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:46.790619 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" podUID="d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Nov 02 18:02:46.838828 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:46.836091 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Nov 02 18:02:46.838828 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:46.836169 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" podUID="7669f8e2-f625-4276-9726-d260732ffa7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Nov 02 18:02:47.016041 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:47.013236 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:47.194480 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:47.194447 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:47.336323 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:47.336232 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:47.498580 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:47.498543 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:47.664684 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:02:47.664605 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:02:47.761761 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:47.761724 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726" Nov 02 18:02:47.795636 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:47.795598 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Nov 02 18:02:47.795941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:47.795906 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" podUID="d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Nov 02 18:02:47.803473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:47.803433 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:47.821321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:47.809338 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Nov 02 18:02:47.821321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:47.809429 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" podUID="7669f8e2-f625-4276-9726-d260732ffa7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Nov 02 18:02:47.961744 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:47.961615 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:48.122433 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:48.122394 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:48.659737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:48.659694 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:48.833978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:48.828639 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Nov 02 18:02:48.833978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:48.828697 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" podUID="d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Nov 02 18:02:48.833978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:48.830930 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:48.833978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:48.831547 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Nov 02 18:02:48.833978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:48.831589 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" podUID="7669f8e2-f625-4276-9726-d260732ffa7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Nov 02 18:02:49.411079 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:49.411037 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:49.611399 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:49.611361 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:49.795023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:49.792500 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Nov 02 18:02:49.795023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:49.792589 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" podUID="d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Nov 02 18:02:49.928976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:49.924245 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:02:49.928976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:49.927416 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:02:50.089505 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:50.089416 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:02:50.169219 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:50.169185 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:50.628979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:50.623524 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:50.871491 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:50.871454 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:02:50.872039 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:50.872008 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:02:50.872833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:50.872815 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:02:51.314990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:51.312994 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:51.942038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:51.941504 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:54.893793 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:54.893755 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:55.210963 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:55.210921 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:55.957971 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:55.954796 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:56.631019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:56.629099 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:57.371567 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:57.371528 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:57.931746 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:57.931707 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk"] Nov 02 18:02:57.954456 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:57.954424 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck"] Nov 02 18:02:58.059443 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:58.059408 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:58.188607 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:58.188527 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:58.505473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:58.505436 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:58.738180 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:58.738152 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:59.391937 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:59.391826 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:59.522295 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:02:59.522261 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:00.134770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:00.134727 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:00.726377 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:00.726340 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j"] Nov 02 18:03:00.726877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:00.726594 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" podUID="3a54d68a-aef4-4f05-ba0c-3571ca39d6a9" containerName="maas-api" containerID="cri-o://5691e1470e5f546fc3848aef2a8e042e78374d05926a73bcb3cd63b4b7396e3d" gracePeriod=30 Nov 02 18:03:00.819883 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:00.819850 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:01.307976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.302797 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:01.402472 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.402438 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" Nov 02 18:03:01.441359 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.435975 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-openshift-service-ca\") pod \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\" (UID: \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\") " Nov 02 18:03:01.441359 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.436197 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-maas-api-tls\") pod \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\" (UID: \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\") " Nov 02 18:03:01.441359 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.436375 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-metrics-certs\") pod \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\" (UID: \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\") " Nov 02 18:03:01.441359 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.436490 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-kube-api-access-94ssg\" (UniqueName: \"kubernetes.io/projected/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-kube-api-access-94ssg\") pod \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\" (UID: \"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9\") " Nov 02 18:03:01.446688 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.446299 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-openshift-service-ca" pod "3a54d68a-aef4-4f05-ba0c-3571ca39d6a9" (UID: "3a54d68a-aef4-4f05-ba0c-3571ca39d6a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:03:01.457569 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.457513 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-kube-api-access-94ssg" pod "3a54d68a-aef4-4f05-ba0c-3571ca39d6a9" (UID: "3a54d68a-aef4-4f05-ba0c-3571ca39d6a9"). InnerVolumeSpecName "kube-api-access-94ssg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:01.459210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.459164 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-maas-api-tls" pod "3a54d68a-aef4-4f05-ba0c-3571ca39d6a9" (UID: "3a54d68a-aef4-4f05-ba0c-3571ca39d6a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:03:01.459462 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.459444 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-metrics-certs" pod "3a54d68a-aef4-4f05-ba0c-3571ca39d6a9" (UID: "3a54d68a-aef4-4f05-ba0c-3571ca39d6a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:03:01.495815 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.495722 5549 generic.go:356] "Generic (PLEG): container finished" podID="3a54d68a-aef4-4f05-ba0c-3571ca39d6a9" containerID="5691e1470e5f546fc3848aef2a8e042e78374d05926a73bcb3cd63b4b7396e3d" exitCode=0 Nov 02 18:03:01.495815 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.495766 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" Nov 02 18:03:01.495815 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.495767 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" event={"ID":"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9","Type":"ContainerDied","Data":"5691e1470e5f546fc3848aef2a8e042e78374d05926a73bcb3cd63b4b7396e3d"} Nov 02 18:03:01.496166 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.495840 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" event={"ID":"3a54d68a-aef4-4f05-ba0c-3571ca39d6a9","Type":"ContainerDied","Data":"ee19324a4dafb53d2f792ce388922b53a807aa710a1cd17a9fb445ea2d38f122"} Nov 02 18:03:01.496166 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.495872 5549 scope.go:122] "RemoveContainer" containerID="5691e1470e5f546fc3848aef2a8e042e78374d05926a73bcb3cd63b4b7396e3d" Nov 02 18:03:01.509028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.509008 5549 scope.go:122] "RemoveContainer" containerID="5691e1470e5f546fc3848aef2a8e042e78374d05926a73bcb3cd63b4b7396e3d" Nov 02 18:03:01.509380 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:03:01.509359 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5691e1470e5f546fc3848aef2a8e042e78374d05926a73bcb3cd63b4b7396e3d\": container with ID starting with 5691e1470e5f546fc3848aef2a8e042e78374d05926a73bcb3cd63b4b7396e3d not found: ID does not exist" containerID="5691e1470e5f546fc3848aef2a8e042e78374d05926a73bcb3cd63b4b7396e3d" Nov 02 18:03:01.509462 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.509393 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5691e1470e5f546fc3848aef2a8e042e78374d05926a73bcb3cd63b4b7396e3d"} err="failed to get container status \"5691e1470e5f546fc3848aef2a8e042e78374d05926a73bcb3cd63b4b7396e3d\": rpc error: code = NotFound desc = could not find container \"5691e1470e5f546fc3848aef2a8e042e78374d05926a73bcb3cd63b4b7396e3d\": container with ID starting with 5691e1470e5f546fc3848aef2a8e042e78374d05926a73bcb3cd63b4b7396e3d not found: ID does not exist" Nov 02 18:03:01.524467 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.524413 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j"] Nov 02 18:03:01.529345 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.529314 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j"] Nov 02 18:03:01.544393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.544360 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:01.544393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.544390 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:01.544659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.544402 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94ssg\" (UniqueName: \"kubernetes.io/projected/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-kube-api-access-94ssg\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:01.544659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:01.544412 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:02.159267 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:02.159179 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:02.323170 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:02.323130 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a54d68a-aef4-4f05-ba0c-3571ca39d6a9" path="/var/lib/kubelet/pods/3a54d68a-aef4-4f05-ba0c-3571ca39d6a9/volumes" Nov 02 18:03:02.932719 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:02.932676 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk" podUID="b077a6d9-5d78-4d91-8223-89244863b8e5" containerName="payload-processing" containerID="cri-o://9f4d9a4eeb1cf97c9c23f0142ee94f9d02a7c1d74caedb3cfc737e79e86f3743" gracePeriod=25 Nov 02 18:03:02.971496 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:02.970674 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck" podUID="7639dacc-94de-425d-8ba4-3fa1baa67fb1" containerName="payload-pre-processing" containerID="cri-o://614b3669c8b417be4d96d3bbbabe9299cb281d105ad661882990db086e9d0de7" gracePeriod=25 Nov 02 18:03:03.032976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.009640 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:03.106141 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:03:03.106108 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7639dacc_94de_425d_8ba4_3fa1baa67fb1.slice/crio-conmon-614b3669c8b417be4d96d3bbbabe9299cb281d105ad661882990db086e9d0de7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7639dacc_94de_425d_8ba4_3fa1baa67fb1.slice/crio-614b3669c8b417be4d96d3bbbabe9299cb281d105ad661882990db086e9d0de7.scope\": RecentStats: unable to find data in memory cache]" Nov 02 18:03:03.106331 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:03:03.106103 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7639dacc_94de_425d_8ba4_3fa1baa67fb1.slice/crio-conmon-614b3669c8b417be4d96d3bbbabe9299cb281d105ad661882990db086e9d0de7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7639dacc_94de_425d_8ba4_3fa1baa67fb1.slice/crio-614b3669c8b417be4d96d3bbbabe9299cb281d105ad661882990db086e9d0de7.scope\": RecentStats: unable to find data in memory cache]" Nov 02 18:03:03.295811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.295778 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk" Nov 02 18:03:03.351707 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.351679 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck" Nov 02 18:03:03.380142 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.380106 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7639dacc-94de-425d-8ba4-3fa1baa67fb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7639dacc-94de-425d-8ba4-3fa1baa67fb1-plugins-config-volume\") pod \"7639dacc-94de-425d-8ba4-3fa1baa67fb1\" (UID: \"7639dacc-94de-425d-8ba4-3fa1baa67fb1\") " Nov 02 18:03:03.380339 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.380307 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b077a6d9-5d78-4d91-8223-89244863b8e5-kube-api-access-hswkd\" (UniqueName: \"kubernetes.io/projected/b077a6d9-5d78-4d91-8223-89244863b8e5-kube-api-access-hswkd\") pod \"b077a6d9-5d78-4d91-8223-89244863b8e5\" (UID: \"b077a6d9-5d78-4d91-8223-89244863b8e5\") " Nov 02 18:03:03.380421 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.380337 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7639dacc-94de-425d-8ba4-3fa1baa67fb1-kube-api-access-vb9dd\" (UniqueName: \"kubernetes.io/projected/7639dacc-94de-425d-8ba4-3fa1baa67fb1-kube-api-access-vb9dd\") pod \"7639dacc-94de-425d-8ba4-3fa1baa67fb1\" (UID: \"7639dacc-94de-425d-8ba4-3fa1baa67fb1\") " Nov 02 18:03:03.380506 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.380467 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7639dacc-94de-425d-8ba4-3fa1baa67fb1-plugins-config-volume" pod "7639dacc-94de-425d-8ba4-3fa1baa67fb1" (UID: "7639dacc-94de-425d-8ba4-3fa1baa67fb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:03:03.380570 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.380502 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b077a6d9-5d78-4d91-8223-89244863b8e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b077a6d9-5d78-4d91-8223-89244863b8e5-plugins-config-volume\") pod \"b077a6d9-5d78-4d91-8223-89244863b8e5\" (UID: \"b077a6d9-5d78-4d91-8223-89244863b8e5\") " Nov 02 18:03:03.381073 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.381046 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b077a6d9-5d78-4d91-8223-89244863b8e5-plugins-config-volume" pod "b077a6d9-5d78-4d91-8223-89244863b8e5" (UID: "b077a6d9-5d78-4d91-8223-89244863b8e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:03:03.381264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.381165 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7639dacc-94de-425d-8ba4-3fa1baa67fb1-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:03.382586 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.382564 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7639dacc-94de-425d-8ba4-3fa1baa67fb1-kube-api-access-vb9dd" pod "7639dacc-94de-425d-8ba4-3fa1baa67fb1" (UID: "7639dacc-94de-425d-8ba4-3fa1baa67fb1"). InnerVolumeSpecName "kube-api-access-vb9dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:03.382861 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.382844 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b077a6d9-5d78-4d91-8223-89244863b8e5-kube-api-access-hswkd" pod "b077a6d9-5d78-4d91-8223-89244863b8e5" (UID: "b077a6d9-5d78-4d91-8223-89244863b8e5"). InnerVolumeSpecName "kube-api-access-hswkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:03.482441 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.482351 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hswkd\" (UniqueName: \"kubernetes.io/projected/b077a6d9-5d78-4d91-8223-89244863b8e5-kube-api-access-hswkd\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:03.482441 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.482384 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb9dd\" (UniqueName: \"kubernetes.io/projected/7639dacc-94de-425d-8ba4-3fa1baa67fb1-kube-api-access-vb9dd\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:03.482441 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.482398 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b077a6d9-5d78-4d91-8223-89244863b8e5-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:03.515036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.514995 5549 generic.go:356] "Generic (PLEG): container finished" podID="b077a6d9-5d78-4d91-8223-89244863b8e5" containerID="9f4d9a4eeb1cf97c9c23f0142ee94f9d02a7c1d74caedb3cfc737e79e86f3743" exitCode=0 Nov 02 18:03:03.515235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.515045 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk" Nov 02 18:03:03.515235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.515077 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk" event={"ID":"b077a6d9-5d78-4d91-8223-89244863b8e5","Type":"ContainerDied","Data":"9f4d9a4eeb1cf97c9c23f0142ee94f9d02a7c1d74caedb3cfc737e79e86f3743"} Nov 02 18:03:03.515235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.515145 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk" event={"ID":"b077a6d9-5d78-4d91-8223-89244863b8e5","Type":"ContainerDied","Data":"cb356b51dd9bc80d33182e4eefc722e15d6170661c745c79f705b5b306fc2293"} Nov 02 18:03:03.515235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.515175 5549 scope.go:122] "RemoveContainer" containerID="9f4d9a4eeb1cf97c9c23f0142ee94f9d02a7c1d74caedb3cfc737e79e86f3743" Nov 02 18:03:03.516657 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.516634 5549 generic.go:356] "Generic (PLEG): container finished" podID="7639dacc-94de-425d-8ba4-3fa1baa67fb1" containerID="614b3669c8b417be4d96d3bbbabe9299cb281d105ad661882990db086e9d0de7" exitCode=0 Nov 02 18:03:03.516769 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.516678 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck" Nov 02 18:03:03.516834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.516796 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck" event={"ID":"7639dacc-94de-425d-8ba4-3fa1baa67fb1","Type":"ContainerDied","Data":"614b3669c8b417be4d96d3bbbabe9299cb281d105ad661882990db086e9d0de7"} Nov 02 18:03:03.516834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.516832 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck" event={"ID":"7639dacc-94de-425d-8ba4-3fa1baa67fb1","Type":"ContainerDied","Data":"813e19c99f372f2741c246e91383546325ba63faee572bf10e57bdd757b922f3"} Nov 02 18:03:03.527321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.527226 5549 scope.go:122] "RemoveContainer" containerID="9f4d9a4eeb1cf97c9c23f0142ee94f9d02a7c1d74caedb3cfc737e79e86f3743" Nov 02 18:03:03.527572 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:03:03.527552 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f4d9a4eeb1cf97c9c23f0142ee94f9d02a7c1d74caedb3cfc737e79e86f3743\": container with ID starting with 9f4d9a4eeb1cf97c9c23f0142ee94f9d02a7c1d74caedb3cfc737e79e86f3743 not found: ID does not exist" containerID="9f4d9a4eeb1cf97c9c23f0142ee94f9d02a7c1d74caedb3cfc737e79e86f3743" Nov 02 18:03:03.527627 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.527582 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f4d9a4eeb1cf97c9c23f0142ee94f9d02a7c1d74caedb3cfc737e79e86f3743"} err="failed to get container status \"9f4d9a4eeb1cf97c9c23f0142ee94f9d02a7c1d74caedb3cfc737e79e86f3743\": rpc error: code = NotFound desc = could not find container \"9f4d9a4eeb1cf97c9c23f0142ee94f9d02a7c1d74caedb3cfc737e79e86f3743\": container with ID starting with 9f4d9a4eeb1cf97c9c23f0142ee94f9d02a7c1d74caedb3cfc737e79e86f3743 not found: ID does not exist" Nov 02 18:03:03.527627 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.527600 5549 scope.go:122] "RemoveContainer" containerID="614b3669c8b417be4d96d3bbbabe9299cb281d105ad661882990db086e9d0de7" Nov 02 18:03:03.538152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.538127 5549 scope.go:122] "RemoveContainer" containerID="614b3669c8b417be4d96d3bbbabe9299cb281d105ad661882990db086e9d0de7" Nov 02 18:03:03.538478 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:03:03.538441 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614b3669c8b417be4d96d3bbbabe9299cb281d105ad661882990db086e9d0de7\": container with ID starting with 614b3669c8b417be4d96d3bbbabe9299cb281d105ad661882990db086e9d0de7 not found: ID does not exist" containerID="614b3669c8b417be4d96d3bbbabe9299cb281d105ad661882990db086e9d0de7" Nov 02 18:03:03.538535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.538491 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614b3669c8b417be4d96d3bbbabe9299cb281d105ad661882990db086e9d0de7"} err="failed to get container status \"614b3669c8b417be4d96d3bbbabe9299cb281d105ad661882990db086e9d0de7\": rpc error: code = NotFound desc = could not find container \"614b3669c8b417be4d96d3bbbabe9299cb281d105ad661882990db086e9d0de7\": container with ID starting with 614b3669c8b417be4d96d3bbbabe9299cb281d105ad661882990db086e9d0de7 not found: ID does not exist" Nov 02 18:03:03.545662 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.545629 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk"] Nov 02 18:03:03.556172 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.556144 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4ea582-66c9c7b47f-jpjdk"] Nov 02 18:03:03.568836 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.568803 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck"] Nov 02 18:03:03.575077 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.575043 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4ea582-59fc9c7867-7gpck"] Nov 02 18:03:03.756138 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:03.753480 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:04.225058 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:04.225012 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ea582-55f87bf6dc-z9v9j" podUID="3a54d68a-aef4-4f05-ba0c-3571ca39d6a9" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.215:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:04.324001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:04.323946 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7639dacc-94de-425d-8ba4-3fa1baa67fb1" path="/var/lib/kubelet/pods/7639dacc-94de-425d-8ba4-3fa1baa67fb1/volumes" Nov 02 18:03:04.324570 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:04.324553 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b077a6d9-5d78-4d91-8223-89244863b8e5" path="/var/lib/kubelet/pods/b077a6d9-5d78-4d91-8223-89244863b8e5/volumes" Nov 02 18:03:04.749021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:04.748981 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:05.081642 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.081560 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:05.231467 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.231432 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:05.577507 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.577472 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:05.718784 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.718580 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:05.854489 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.854408 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn"] Nov 02 18:03:05.855449 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.855428 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7639dacc-94de-425d-8ba4-3fa1baa67fb1" containerName="payload-pre-processing" Nov 02 18:03:05.855580 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.855450 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7639dacc-94de-425d-8ba4-3fa1baa67fb1" containerName="payload-pre-processing" Nov 02 18:03:05.855580 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.855483 5549 cpu_manager.go:414] "containerMap: removing container" podUID="3a54d68a-aef4-4f05-ba0c-3571ca39d6a9" containerName="maas-api" Nov 02 18:03:05.855580 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.855489 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a54d68a-aef4-4f05-ba0c-3571ca39d6a9" containerName="maas-api" Nov 02 18:03:05.855580 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.855510 5549 cpu_manager.go:414] "containerMap: removing container" podUID="b077a6d9-5d78-4d91-8223-89244863b8e5" containerName="payload-processing" Nov 02 18:03:05.855580 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.855515 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b077a6d9-5d78-4d91-8223-89244863b8e5" containerName="payload-processing" Nov 02 18:03:05.855580 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.855534 5549 cpu_manager.go:414] "containerMap: removing container" podUID="501cea11-004b-4161-b64f-552baabdd8bd" containerName="main" Nov 02 18:03:05.855580 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.855540 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="501cea11-004b-4161-b64f-552baabdd8bd" containerName="main" Nov 02 18:03:05.855811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.855665 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a54d68a-aef4-4f05-ba0c-3571ca39d6a9" containerName="maas-api" Nov 02 18:03:05.855811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.855677 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="b077a6d9-5d78-4d91-8223-89244863b8e5" containerName="payload-processing" Nov 02 18:03:05.855811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.855686 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="501cea11-004b-4161-b64f-552baabdd8bd" containerName="main" Nov 02 18:03:05.855811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.855695 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7639dacc-94de-425d-8ba4-3fa1baa67fb1" containerName="payload-pre-processing" Nov 02 18:03:05.861888 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.861853 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:05.897894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.897809 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn"] Nov 02 18:03:05.917098 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.911697 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx"] Nov 02 18:03:05.922205 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.922176 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:05.944680 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.944635 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx"] Nov 02 18:03:05.956003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.951655 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl"] Nov 02 18:03:05.956003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.951890 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" podUID="fa70aff2-1e86-4d0c-a95a-57cc50537c65" containerName="istio-proxy" containerID="cri-o://fa2a441db385a2d0f3e298f7bf6b5254630aa859c097399e5ef3eb0ff560d7b2" gracePeriod=30 Nov 02 18:03:05.959894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.959862 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:05.960034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.959903 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:05.960118 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.960075 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:05.960181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.960128 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:05.960181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.960151 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:05.960284 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.960230 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:05.960284 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.960249 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:05.960284 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.960266 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:05.960463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.960288 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:05.960463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.960326 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:05.960463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.960344 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:05.960463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.960362 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:05.960463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.960399 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhj65\" (UniqueName: \"kubernetes.io/projected/70149a4a-e73e-4b5e-8d39-3877dd4c5583-kube-api-access-qhj65\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:05.960463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.960421 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:05.960809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.960515 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:05.960809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.960551 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:05.960809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.960567 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:05.960809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.960623 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwzcv\" (UniqueName: \"kubernetes.io/projected/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-kube-api-access-lwzcv\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:05.977805 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:05.977777 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:06.086476 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.086410 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.087104 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.087150 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.087230 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhj65\" (UniqueName: \"kubernetes.io/projected/70149a4a-e73e-4b5e-8d39-3877dd4c5583-kube-api-access-qhj65\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.087287 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.087490 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.087564 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.087591 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.087715 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwzcv\" (UniqueName: \"kubernetes.io/projected/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-kube-api-access-lwzcv\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.087875 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.087906 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.088062 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.088109 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.088134 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.088221 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.088262 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.088287 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.088340 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.092231 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.087030 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.093156 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.165980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.109604 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.165980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.110237 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.165980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.110814 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.165980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.117915 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.165980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.118245 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.165980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.119136 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.165980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.126270 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.166859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.166820 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwzcv\" (UniqueName: \"kubernetes.io/projected/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-kube-api-access-lwzcv\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.186976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.183509 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.230299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.201659 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhj65\" (UniqueName: \"kubernetes.io/projected/70149a4a-e73e-4b5e-8d39-3877dd4c5583-kube-api-access-qhj65\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.230299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.222736 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.230299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.223112 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.230299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.223645 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.230896 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.230846 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.260591 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.260310 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:06.327128 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.325452 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.356672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.353831 5549 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:03:06.356672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.353902 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" podUID="fa70aff2-1e86-4d0c-a95a-57cc50537c65" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:03:06.533066 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.533034 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:06.652516 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.632858 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:06.738661 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.736735 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx"] Nov 02 18:03:06.739814 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.739772 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:03:06.740048 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.739862 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:03:06.740048 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.739907 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:03:06.835921 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:03:06.835876 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70149a4a_e73e_4b5e_8d39_3877dd4c5583.slice/crio-4c8742aec990ddbfb0da47cb13364797f38e1af4f4988a6fdaf51fdebbe2f5aa WatchSource:0}: Error finding container 4c8742aec990ddbfb0da47cb13364797f38e1af4f4988a6fdaf51fdebbe2f5aa: Status 404 returned error can't find the container with id 4c8742aec990ddbfb0da47cb13364797f38e1af4f4988a6fdaf51fdebbe2f5aa Nov 02 18:03:06.836896 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.836871 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn"] Nov 02 18:03:06.839550 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.839517 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:03:06.839637 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.839583 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:03:06.839637 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:06.839611 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:03:07.644979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:07.643290 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:07.891088 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:07.891051 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:07.900402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:07.897507 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" event={"ID":"b20603e9-d7ea-4bac-bf54-c1c59d343fa8","Type":"ContainerStarted","Data":"56414aed1f8107254cf65658482003cf8d5d70f056894ec6379e343631faa47f"} Nov 02 18:03:07.900699 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:07.900660 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" event={"ID":"b20603e9-d7ea-4bac-bf54-c1c59d343fa8","Type":"ContainerStarted","Data":"d0a09a5b1e71d134bd5e4d4a0a753f754858db3e69bb8a5bbdc384ff512f7e35"} Nov 02 18:03:07.945832 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:07.945763 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" event={"ID":"70149a4a-e73e-4b5e-8d39-3877dd4c5583","Type":"ContainerStarted","Data":"f5193e89ef6ab472e98e25776046fbd7ca0bca3d1f9e4ffc4f6411f223620a93"} Nov 02 18:03:07.946191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:07.946160 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" event={"ID":"70149a4a-e73e-4b5e-8d39-3877dd4c5583","Type":"ContainerStarted","Data":"4c8742aec990ddbfb0da47cb13364797f38e1af4f4988a6fdaf51fdebbe2f5aa"} Nov 02 18:03:08.151854 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:08.151737 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" podStartSLOduration=3.151717623 podStartE2EDuration="3.151717623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:08.106189161 +0000 UTC m=+1759.894694690" watchObservedRunningTime="2026-11-02 18:03:08.151717623 +0000 UTC m=+1759.940223357" Nov 02 18:03:08.218976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:08.217921 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" podStartSLOduration=3.217897544 podStartE2EDuration="3.217897544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:08.216356226 +0000 UTC m=+1760.004861756" watchObservedRunningTime="2026-11-02 18:03:08.217897544 +0000 UTC m=+1760.006403078" Nov 02 18:03:08.261444 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:08.261404 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:08.262785 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:08.262752 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 02 18:03:08.262931 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:08.262817 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" podUID="b20603e9-d7ea-4bac-bf54-c1c59d343fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 02 18:03:08.447270 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:08.439590 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:08.623976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:08.606009 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:08.623976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:08.610792 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:08.623976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:08.621438 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 02 18:03:08.623976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:08.621510 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" podUID="70149a4a-e73e-4b5e-8d39-3877dd4c5583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 02 18:03:08.915028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:08.914686 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:09.280986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:09.261235 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 02 18:03:09.280986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:09.261301 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" podUID="b20603e9-d7ea-4bac-bf54-c1c59d343fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 02 18:03:09.534239 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:09.534155 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 02 18:03:09.534239 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:09.534209 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" podUID="70149a4a-e73e-4b5e-8d39-3877dd4c5583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 02 18:03:10.055974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:10.051488 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:10.242063 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:10.234485 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:10.275998 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:10.272433 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 02 18:03:10.275998 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:10.272506 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" podUID="b20603e9-d7ea-4bac-bf54-c1c59d343fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 02 18:03:10.431029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:10.428322 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:10.541977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:10.535981 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 02 18:03:10.541977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:10.536044 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" podUID="70149a4a-e73e-4b5e-8d39-3877dd4c5583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 02 18:03:10.608978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:10.608066 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:10.722978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:10.708026 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:10.948988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:10.933740 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:11.029542 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:11.027348 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:11.081711 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:11.081121 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:11.285036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:11.278489 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 02 18:03:11.285036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:11.278566 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" podUID="b20603e9-d7ea-4bac-bf54-c1c59d343fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 02 18:03:11.326176 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:11.297412 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:11.460987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:11.447771 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:11.609033 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:11.592763 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 02 18:03:11.609033 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:11.592872 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" podUID="70149a4a-e73e-4b5e-8d39-3877dd4c5583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 02 18:03:11.679000 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:11.676827 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:12.046975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:12.042579 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 18:03:12.046975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:12.043393 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 18:03:12.171358 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:12.165143 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:12.171358 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:12.166862 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 18:03:12.428666 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:12.328306 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 02 18:03:12.428666 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:12.328374 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" podUID="b20603e9-d7ea-4bac-bf54-c1c59d343fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 02 18:03:12.428666 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:12.328937 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:12.428666 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:12.346099 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 18:03:12.466743 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:12.466710 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 18:03:12.562027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:12.560094 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:12.562027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:12.561576 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 18:03:12.574416 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:12.574382 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 02 18:03:12.575756 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:12.575725 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" podUID="70149a4a-e73e-4b5e-8d39-3877dd4c5583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 02 18:03:12.781120 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:12.781090 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 18:03:12.863930 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:12.863894 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:13.183387 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.183295 5549 generic.go:356] "Generic (PLEG): container finished" podID="fa70aff2-1e86-4d0c-a95a-57cc50537c65" containerID="fa2a441db385a2d0f3e298f7bf6b5254630aa859c097399e5ef3eb0ff560d7b2" exitCode=0 Nov 02 18:03:13.212977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.184225 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" event={"ID":"fa70aff2-1e86-4d0c-a95a-57cc50537c65","Type":"ContainerDied","Data":"fa2a441db385a2d0f3e298f7bf6b5254630aa859c097399e5ef3eb0ff560d7b2"} Nov 02 18:03:13.375893 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.368342 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 02 18:03:13.375893 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.368409 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" podUID="b20603e9-d7ea-4bac-bf54-c1c59d343fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 02 18:03:13.409990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.409794 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:03:13.471685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.450577 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-workload-socket\") pod \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " Nov 02 18:03:13.471685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.450633 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-podinfo\") pod \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " Nov 02 18:03:13.471685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.450735 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-data\") pod \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " Nov 02 18:03:13.471685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.450814 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-envoy\") pod \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " Nov 02 18:03:13.471685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.450972 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-credential-socket\") pod \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " Nov 02 18:03:13.471685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.452073 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-token\" (UniqueName: \"kubernetes.io/projected/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-token\") pod \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " Nov 02 18:03:13.471685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.452265 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa70aff2-1e86-4d0c-a95a-57cc50537c65-kube-api-access-hmtds\" (UniqueName: \"kubernetes.io/projected/fa70aff2-1e86-4d0c-a95a-57cc50537c65-kube-api-access-hmtds\") pod \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " Nov 02 18:03:13.471685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.452300 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-workload-certs\") pod \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " Nov 02 18:03:13.471685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.452330 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istiod-ca-cert\") pod \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\" (UID: \"fa70aff2-1e86-4d0c-a95a-57cc50537c65\") " Nov 02 18:03:13.471685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.459748 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-workload-socket" pod "fa70aff2-1e86-4d0c-a95a-57cc50537c65" (UID: "fa70aff2-1e86-4d0c-a95a-57cc50537c65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:13.471685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.465354 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-workload-certs" pod "fa70aff2-1e86-4d0c-a95a-57cc50537c65" (UID: "fa70aff2-1e86-4d0c-a95a-57cc50537c65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:13.497974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.488703 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-data" pod "fa70aff2-1e86-4d0c-a95a-57cc50537c65" (UID: "fa70aff2-1e86-4d0c-a95a-57cc50537c65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:13.498420 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.498380 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-credential-socket" pod "fa70aff2-1e86-4d0c-a95a-57cc50537c65" (UID: "fa70aff2-1e86-4d0c-a95a-57cc50537c65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:13.511981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.499618 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:13.512249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.512222 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:13.512369 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.512356 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:13.512457 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.512446 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:13.596269 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.596213 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istiod-ca-cert" pod "fa70aff2-1e86-4d0c-a95a-57cc50537c65" (UID: "fa70aff2-1e86-4d0c-a95a-57cc50537c65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:03:13.749204 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.688218 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa70aff2-1e86-4d0c-a95a-57cc50537c65-kube-api-access-hmtds" pod "fa70aff2-1e86-4d0c-a95a-57cc50537c65" (UID: "fa70aff2-1e86-4d0c-a95a-57cc50537c65"). InnerVolumeSpecName "kube-api-access-hmtds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:13.749204 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.688316 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-envoy" pod "fa70aff2-1e86-4d0c-a95a-57cc50537c65" (UID: "fa70aff2-1e86-4d0c-a95a-57cc50537c65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:13.749204 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.657942 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 02 18:03:13.749204 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.689302 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" podUID="70149a4a-e73e-4b5e-8d39-3877dd4c5583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 02 18:03:13.749204 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.736551 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:13.784496 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.760531 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmtds\" (UniqueName: \"kubernetes.io/projected/fa70aff2-1e86-4d0c-a95a-57cc50537c65-kube-api-access-hmtds\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:13.784496 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.760572 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:13.784496 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.760591 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:13.868114 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.867240 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-token" pod "fa70aff2-1e86-4d0c-a95a-57cc50537c65" (UID: "fa70aff2-1e86-4d0c-a95a-57cc50537c65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:13.978334 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.975749 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:13.978334 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:13.977665 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-podinfo" pod "fa70aff2-1e86-4d0c-a95a-57cc50537c65" (UID: "fa70aff2-1e86-4d0c-a95a-57cc50537c65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:03:14.077697 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:14.077607 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa70aff2-1e86-4d0c-a95a-57cc50537c65-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:14.293978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:14.286789 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 02 18:03:14.294457 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:14.294022 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" podUID="b20603e9-d7ea-4bac-bf54-c1c59d343fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 02 18:03:14.470716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:14.470676 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" Nov 02 18:03:14.472663 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:14.472625 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl" event={"ID":"fa70aff2-1e86-4d0c-a95a-57cc50537c65","Type":"ContainerDied","Data":"6d9435f17f38ea4e1f2940ebbcbab0b7268f4435ee34419b7dcd4a12f118774a"} Nov 02 18:03:14.472789 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:14.472690 5549 scope.go:122] "RemoveContainer" containerID="fa2a441db385a2d0f3e298f7bf6b5254630aa859c097399e5ef3eb0ff560d7b2" Nov 02 18:03:14.553645 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:14.553191 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 02 18:03:14.553645 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:14.553284 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" podUID="70149a4a-e73e-4b5e-8d39-3877dd4c5583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 02 18:03:14.661264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:14.661222 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl"] Nov 02 18:03:14.666034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:14.665996 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-krnwl"] Nov 02 18:03:14.815033 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:14.794095 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:14.996134 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:14.996079 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:15.261293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:15.261257 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 02 18:03:15.261492 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:15.261312 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" podUID="b20603e9-d7ea-4bac-bf54-c1c59d343fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 02 18:03:15.553031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:15.552080 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 02 18:03:15.553031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:15.552145 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" podUID="70149a4a-e73e-4b5e-8d39-3877dd4c5583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 02 18:03:15.910039 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:15.909925 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:16.343527 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:16.343498 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa70aff2-1e86-4d0c-a95a-57cc50537c65" path="/var/lib/kubelet/pods/fa70aff2-1e86-4d0c-a95a-57cc50537c65/volumes" Nov 02 18:03:16.439840 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:16.439789 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:16.442174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:16.442140 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:16.444072 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:16.444036 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:03:16.762905 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:16.762850 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:16.763514 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:16.763394 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:16.764469 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:16.764429 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:03:17.091774 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:17.091654 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:17.301558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:17.301520 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:17.612368 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:03:17.612326 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:03:18.163593 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.163556 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:18.473624 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.473588 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:18.517232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.517193 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j"] Nov 02 18:03:18.518653 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.518631 5549 cpu_manager.go:414] "containerMap: removing container" podUID="fa70aff2-1e86-4d0c-a95a-57cc50537c65" containerName="istio-proxy" Nov 02 18:03:18.518715 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.518655 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa70aff2-1e86-4d0c-a95a-57cc50537c65" containerName="istio-proxy" Nov 02 18:03:18.518799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.518789 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa70aff2-1e86-4d0c-a95a-57cc50537c65" containerName="istio-proxy" Nov 02 18:03:18.524419 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.524399 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r"] Nov 02 18:03:18.524537 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.524518 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" Nov 02 18:03:18.529045 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.529016 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-36a991\"" Nov 02 18:03:18.530060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.529751 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" Nov 02 18:03:18.533703 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.533654 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-36a991-dockercfg-gdfj8\"" Nov 02 18:03:18.538032 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.535497 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j"] Nov 02 18:03:18.541341 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.541318 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r"] Nov 02 18:03:18.581938 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.581897 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpcgd\" (UniqueName: \"kubernetes.io/projected/b612d0da-2500-4091-a03e-89faeb9c435d-kube-api-access-bpcgd\") pod \"payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j\" (UID: \"b612d0da-2500-4091-a03e-89faeb9c435d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" Nov 02 18:03:18.582134 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.582106 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp5tr\" (UniqueName: \"kubernetes.io/projected/31758614-1d84-4256-ba23-aed7dd3dd328-kube-api-access-qp5tr\") pod \"payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r\" (UID: \"31758614-1d84-4256-ba23-aed7dd3dd328\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" Nov 02 18:03:18.582211 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.582169 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31758614-1d84-4256-ba23-aed7dd3dd328-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r\" (UID: \"31758614-1d84-4256-ba23-aed7dd3dd328\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" Nov 02 18:03:18.582277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.582262 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b612d0da-2500-4091-a03e-89faeb9c435d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j\" (UID: \"b612d0da-2500-4091-a03e-89faeb9c435d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" Nov 02 18:03:18.655980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.631832 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:18.778028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.720531 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31758614-1d84-4256-ba23-aed7dd3dd328-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r\" (UID: \"31758614-1d84-4256-ba23-aed7dd3dd328\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" Nov 02 18:03:18.795027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.788287 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b612d0da-2500-4091-a03e-89faeb9c435d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j\" (UID: \"b612d0da-2500-4091-a03e-89faeb9c435d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" Nov 02 18:03:18.819375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.819328 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpcgd\" (UniqueName: \"kubernetes.io/projected/b612d0da-2500-4091-a03e-89faeb9c435d-kube-api-access-bpcgd\") pod \"payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j\" (UID: \"b612d0da-2500-4091-a03e-89faeb9c435d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" Nov 02 18:03:18.819866 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.819843 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp5tr\" (UniqueName: \"kubernetes.io/projected/31758614-1d84-4256-ba23-aed7dd3dd328-kube-api-access-qp5tr\") pod \"payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r\" (UID: \"31758614-1d84-4256-ba23-aed7dd3dd328\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" Nov 02 18:03:18.834280 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.823384 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b612d0da-2500-4091-a03e-89faeb9c435d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j\" (UID: \"b612d0da-2500-4091-a03e-89faeb9c435d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" Nov 02 18:03:18.854829 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.834629 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31758614-1d84-4256-ba23-aed7dd3dd328-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r\" (UID: \"31758614-1d84-4256-ba23-aed7dd3dd328\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" Nov 02 18:03:18.901241 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.901135 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpcgd\" (UniqueName: \"kubernetes.io/projected/b612d0da-2500-4091-a03e-89faeb9c435d-kube-api-access-bpcgd\") pod \"payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j\" (UID: \"b612d0da-2500-4091-a03e-89faeb9c435d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" Nov 02 18:03:18.903478 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:18.902616 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp5tr\" (UniqueName: \"kubernetes.io/projected/31758614-1d84-4256-ba23-aed7dd3dd328-kube-api-access-qp5tr\") pod \"payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r\" (UID: \"31758614-1d84-4256-ba23-aed7dd3dd328\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" Nov 02 18:03:19.154138 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:19.154039 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" Nov 02 18:03:19.163896 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:19.163864 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" Nov 02 18:03:19.262291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:19.255712 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:19.565979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:19.564778 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r"] Nov 02 18:03:19.565979 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:03:19.565103 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31758614_1d84_4256_ba23_aed7dd3dd328.slice/crio-35fa50fe64c24a7d87412c4e62a8b03b156c20d3a07f2d239dbc636e485c29a8 WatchSource:0}: Error finding container 35fa50fe64c24a7d87412c4e62a8b03b156c20d3a07f2d239dbc636e485c29a8: Status 404 returned error can't find the container with id 35fa50fe64c24a7d87412c4e62a8b03b156c20d3a07f2d239dbc636e485c29a8 Nov 02 18:03:19.596607 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:19.596569 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" event={"ID":"31758614-1d84-4256-ba23-aed7dd3dd328","Type":"ContainerStarted","Data":"35fa50fe64c24a7d87412c4e62a8b03b156c20d3a07f2d239dbc636e485c29a8"} Nov 02 18:03:19.655527 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:03:19.655496 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb612d0da_2500_4091_a03e_89faeb9c435d.slice/crio-bff7ee067bc4eb580fea8e6c4b73c5ec80c840b12ed3974864a3415759d44ff3 WatchSource:0}: Error finding container bff7ee067bc4eb580fea8e6c4b73c5ec80c840b12ed3974864a3415759d44ff3: Status 404 returned error can't find the container with id bff7ee067bc4eb580fea8e6c4b73c5ec80c840b12ed3974864a3415759d44ff3 Nov 02 18:03:19.656172 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:19.656147 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j"] Nov 02 18:03:19.982654 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:19.982573 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:20.650564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:20.649191 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:20.654984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:20.653624 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" event={"ID":"b612d0da-2500-4091-a03e-89faeb9c435d","Type":"ContainerStarted","Data":"993e4c03ff25143e6426ddfa51cc5c0be68e31dc9b4ef457b9090f473d463847"} Nov 02 18:03:20.654984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:20.653689 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" event={"ID":"b612d0da-2500-4091-a03e-89faeb9c435d","Type":"ContainerStarted","Data":"bff7ee067bc4eb580fea8e6c4b73c5ec80c840b12ed3974864a3415759d44ff3"} Nov 02 18:03:20.658021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:20.655789 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" Nov 02 18:03:20.658021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:20.656205 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" event={"ID":"31758614-1d84-4256-ba23-aed7dd3dd328","Type":"ContainerStarted","Data":"e6acfb570aa43edbf6e4305690fc7665a2fbf91ece0e23730519ef1c007d2293"} Nov 02 18:03:20.658021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:20.656465 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" Nov 02 18:03:20.730487 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:20.730404 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" podStartSLOduration=2.7303820500000002 podStartE2EDuration="2.73038205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:20.728350343 +0000 UTC m=+1772.516855872" watchObservedRunningTime="2026-11-02 18:03:20.73038205 +0000 UTC m=+1772.518887585" Nov 02 18:03:20.731523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:20.731476 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" podStartSLOduration=2.73146287 podStartE2EDuration="2.73146287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:20.708508678 +0000 UTC m=+1772.497014208" watchObservedRunningTime="2026-11-02 18:03:20.73146287 +0000 UTC m=+1772.519968400" Nov 02 18:03:21.568129 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:21.567840 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:22.619974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:22.617186 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:23.425569 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:23.425526 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:23.734069 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:23.734027 5549 generic.go:356] "Generic (PLEG): container finished" podID="3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" containerID="4e035c6af637a2468e6356bfbfe9ed42c99f034770c74211741644eed8f4f062" exitCode=0 Nov 02 18:03:23.734518 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:23.734098 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" event={"ID":"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2","Type":"ContainerDied","Data":"4e035c6af637a2468e6356bfbfe9ed42c99f034770c74211741644eed8f4f062"} Nov 02 18:03:23.734596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:23.734581 5549 scope.go:122] "RemoveContainer" containerID="4e035c6af637a2468e6356bfbfe9ed42c99f034770c74211741644eed8f4f062" Nov 02 18:03:23.736554 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:23.736524 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:03:23.736633 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:23.736595 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:03:23.736633 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:23.736627 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:03:24.218975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:24.217931 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:24.745149 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:24.745108 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" event={"ID":"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2","Type":"ContainerStarted","Data":"f0e76d3456b290b8ef8494ed513e6398a2feb3ddc5a8d29f6db44023bd39d761"} Nov 02 18:03:24.817512 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:24.817477 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:03:25.008012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:25.005416 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:25.077009 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:25.075377 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:03:25.077009 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:25.075569 5549 patch_prober.go:29] interesting pod/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 02 18:03:25.077009 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:25.075625 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" podUID="3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 02 18:03:25.693979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:25.686492 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:26.074987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:26.074933 5549 patch_prober.go:29] interesting pod/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 02 18:03:26.075394 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:26.075007 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" podUID="3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 02 18:03:26.452994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:26.452902 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:26.619753 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:26.619719 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:27.074758 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:27.074723 5549 patch_prober.go:29] interesting pod/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 02 18:03:27.074999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:27.074776 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" podUID="3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 02 18:03:27.147089 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:27.133634 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:27.512717 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:27.512673 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:27.667778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:27.667739 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:27.833486 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:27.833372 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:27.989834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:27.989798 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:28.102360 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:28.097106 5549 patch_prober.go:29] interesting pod/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 02 18:03:28.117393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:28.117336 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" podUID="3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 02 18:03:28.158303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:28.158265 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:28.326415 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:28.326378 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:28.808297 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:28.807038 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:28.963804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:28.963770 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:29.079034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:29.076270 5549 patch_prober.go:29] interesting pod/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 02 18:03:29.079034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:29.076347 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" podUID="3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 02 18:03:29.431729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:29.431594 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:29.768397 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:29.768357 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:30.075034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:30.074509 5549 patch_prober.go:29] interesting pod/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 02 18:03:30.075034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:30.074596 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" podUID="3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 02 18:03:30.595994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:30.595916 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:31.079675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:31.079642 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:03:31.081030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:31.081004 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:03:31.338091 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:31.337807 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:31.707130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:31.707102 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" Nov 02 18:03:31.707531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:31.707171 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" Nov 02 18:03:32.041343 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:32.041242 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:32.715734 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:32.715695 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:33.424655 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:33.424608 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:34.246309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:34.246060 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:34.926388 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:34.926347 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:35.652711 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:35.652675 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:36.398616 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:36.398582 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:36.554831 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:36.554786 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:37.352237 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:37.350322 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:38.058720 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:38.058672 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:38.768776 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:38.768727 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:39.626577 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:39.626521 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:40.348993 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:40.340526 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:41.036982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:41.029371 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:41.732227 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:41.732125 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:42.525298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:42.525259 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:43.232467 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:43.232433 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:43.952320 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:43.951944 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:44.685181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:44.685147 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:03:44.732934 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:44.732878 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:45.616835 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:45.616794 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:46.854427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:46.854384 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:47.590148 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:03:47.590110 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:03:47.668805 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:47.668768 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:48.516528 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:48.515943 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:49.273402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:49.273363 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:49.676812 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:49.676735 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:03:49.892204 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:49.892175 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:03:50.071258 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:50.071207 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:50.851987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:50.851450 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:51.756981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:51.746502 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:52.541024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:52.533499 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:53.348011 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:53.347120 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:54.081586 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:54.081533 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:54.747535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:54.740069 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:55.445629 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:55.445506 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:56.215179 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:56.211302 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:56.888611 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:56.888508 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:57.646999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:57.646620 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:58.386977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:58.376915 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:59.077751 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:59.077719 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:59.488554 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:59.488524 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:59.765183 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:03:59.765028 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:00.550533 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:00.550335 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:01.260560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:01.259364 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:01.926049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:01.926006 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:02.730013 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:02.729915 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:03.466235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:03.446695 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:04.309074 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:04.298225 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:05.051137 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:05.051096 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:05.854999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:05.848531 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:06.604979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:06.594212 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:07.288153 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:07.288110 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:08.148450 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:08.148415 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:08.840975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:08.840734 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:09.607636 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:09.607519 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:10.128690 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:10.128658 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:10.267132 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:10.267096 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:10.412055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:10.411900 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:11.237002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:11.235137 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:12.057974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:12.046577 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:12.830976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:12.830872 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:13.528024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:13.527990 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:14.188990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:14.188439 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:15.024529 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:15.024480 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:15.997025 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:15.996988 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:16.831787 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:16.831749 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:17.537030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:17.535598 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:17.622536 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:04:17.622497 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:04:18.240329 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:18.240289 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:18.419470 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:18.419436 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:19.067975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:19.065288 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:19.567373 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:19.567342 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:19.930024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:19.913776 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:20.094874 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:20.082669 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:20.269983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:20.269498 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:21.439023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:21.438944 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:22.279655 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:22.279614 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:22.484697 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:22.484662 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:22.644141 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:22.644064 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:22.808977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:22.805440 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:22.946977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:22.946867 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:23.207490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:23.207400 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:23.353757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:23.353724 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:23.572197 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:23.572099 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:23.736381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:23.736346 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:24.020609 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:24.020570 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:24.158092 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:24.158060 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:24.314179 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:24.314087 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:25.126261 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:25.121604 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:25.967761 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:25.967679 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:26.723531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:26.723447 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:27.396272 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:27.396234 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:28.133984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:28.133567 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:28.823524 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:28.815751 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:29.576423 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:29.576325 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:30.343837 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:30.343787 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:31.057226 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:31.057155 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:31.885933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:31.885897 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:32.721991 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:32.719597 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:33.335873 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:33.335836 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:34.038420 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:34.038377 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:34.846743 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:34.846703 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:35.650982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:35.645913 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:36.385500 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:36.385464 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:37.163816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:37.159606 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:37.946420 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:37.946384 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:38.601143 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:38.601105 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:39.345971 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:39.345916 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:40.135576 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:40.135540 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:40.527243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:40.527209 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:04:40.962974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:40.957515 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:42.139741 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:42.139701 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:43.163977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:43.156058 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:44.138972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:44.138502 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:44.820198 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:44.820163 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:44.973552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:44.973518 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:45.124875 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:45.124793 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:45.566975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:45.528289 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:45.727627 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:45.727521 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:46.930973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:46.918221 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:47.111986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:47.111934 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:47.261621 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:47.261580 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:47.396152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:47.396116 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:47.780891 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:04:47.780836 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:04:48.365773 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:48.365675 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:49.066411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:49.065486 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:49.284257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:49.284205 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:49.426102 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:49.426006 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:50.120445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:50.120398 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:50.260433 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:50.260399 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:51.126781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:51.126739 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:51.831558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:51.831502 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:52.593567 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:52.591510 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:53.352946 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:53.352906 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:54.096106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:54.096061 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:54.844230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:54.844190 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:55.484993 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:55.484328 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:56.292164 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:56.292106 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:56.825060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:56.824944 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:56.993374 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:56.993339 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:57.284973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:57.282053 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:58.118981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:58.118853 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:58.822017 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:58.820484 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:59.536282 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:59.536248 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:59.667369 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:04:59.667317 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:00.369538 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:00.369499 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:01.215017 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:01.214946 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:02.021499 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:02.021460 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:02.621106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:02.621065 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:02.757870 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:02.757838 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:02.931107 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:02.927401 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:03.245993 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:03.245941 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:03.834579 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:03.834493 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:04.540998 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:04.534048 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:05.265979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:05.257489 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:06.047095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:06.047057 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:06.715105 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:06.715067 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:06.861078 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:06.861046 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:07.444977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:07.440050 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:08.356762 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:08.356721 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:08.528563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:08.528534 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:08.672941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:08.672855 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:10.096837 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:10.096803 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:10.234138 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:10.234103 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:10.395247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:10.387984 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:10.568663 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:10.568557 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:10.711356 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:10.711320 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:11.030288 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:11.025711 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:11.752095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:11.752060 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:12.434603 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:12.434564 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:13.210432 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:13.210386 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:14.105232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:14.105198 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:14.934975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:14.929470 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:15.630014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:15.629975 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:16.122810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:16.122773 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:16.277800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:16.277762 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:16.491065 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:16.491016 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:17.274422 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:17.274384 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:17.589973 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:05:17.589869 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:05:17.996471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:17.996438 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:18.136573 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:18.136540 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:18.457092 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:18.457060 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:18.853078 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:18.849995 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:19.884599 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:19.884562 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:20.019891 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:20.019856 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:20.651610 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:20.651573 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5"] Nov 02 18:05:21.031079 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:21.031004 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:21.313755 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:21.313654 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:21.767197 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:21.767156 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:22.500446 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:22.500407 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:22.654498 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:22.654464 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:23.214673 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:23.214636 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:23.372204 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:23.372169 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:23.596563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:23.596475 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:23.762856 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:23.762821 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:23.943476 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:23.943392 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:24.390717 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:24.390675 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:25.328001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:25.327509 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:25.652179 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:25.652095 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" podUID="90faafa6-7183-4a82-86cc-17721fcf2368" containerName="payload-processing" containerID="cri-o://de8e87f17050edae24a761da66bc431590f89bf67047cc33dae7afacd924cbf1" gracePeriod=25 Nov 02 18:05:25.953048 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:25.953021 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" Nov 02 18:05:25.987866 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:25.987829 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7"] Nov 02 18:05:25.988853 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:25.988827 5549 cpu_manager.go:414] "containerMap: removing container" podUID="90faafa6-7183-4a82-86cc-17721fcf2368" containerName="payload-processing" Nov 02 18:05:25.988853 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:25.988849 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90faafa6-7183-4a82-86cc-17721fcf2368" containerName="payload-processing" Nov 02 18:05:25.989076 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:25.989017 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="90faafa6-7183-4a82-86cc-17721fcf2368" containerName="payload-processing" Nov 02 18:05:25.993161 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:25.993137 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" Nov 02 18:05:26.002931 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.002539 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"routing-overlay\"" Nov 02 18:05:26.003185 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.003158 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"e2e-cmp-legacy-6e0be3-provider-cred\"" Nov 02 18:05:26.003318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.003279 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"kube-root-ca.crt\"" Nov 02 18:05:26.003395 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.003342 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a8e574\"" Nov 02 18:05:26.003459 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.003431 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"payload-processing-external-model-e2e--a4661749-dockercfg-6cj9b\"" Nov 02 18:05:26.003523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.003504 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"openshift-service-ca.crt\"" Nov 02 18:05:26.003889 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.003868 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7"] Nov 02 18:05:26.021777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.021742 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:26.050396 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.050359 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90faafa6-7183-4a82-86cc-17721fcf2368-kube-api-access-t5plx\" (UniqueName: \"kubernetes.io/projected/90faafa6-7183-4a82-86cc-17721fcf2368-kube-api-access-t5plx\") pod \"90faafa6-7183-4a82-86cc-17721fcf2368\" (UID: \"90faafa6-7183-4a82-86cc-17721fcf2368\") " Nov 02 18:05:26.050572 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.050417 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90faafa6-7183-4a82-86cc-17721fcf2368-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90faafa6-7183-4a82-86cc-17721fcf2368-routing-overlay-volume\") pod \"90faafa6-7183-4a82-86cc-17721fcf2368\" (UID: \"90faafa6-7183-4a82-86cc-17721fcf2368\") " Nov 02 18:05:26.050572 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.050529 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90faafa6-7183-4a82-86cc-17721fcf2368-provider-credentials\" (UniqueName: \"kubernetes.io/projected/90faafa6-7183-4a82-86cc-17721fcf2368-provider-credentials\") pod \"90faafa6-7183-4a82-86cc-17721fcf2368\" (UID: \"90faafa6-7183-4a82-86cc-17721fcf2368\") " Nov 02 18:05:26.050741 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.050722 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90faafa6-7183-4a82-86cc-17721fcf2368-routing-overlay-volume" pod "90faafa6-7183-4a82-86cc-17721fcf2368" (UID: "90faafa6-7183-4a82-86cc-17721fcf2368"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:26.050789 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.050741 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90faafa6-7183-4a82-86cc-17721fcf2368-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90faafa6-7183-4a82-86cc-17721fcf2368-plugins-config-volume\") pod \"90faafa6-7183-4a82-86cc-17721fcf2368\" (UID: \"90faafa6-7183-4a82-86cc-17721fcf2368\") " Nov 02 18:05:26.051126 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.051101 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90faafa6-7183-4a82-86cc-17721fcf2368-plugins-config-volume" pod "90faafa6-7183-4a82-86cc-17721fcf2368" (UID: "90faafa6-7183-4a82-86cc-17721fcf2368"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:26.051541 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.051521 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90faafa6-7183-4a82-86cc-17721fcf2368-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:26.051541 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.051540 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90faafa6-7183-4a82-86cc-17721fcf2368-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:26.052854 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.052830 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90faafa6-7183-4a82-86cc-17721fcf2368-provider-credentials" pod "90faafa6-7183-4a82-86cc-17721fcf2368" (UID: "90faafa6-7183-4a82-86cc-17721fcf2368"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:26.052928 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.052861 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90faafa6-7183-4a82-86cc-17721fcf2368-kube-api-access-t5plx" pod "90faafa6-7183-4a82-86cc-17721fcf2368" (UID: "90faafa6-7183-4a82-86cc-17721fcf2368"). InnerVolumeSpecName "kube-api-access-t5plx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:26.159749 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.158864 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ead97eb-4321-4214-92ef-ae885540da87-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7\" (UID: \"1ead97eb-4321-4214-92ef-ae885540da87\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" Nov 02 18:05:26.159749 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.159107 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ead97eb-4321-4214-92ef-ae885540da87-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7\" (UID: \"1ead97eb-4321-4214-92ef-ae885540da87\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" Nov 02 18:05:26.159749 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.159186 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhcxc\" (UniqueName: \"kubernetes.io/projected/1ead97eb-4321-4214-92ef-ae885540da87-kube-api-access-qhcxc\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7\" (UID: \"1ead97eb-4321-4214-92ef-ae885540da87\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" Nov 02 18:05:26.159749 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.159278 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ead97eb-4321-4214-92ef-ae885540da87-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7\" (UID: \"1ead97eb-4321-4214-92ef-ae885540da87\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" Nov 02 18:05:26.159749 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.159545 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5plx\" (UniqueName: \"kubernetes.io/projected/90faafa6-7183-4a82-86cc-17721fcf2368-kube-api-access-t5plx\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:26.159749 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.159565 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90faafa6-7183-4a82-86cc-17721fcf2368-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:26.261192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.261155 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhcxc\" (UniqueName: \"kubernetes.io/projected/1ead97eb-4321-4214-92ef-ae885540da87-kube-api-access-qhcxc\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7\" (UID: \"1ead97eb-4321-4214-92ef-ae885540da87\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" Nov 02 18:05:26.261418 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.261238 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ead97eb-4321-4214-92ef-ae885540da87-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7\" (UID: \"1ead97eb-4321-4214-92ef-ae885540da87\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" Nov 02 18:05:26.261418 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.261377 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ead97eb-4321-4214-92ef-ae885540da87-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7\" (UID: \"1ead97eb-4321-4214-92ef-ae885540da87\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" Nov 02 18:05:26.261557 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.261481 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ead97eb-4321-4214-92ef-ae885540da87-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7\" (UID: \"1ead97eb-4321-4214-92ef-ae885540da87\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" Nov 02 18:05:26.262055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.262031 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ead97eb-4321-4214-92ef-ae885540da87-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7\" (UID: \"1ead97eb-4321-4214-92ef-ae885540da87\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" Nov 02 18:05:26.262190 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.262112 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ead97eb-4321-4214-92ef-ae885540da87-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7\" (UID: \"1ead97eb-4321-4214-92ef-ae885540da87\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" Nov 02 18:05:26.264024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.264003 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ead97eb-4321-4214-92ef-ae885540da87-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7\" (UID: \"1ead97eb-4321-4214-92ef-ae885540da87\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" Nov 02 18:05:26.270987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.270927 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhcxc\" (UniqueName: \"kubernetes.io/projected/1ead97eb-4321-4214-92ef-ae885540da87-kube-api-access-qhcxc\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7\" (UID: \"1ead97eb-4321-4214-92ef-ae885540da87\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" Nov 02 18:05:26.296586 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.296552 5549 generic.go:356] "Generic (PLEG): container finished" podID="90faafa6-7183-4a82-86cc-17721fcf2368" containerID="de8e87f17050edae24a761da66bc431590f89bf67047cc33dae7afacd924cbf1" exitCode=0 Nov 02 18:05:26.296770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.296603 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" Nov 02 18:05:26.296770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.296638 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" event={"ID":"90faafa6-7183-4a82-86cc-17721fcf2368","Type":"ContainerDied","Data":"de8e87f17050edae24a761da66bc431590f89bf67047cc33dae7afacd924cbf1"} Nov 02 18:05:26.296770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.296695 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5" event={"ID":"90faafa6-7183-4a82-86cc-17721fcf2368","Type":"ContainerDied","Data":"e43e054fd33acf079c05d6ed9e9e8dac5b4346e69c65b956867aba3ba6971789"} Nov 02 18:05:26.296770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.296718 5549 scope.go:122] "RemoveContainer" containerID="de8e87f17050edae24a761da66bc431590f89bf67047cc33dae7afacd924cbf1" Nov 02 18:05:26.309465 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.309444 5549 scope.go:122] "RemoveContainer" containerID="de8e87f17050edae24a761da66bc431590f89bf67047cc33dae7afacd924cbf1" Nov 02 18:05:26.309812 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:05:26.309786 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8e87f17050edae24a761da66bc431590f89bf67047cc33dae7afacd924cbf1\": container with ID starting with de8e87f17050edae24a761da66bc431590f89bf67047cc33dae7afacd924cbf1 not found: ID does not exist" containerID="de8e87f17050edae24a761da66bc431590f89bf67047cc33dae7afacd924cbf1" Nov 02 18:05:26.309893 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.309821 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8e87f17050edae24a761da66bc431590f89bf67047cc33dae7afacd924cbf1"} err="failed to get container status \"de8e87f17050edae24a761da66bc431590f89bf67047cc33dae7afacd924cbf1\": rpc error: code = NotFound desc = could not find container \"de8e87f17050edae24a761da66bc431590f89bf67047cc33dae7afacd924cbf1\": container with ID starting with de8e87f17050edae24a761da66bc431590f89bf67047cc33dae7afacd924cbf1 not found: ID does not exist" Nov 02 18:05:26.321339 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.321303 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" Nov 02 18:05:26.336374 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.336345 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5"] Nov 02 18:05:26.337730 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.337704 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cw7qc5"] Nov 02 18:05:26.583974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.581140 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:26.648670 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:05:26.648639 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ead97eb_4321_4214_92ef_ae885540da87.slice/crio-77318d6d2e228bb24314cb4621cb080768b618593526f7e4e734e83fe2f1fe75 WatchSource:0}: Error finding container 77318d6d2e228bb24314cb4621cb080768b618593526f7e4e734e83fe2f1fe75: Status 404 returned error can't find the container with id 77318d6d2e228bb24314cb4621cb080768b618593526f7e4e734e83fe2f1fe75 Nov 02 18:05:26.648904 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.648886 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7"] Nov 02 18:05:26.726417 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.726389 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:26.998710 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:26.998518 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:27.308128 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:27.308029 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" event={"ID":"1ead97eb-4321-4214-92ef-ae885540da87","Type":"ContainerStarted","Data":"76041e2a096690e873ef5e148abc9f459afe7782ccb241e0fa764903e4aa0e25"} Nov 02 18:05:27.308128 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:27.308098 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" event={"ID":"1ead97eb-4321-4214-92ef-ae885540da87","Type":"ContainerStarted","Data":"77318d6d2e228bb24314cb4621cb080768b618593526f7e4e734e83fe2f1fe75"} Nov 02 18:05:27.308128 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:27.308127 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" Nov 02 18:05:27.329532 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:27.329461 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" podStartSLOduration=8.329441445 podStartE2EDuration="8.329441445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:27.328928027 +0000 UTC m=+1899.117433555" watchObservedRunningTime="2026-11-02 18:05:27.329441445 +0000 UTC m=+1899.117946974" Nov 02 18:05:27.843976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:27.843749 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:28.058613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:28.058576 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:28.323340 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:28.323308 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90faafa6-7183-4a82-86cc-17721fcf2368" path="/var/lib/kubelet/pods/90faafa6-7183-4a82-86cc-17721fcf2368/volumes" Nov 02 18:05:28.549977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:28.549698 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:29.450902 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:29.449610 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:30.240254 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:30.240053 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:30.926836 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:30.926794 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:31.760466 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:31.759521 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:32.537393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:32.537311 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:33.455842 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:33.455807 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:34.219803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:34.219727 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:34.900707 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:34.900666 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:35.061985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:35.061094 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:35.214156 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:35.214123 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:35.845980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:35.842385 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:36.580236 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:36.580195 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:37.458974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:37.451208 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:38.245931 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:38.245894 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:38.326315 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:38.326280 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" Nov 02 18:05:39.136983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:39.136906 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:40.088631 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:40.088594 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:40.736815 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:40.735938 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:41.243024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:41.242994 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:41.406503 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:41.406466 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:42.090098 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:42.090048 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:42.262977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:42.262915 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:42.860323 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:42.858992 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:43.634550 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:43.634501 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:44.351978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:44.342384 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:44.976980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:44.960380 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:45.812988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:45.812297 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:46.621111 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:46.621078 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:47.440654 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:47.440619 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:47.593308 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:05:47.593272 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:05:48.160690 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:48.158488 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:48.751866 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:48.751832 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:48.894368 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:48.894331 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:49.051470 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:49.051384 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:49.241976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:49.241927 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:49.767976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:49.767914 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:49.904690 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:49.904655 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:05:49.940529 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:49.940492 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:50.111638 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:50.111554 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:51.131725 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:51.131690 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:52.418898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:52.418784 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:53.244973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:53.242708 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:54.137766 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:54.136513 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:54.813003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:54.812946 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:55.520677 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:55.520575 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:56.437164 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:56.437118 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:57.109142 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:57.109106 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:57.667977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:57.657002 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:57.798327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:57.798289 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:58.742346 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:58.742314 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:58.874444 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:58.874409 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:59.525511 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:59.525473 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:59.660237 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:05:59.660208 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:00.215406 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:00.215369 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:00.996994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:00.988271 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:01.776975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:01.772258 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:02.397494 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:02.397441 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:03.061758 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:03.061723 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:03.828507 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:03.828467 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:04.626480 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:04.626449 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:04.755882 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:04.755829 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:04.919839 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:04.919748 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:05.632105 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:05.632073 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:05.762897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:05.762851 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:06.372058 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:06.372018 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:07.084519 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:07.084480 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:07.883944 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:07.883906 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:08.561971 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:08.558860 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:09.339194 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:09.338061 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:10.038980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:10.036540 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:10.732471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:10.732437 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:11.443973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:11.434048 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:12.004736 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:12.004700 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:12.153446 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:12.153415 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:12.305560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:12.305482 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:12.927037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:12.927008 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:13.866734 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:13.866684 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:14.530974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:14.528788 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:14.729807 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:14.729769 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:14.870234 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:14.870133 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:15.220984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:15.218528 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:15.989932 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:15.982140 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:16.771820 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:16.771771 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:17.537176 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:17.537142 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:17.591079 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:06:17.591044 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:06:17.700475 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:17.700440 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:18.767557 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:18.767519 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:18.938523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:18.938489 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:19.577160 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:19.577121 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:20.271873 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:20.271832 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:20.924724 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:20.924685 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:21.625909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:21.625854 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:22.442976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:22.440270 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:23.113972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:23.112144 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:23.911973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:23.909929 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:24.739818 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:24.739781 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:24.891931 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:24.891898 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:25.036412 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:25.036329 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:25.643897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:25.643856 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:26.500108 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:26.500078 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:26.943982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:26.943875 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:27.683642 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:27.683599 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:28.433838 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:28.433804 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:29.145988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:29.144570 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:30.038118 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:30.027323 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:30.722564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:30.722522 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:31.553279 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:31.542572 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:32.262224 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:32.258755 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:32.943973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:32.939838 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:33.622935 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:33.622826 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:34.333995 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:34.332443 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:34.933130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:34.933095 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:35.746974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:35.746083 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:36.426975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:36.425486 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:37.163972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:37.163650 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:37.822148 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:37.822109 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:38.479308 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:38.478842 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:39.310703 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:39.310665 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:40.036614 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:40.024495 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:40.767998 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:40.759848 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:41.630307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:41.630256 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:41.787023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:41.786991 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:42.397087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:42.396996 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:42.673073 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:42.672997 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:06:43.044737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:43.044698 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:44.307501 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:44.307462 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:45.237936 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:45.237894 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:46.103330 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:46.103293 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:46.740017 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:46.739889 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:47.593590 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:06:47.593545 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:06:47.616129 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:47.616099 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:47.767974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:47.767062 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:47.998475 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:47.998417 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:48.712434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:48.712364 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:48.848674 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:48.848625 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:49.424204 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:49.424153 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:50.370381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:50.367548 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:51.023756 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:51.023682 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:51.752576 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:51.751310 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:52.637036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:52.636920 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:53.305364 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:53.305325 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:54.114654 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:54.114609 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:54.836228 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:54.836184 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:55.003216 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:55.003179 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:55.143506 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:55.143416 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:55.570852 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:55.570816 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:56.150974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:56.146629 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:56.916262 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:56.916203 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:57.835548 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:57.835499 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:58.186943 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:58.186855 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:58.385977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:58.362666 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:58.513489 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:58.513451 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:58.863713 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:58.863500 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:59.010605 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:59.010571 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:59.157694 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:59.157596 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:59.309141 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:59.309104 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:59.479540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:59.479507 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:59.630473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:59.630435 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:59.782034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:06:59.781908 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:00.033025 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:00.015760 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:00.163411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:00.163376 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:00.298423 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:00.298289 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:00.460981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:00.451783 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:00.803664 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:00.803626 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:01.320911 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:01.320876 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:02.034974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:02.028193 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:02.721855 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:02.721815 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:03.623346 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:03.623282 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:04.336074 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:04.336037 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:04.847291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:04.847065 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7"] Nov 02 18:07:04.922581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:04.922524 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:05.104008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:05.103902 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:05.596535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:05.590043 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:05.730531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:05.730494 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:05.882374 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:05.882280 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:06.024181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:06.024142 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:06.257109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:06.257065 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:06.779421 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:06.779389 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:06.946981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:06.944004 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:07.414247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:07.414191 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:07.557041 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:07.557007 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:08.230987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:08.230034 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:09.013361 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:09.013321 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:09.300470 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:09.292638 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:09.533265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:09.531667 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:09.884834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:09.884771 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" podUID="1ead97eb-4321-4214-92ef-ae885540da87" containerName="payload-processing" containerID="cri-o://76041e2a096690e873ef5e148abc9f459afe7782ccb241e0fa764903e4aa0e25" gracePeriod=25 Nov 02 18:07:10.093483 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.093368 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:10.243975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.243519 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:10.450498 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.450468 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" Nov 02 18:07:10.539036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.539003 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ead97eb-4321-4214-92ef-ae885540da87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ead97eb-4321-4214-92ef-ae885540da87-plugins-config-volume\") pod \"1ead97eb-4321-4214-92ef-ae885540da87\" (UID: \"1ead97eb-4321-4214-92ef-ae885540da87\") " Nov 02 18:07:10.539231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.539047 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ead97eb-4321-4214-92ef-ae885540da87-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ead97eb-4321-4214-92ef-ae885540da87-routing-overlay-volume\") pod \"1ead97eb-4321-4214-92ef-ae885540da87\" (UID: \"1ead97eb-4321-4214-92ef-ae885540da87\") " Nov 02 18:07:10.539231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.539159 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ead97eb-4321-4214-92ef-ae885540da87-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ead97eb-4321-4214-92ef-ae885540da87-provider-credentials\") pod \"1ead97eb-4321-4214-92ef-ae885540da87\" (UID: \"1ead97eb-4321-4214-92ef-ae885540da87\") " Nov 02 18:07:10.539231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.539216 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ead97eb-4321-4214-92ef-ae885540da87-kube-api-access-qhcxc\" (UniqueName: \"kubernetes.io/projected/1ead97eb-4321-4214-92ef-ae885540da87-kube-api-access-qhcxc\") pod \"1ead97eb-4321-4214-92ef-ae885540da87\" (UID: \"1ead97eb-4321-4214-92ef-ae885540da87\") " Nov 02 18:07:10.539468 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.539418 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ead97eb-4321-4214-92ef-ae885540da87-routing-overlay-volume" pod "1ead97eb-4321-4214-92ef-ae885540da87" (UID: "1ead97eb-4321-4214-92ef-ae885540da87"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:10.539563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.539481 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ead97eb-4321-4214-92ef-ae885540da87-plugins-config-volume" pod "1ead97eb-4321-4214-92ef-ae885540da87" (UID: "1ead97eb-4321-4214-92ef-ae885540da87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:10.540091 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.540068 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ead97eb-4321-4214-92ef-ae885540da87-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:10.540091 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.540092 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ead97eb-4321-4214-92ef-ae885540da87-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:10.541535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.541507 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ead97eb-4321-4214-92ef-ae885540da87-provider-credentials" pod "1ead97eb-4321-4214-92ef-ae885540da87" (UID: "1ead97eb-4321-4214-92ef-ae885540da87"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:10.541660 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.541594 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ead97eb-4321-4214-92ef-ae885540da87-kube-api-access-qhcxc" pod "1ead97eb-4321-4214-92ef-ae885540da87" (UID: "1ead97eb-4321-4214-92ef-ae885540da87"). InnerVolumeSpecName "kube-api-access-qhcxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:10.642245 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.642211 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ead97eb-4321-4214-92ef-ae885540da87-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:10.642245 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.642239 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhcxc\" (UniqueName: \"kubernetes.io/projected/1ead97eb-4321-4214-92ef-ae885540da87-kube-api-access-qhcxc\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:10.966035 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.966002 5549 generic.go:356] "Generic (PLEG): container finished" podID="1ead97eb-4321-4214-92ef-ae885540da87" containerID="76041e2a096690e873ef5e148abc9f459afe7782ccb241e0fa764903e4aa0e25" exitCode=0 Nov 02 18:07:10.966219 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.966047 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" Nov 02 18:07:10.966219 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.966083 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" event={"ID":"1ead97eb-4321-4214-92ef-ae885540da87","Type":"ContainerDied","Data":"76041e2a096690e873ef5e148abc9f459afe7782ccb241e0fa764903e4aa0e25"} Nov 02 18:07:10.966219 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.966145 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7" event={"ID":"1ead97eb-4321-4214-92ef-ae885540da87","Type":"ContainerDied","Data":"77318d6d2e228bb24314cb4621cb080768b618593526f7e4e734e83fe2f1fe75"} Nov 02 18:07:10.966219 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.966175 5549 scope.go:122] "RemoveContainer" containerID="76041e2a096690e873ef5e148abc9f459afe7782ccb241e0fa764903e4aa0e25" Nov 02 18:07:10.978783 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.978761 5549 scope.go:122] "RemoveContainer" containerID="76041e2a096690e873ef5e148abc9f459afe7782ccb241e0fa764903e4aa0e25" Nov 02 18:07:10.979212 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:07:10.979187 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76041e2a096690e873ef5e148abc9f459afe7782ccb241e0fa764903e4aa0e25\": container with ID starting with 76041e2a096690e873ef5e148abc9f459afe7782ccb241e0fa764903e4aa0e25 not found: ID does not exist" containerID="76041e2a096690e873ef5e148abc9f459afe7782ccb241e0fa764903e4aa0e25" Nov 02 18:07:10.979275 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.979234 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76041e2a096690e873ef5e148abc9f459afe7782ccb241e0fa764903e4aa0e25"} err="failed to get container status \"76041e2a096690e873ef5e148abc9f459afe7782ccb241e0fa764903e4aa0e25\": rpc error: code = NotFound desc = could not find container \"76041e2a096690e873ef5e148abc9f459afe7782ccb241e0fa764903e4aa0e25\": container with ID starting with 76041e2a096690e873ef5e148abc9f459afe7782ccb241e0fa764903e4aa0e25 not found: ID does not exist" Nov 02 18:07:10.993150 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.993114 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7"] Nov 02 18:07:10.998577 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:10.998550 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5976wtlh7"] Nov 02 18:07:11.079977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:11.075707 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:11.776548 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:11.774830 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:12.322585 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:12.322552 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ead97eb-4321-4214-92ef-ae885540da87" path="/var/lib/kubelet/pods/1ead97eb-4321-4214-92ef-ae885540da87/volumes" Nov 02 18:07:12.447791 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:12.447748 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:13.129980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:13.128028 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:13.790975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:13.790104 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:14.189285 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:14.189203 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:14.339997 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:14.339961 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:14.491550 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:14.491505 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:14.690454 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:14.690421 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:14.864361 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:14.864273 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:15.055292 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:15.052631 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:15.543570 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:15.543525 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:15.708094 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:15.705689 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:16.142551 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.142517 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8"] Nov 02 18:07:16.143548 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.143525 5549 cpu_manager.go:414] "containerMap: removing container" podUID="1ead97eb-4321-4214-92ef-ae885540da87" containerName="payload-processing" Nov 02 18:07:16.143614 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.143555 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ead97eb-4321-4214-92ef-ae885540da87" containerName="payload-processing" Nov 02 18:07:16.143782 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.143770 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ead97eb-4321-4214-92ef-ae885540da87" containerName="payload-processing" Nov 02 18:07:16.148331 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.148311 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" Nov 02 18:07:16.156132 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.156077 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5ec436\"/\"agc-cred-c13e2e0d36b2\"" Nov 02 18:07:16.157729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.157704 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5ec436\"/\"payload-processing-external-model-plugins-e2e-extmod-a-5ec436\"" Nov 02 18:07:16.157887 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.157724 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5ec436\"/\"openshift-service-ca.crt\"" Nov 02 18:07:16.157887 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.157788 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5ec436\"/\"routing-overlay\"" Nov 02 18:07:16.157887 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.157810 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5ec436\"/\"kube-root-ca.crt\"" Nov 02 18:07:16.163297 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.163272 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8"] Nov 02 18:07:16.188157 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.188120 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:16.274497 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.274444 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxdjp\" (UniqueName: \"kubernetes.io/projected/225611b2-2186-40eb-895c-2219486939af-kube-api-access-jxdjp\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8\" (UID: \"225611b2-2186-40eb-895c-2219486939af\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" Nov 02 18:07:16.274691 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.274536 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/225611b2-2186-40eb-895c-2219486939af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8\" (UID: \"225611b2-2186-40eb-895c-2219486939af\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" Nov 02 18:07:16.275013 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.274946 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/225611b2-2186-40eb-895c-2219486939af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8\" (UID: \"225611b2-2186-40eb-895c-2219486939af\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" Nov 02 18:07:16.275167 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.275088 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225611b2-2186-40eb-895c-2219486939af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8\" (UID: \"225611b2-2186-40eb-895c-2219486939af\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" Nov 02 18:07:16.340142 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.340072 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:16.376938 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.376893 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/225611b2-2186-40eb-895c-2219486939af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8\" (UID: \"225611b2-2186-40eb-895c-2219486939af\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" Nov 02 18:07:16.377148 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.376948 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225611b2-2186-40eb-895c-2219486939af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8\" (UID: \"225611b2-2186-40eb-895c-2219486939af\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" Nov 02 18:07:16.377419 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.377392 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxdjp\" (UniqueName: \"kubernetes.io/projected/225611b2-2186-40eb-895c-2219486939af-kube-api-access-jxdjp\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8\" (UID: \"225611b2-2186-40eb-895c-2219486939af\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" Nov 02 18:07:16.377551 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.377469 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/225611b2-2186-40eb-895c-2219486939af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8\" (UID: \"225611b2-2186-40eb-895c-2219486939af\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" Nov 02 18:07:16.377938 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.377910 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225611b2-2186-40eb-895c-2219486939af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8\" (UID: \"225611b2-2186-40eb-895c-2219486939af\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" Nov 02 18:07:16.378115 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.377988 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/225611b2-2186-40eb-895c-2219486939af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8\" (UID: \"225611b2-2186-40eb-895c-2219486939af\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" Nov 02 18:07:16.380480 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.380458 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/225611b2-2186-40eb-895c-2219486939af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8\" (UID: \"225611b2-2186-40eb-895c-2219486939af\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" Nov 02 18:07:16.385773 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.385746 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxdjp\" (UniqueName: \"kubernetes.io/projected/225611b2-2186-40eb-895c-2219486939af-kube-api-access-jxdjp\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8\" (UID: \"225611b2-2186-40eb-895c-2219486939af\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" Nov 02 18:07:16.479810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.479239 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" Nov 02 18:07:16.486766 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:16.482746 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:17.185878 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:17.185843 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8"] Nov 02 18:07:17.314115 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:17.314085 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9"] Nov 02 18:07:17.314285 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:17.314261 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" podUID="54201446-6418-4b65-b291-9263a4781e30" containerName="istio-proxy" containerID="cri-o://4ac12647202ab8c7c8f309aee5b5de594ec37e69391e30ee210cd434b02936f0" gracePeriod=30 Nov 02 18:07:17.590756 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:07:17.590719 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:07:18.101330 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:18.101293 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" event={"ID":"225611b2-2186-40eb-895c-2219486939af","Type":"ContainerStarted","Data":"f29357c8c643b65c0c55148e63dfd2d60c1c448a81c71b005e1ceafe7eb9583a"} Nov 02 18:07:18.101495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:18.101352 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" event={"ID":"225611b2-2186-40eb-895c-2219486939af","Type":"ContainerStarted","Data":"05b87a9fc1d659882ec096cced9b2a2a7733ae85f0d028cb4adbf23ca8af08f4"} Nov 02 18:07:18.101495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:18.101400 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" Nov 02 18:07:18.157975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:18.153157 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" podStartSLOduration=2.153135347 podStartE2EDuration="2.153135347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:18.120537666 +0000 UTC m=+2009.909043206" watchObservedRunningTime="2026-11-02 18:07:18.153135347 +0000 UTC m=+2009.941640876" Nov 02 18:07:18.165980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:18.160418 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:18.994978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:18.994368 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:19.809973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:19.808869 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:20.942977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:20.942927 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:21.172244 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:21.172211 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:21.352230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:21.352196 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:22.064526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.064486 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:22.406735 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.406702 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw"] Nov 02 18:07:22.411729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.411711 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" Nov 02 18:07:22.415460 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.415413 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:22.416151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.416126 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-2db162\"" Nov 02 18:07:22.418881 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.418858 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw"] Nov 02 18:07:22.425027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.425005 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-2db162-dockercfg-68pns\"" Nov 02 18:07:22.496327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.496294 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95748775-ad15-4a15-8031-a0486a2cf317-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw\" (UID: \"95748775-ad15-4a15-8031-a0486a2cf317\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" Nov 02 18:07:22.496494 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.496383 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb9gv\" (UniqueName: \"kubernetes.io/projected/95748775-ad15-4a15-8031-a0486a2cf317-kube-api-access-jb9gv\") pod \"payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw\" (UID: \"95748775-ad15-4a15-8031-a0486a2cf317\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" Nov 02 18:07:22.586046 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.585856 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:22.620196 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.619295 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95748775-ad15-4a15-8031-a0486a2cf317-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw\" (UID: \"95748775-ad15-4a15-8031-a0486a2cf317\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" Nov 02 18:07:22.620196 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.619429 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb9gv\" (UniqueName: \"kubernetes.io/projected/95748775-ad15-4a15-8031-a0486a2cf317-kube-api-access-jb9gv\") pod \"payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw\" (UID: \"95748775-ad15-4a15-8031-a0486a2cf317\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" Nov 02 18:07:22.718976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.620940 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95748775-ad15-4a15-8031-a0486a2cf317-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw\" (UID: \"95748775-ad15-4a15-8031-a0486a2cf317\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" Nov 02 18:07:22.895341 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.890517 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb9gv\" (UniqueName: \"kubernetes.io/projected/95748775-ad15-4a15-8031-a0486a2cf317-kube-api-access-jb9gv\") pod \"payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw\" (UID: \"95748775-ad15-4a15-8031-a0486a2cf317\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" Nov 02 18:07:22.980705 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.980675 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 18:07:22.995641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.995285 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-workload-certs\") pod \"54201446-6418-4b65-b291-9263a4781e30\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " Nov 02 18:07:22.995641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.995441 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54201446-6418-4b65-b291-9263a4781e30-kube-api-access-6t5cp\" (UniqueName: \"kubernetes.io/projected/54201446-6418-4b65-b291-9263a4781e30-kube-api-access-6t5cp\") pod \"54201446-6418-4b65-b291-9263a4781e30\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " Nov 02 18:07:22.995641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.995557 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-istio-envoy\") pod \"54201446-6418-4b65-b291-9263a4781e30\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " Nov 02 18:07:22.995641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.995601 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-workload-certs" pod "54201446-6418-4b65-b291-9263a4781e30" (UID: "54201446-6418-4b65-b291-9263a4781e30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:22.996106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.995724 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-workload-socket\") pod \"54201446-6418-4b65-b291-9263a4781e30\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " Nov 02 18:07:22.996106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.995762 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54201446-6418-4b65-b291-9263a4781e30-istio-token\" (UniqueName: \"kubernetes.io/projected/54201446-6418-4b65-b291-9263a4781e30-istio-token\") pod \"54201446-6418-4b65-b291-9263a4781e30\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " Nov 02 18:07:22.996106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.995805 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54201446-6418-4b65-b291-9263a4781e30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54201446-6418-4b65-b291-9263a4781e30-istio-podinfo\") pod \"54201446-6418-4b65-b291-9263a4781e30\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " Nov 02 18:07:22.996106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.995858 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-credential-socket\") pod \"54201446-6418-4b65-b291-9263a4781e30\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " Nov 02 18:07:22.996106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.995936 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-istio-data\") pod \"54201446-6418-4b65-b291-9263a4781e30\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " Nov 02 18:07:22.997149 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.996381 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54201446-6418-4b65-b291-9263a4781e30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54201446-6418-4b65-b291-9263a4781e30-istiod-ca-cert\") pod \"54201446-6418-4b65-b291-9263a4781e30\" (UID: \"54201446-6418-4b65-b291-9263a4781e30\") " Nov 02 18:07:22.997346 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.997323 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:22.997732 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.997705 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-credential-socket" pod "54201446-6418-4b65-b291-9263a4781e30" (UID: "54201446-6418-4b65-b291-9263a4781e30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:22.999996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.998162 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54201446-6418-4b65-b291-9263a4781e30-istiod-ca-cert" pod "54201446-6418-4b65-b291-9263a4781e30" (UID: "54201446-6418-4b65-b291-9263a4781e30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:22.999996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.998549 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-istio-data" pod "54201446-6418-4b65-b291-9263a4781e30" (UID: "54201446-6418-4b65-b291-9263a4781e30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:22.999996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:22.998681 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-workload-socket" pod "54201446-6418-4b65-b291-9263a4781e30" (UID: "54201446-6418-4b65-b291-9263a4781e30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:23.000450 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.000420 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54201446-6418-4b65-b291-9263a4781e30-istio-token" pod "54201446-6418-4b65-b291-9263a4781e30" (UID: "54201446-6418-4b65-b291-9263a4781e30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:23.001602 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.001578 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54201446-6418-4b65-b291-9263a4781e30-istio-podinfo" pod "54201446-6418-4b65-b291-9263a4781e30" (UID: "54201446-6418-4b65-b291-9263a4781e30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:07:23.001672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.001593 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-istio-envoy" pod "54201446-6418-4b65-b291-9263a4781e30" (UID: "54201446-6418-4b65-b291-9263a4781e30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:23.001935 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.001918 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54201446-6418-4b65-b291-9263a4781e30-kube-api-access-6t5cp" pod "54201446-6418-4b65-b291-9263a4781e30" (UID: "54201446-6418-4b65-b291-9263a4781e30"). InnerVolumeSpecName "kube-api-access-6t5cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:23.026002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.025964 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv"] Nov 02 18:07:23.027210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.027182 5549 cpu_manager.go:414] "containerMap: removing container" podUID="54201446-6418-4b65-b291-9263a4781e30" containerName="istio-proxy" Nov 02 18:07:23.027210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.027205 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54201446-6418-4b65-b291-9263a4781e30" containerName="istio-proxy" Nov 02 18:07:23.027379 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.027369 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="54201446-6418-4b65-b291-9263a4781e30" containerName="istio-proxy" Nov 02 18:07:23.031862 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.031840 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" Nov 02 18:07:23.042349 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.042319 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" Nov 02 18:07:23.044360 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.043929 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv"] Nov 02 18:07:23.099240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.099196 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxwfp\" (UniqueName: \"kubernetes.io/projected/d132c2c1-5fab-4ab4-82f1-892f5aad2771-kube-api-access-nxwfp\") pod \"payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv\" (UID: \"d132c2c1-5fab-4ab4-82f1-892f5aad2771\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" Nov 02 18:07:23.099798 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.099365 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d132c2c1-5fab-4ab4-82f1-892f5aad2771-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv\" (UID: \"d132c2c1-5fab-4ab4-82f1-892f5aad2771\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" Nov 02 18:07:23.099798 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.099776 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:23.099926 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.099806 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54201446-6418-4b65-b291-9263a4781e30-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:23.099926 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.099822 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54201446-6418-4b65-b291-9263a4781e30-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:23.099926 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.099835 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:23.099926 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.099844 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:23.099926 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.099852 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54201446-6418-4b65-b291-9263a4781e30-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:23.099926 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.099862 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t5cp\" (UniqueName: \"kubernetes.io/projected/54201446-6418-4b65-b291-9263a4781e30-kube-api-access-6t5cp\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:23.099926 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.099871 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54201446-6418-4b65-b291-9263a4781e30-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:23.105047 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.105020 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h"] Nov 02 18:07:23.105286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.105259 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" podUID="7095e3de-ee27-40f3-aabc-c7c149a5e66f" containerName="istio-proxy" containerID="cri-o://650575ab2dc870660acec8c54713817b64f073630aa09866d76717b5e52fe3fd" gracePeriod=30 Nov 02 18:07:23.149760 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.149727 5549 generic.go:356] "Generic (PLEG): container finished" podID="54201446-6418-4b65-b291-9263a4781e30" containerID="4ac12647202ab8c7c8f309aee5b5de594ec37e69391e30ee210cd434b02936f0" exitCode=0 Nov 02 18:07:23.149999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.149801 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" event={"ID":"54201446-6418-4b65-b291-9263a4781e30","Type":"ContainerDied","Data":"4ac12647202ab8c7c8f309aee5b5de594ec37e69391e30ee210cd434b02936f0"} Nov 02 18:07:23.149999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.149805 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" Nov 02 18:07:23.149999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.149840 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9" event={"ID":"54201446-6418-4b65-b291-9263a4781e30","Type":"ContainerDied","Data":"accf20606a49512d0b028d5eabeca19ce3683e48ddd4bc3a0039ddc7cef5f056"} Nov 02 18:07:23.149999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.149862 5549 scope.go:122] "RemoveContainer" containerID="4ac12647202ab8c7c8f309aee5b5de594ec37e69391e30ee210cd434b02936f0" Nov 02 18:07:23.163648 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.163610 5549 scope.go:122] "RemoveContainer" containerID="4ac12647202ab8c7c8f309aee5b5de594ec37e69391e30ee210cd434b02936f0" Nov 02 18:07:23.164211 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:07:23.164172 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac12647202ab8c7c8f309aee5b5de594ec37e69391e30ee210cd434b02936f0\": container with ID starting with 4ac12647202ab8c7c8f309aee5b5de594ec37e69391e30ee210cd434b02936f0 not found: ID does not exist" containerID="4ac12647202ab8c7c8f309aee5b5de594ec37e69391e30ee210cd434b02936f0" Nov 02 18:07:23.164308 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.164216 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac12647202ab8c7c8f309aee5b5de594ec37e69391e30ee210cd434b02936f0"} err="failed to get container status \"4ac12647202ab8c7c8f309aee5b5de594ec37e69391e30ee210cd434b02936f0\": rpc error: code = NotFound desc = could not find container \"4ac12647202ab8c7c8f309aee5b5de594ec37e69391e30ee210cd434b02936f0\": container with ID starting with 4ac12647202ab8c7c8f309aee5b5de594ec37e69391e30ee210cd434b02936f0 not found: ID does not exist" Nov 02 18:07:23.178511 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.178479 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9"] Nov 02 18:07:23.190377 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.190347 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-r4pz9"] Nov 02 18:07:23.202940 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.201390 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxwfp\" (UniqueName: \"kubernetes.io/projected/d132c2c1-5fab-4ab4-82f1-892f5aad2771-kube-api-access-nxwfp\") pod \"payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv\" (UID: \"d132c2c1-5fab-4ab4-82f1-892f5aad2771\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" Nov 02 18:07:23.202940 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.201721 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d132c2c1-5fab-4ab4-82f1-892f5aad2771-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv\" (UID: \"d132c2c1-5fab-4ab4-82f1-892f5aad2771\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" Nov 02 18:07:23.202940 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.202399 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d132c2c1-5fab-4ab4-82f1-892f5aad2771-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv\" (UID: \"d132c2c1-5fab-4ab4-82f1-892f5aad2771\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" Nov 02 18:07:23.209912 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.209887 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxwfp\" (UniqueName: \"kubernetes.io/projected/d132c2c1-5fab-4ab4-82f1-892f5aad2771-kube-api-access-nxwfp\") pod \"payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv\" (UID: \"d132c2c1-5fab-4ab4-82f1-892f5aad2771\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" Nov 02 18:07:23.324177 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.324090 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:23.368982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.367261 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" Nov 02 18:07:23.404130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.403853 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw"] Nov 02 18:07:23.404860 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:07:23.404807 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95748775_ad15_4a15_8031_a0486a2cf317.slice/crio-739f1d6caea4ce1da77e8a051191c6ab54d7f7a8eb113f496fd19df5a39aa378 WatchSource:0}: Error finding container 739f1d6caea4ce1da77e8a051191c6ab54d7f7a8eb113f496fd19df5a39aa378: Status 404 returned error can't find the container with id 739f1d6caea4ce1da77e8a051191c6ab54d7f7a8eb113f496fd19df5a39aa378 Nov 02 18:07:23.721446 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:07:23.721400 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd132c2c1_5fab_4ab4_82f1_892f5aad2771.slice/crio-bf91f898f0590d2819c93eb53ded361f7969e292cf5ecde178b377f9ba780e5d WatchSource:0}: Error finding container bf91f898f0590d2819c93eb53ded361f7969e292cf5ecde178b377f9ba780e5d: Status 404 returned error can't find the container with id bf91f898f0590d2819c93eb53ded361f7969e292cf5ecde178b377f9ba780e5d Nov 02 18:07:23.721713 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:23.721684 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv"] Nov 02 18:07:24.192710 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:24.192610 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:24.209635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:24.209585 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" event={"ID":"d132c2c1-5fab-4ab4-82f1-892f5aad2771","Type":"ContainerStarted","Data":"39299b653b1bd4eeea63c84ba65a457d0659af67b23358e96a795126a2ea8084"} Nov 02 18:07:24.209810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:24.209687 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" event={"ID":"d132c2c1-5fab-4ab4-82f1-892f5aad2771","Type":"ContainerStarted","Data":"bf91f898f0590d2819c93eb53ded361f7969e292cf5ecde178b377f9ba780e5d"} Nov 02 18:07:24.209810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:24.209743 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" Nov 02 18:07:24.211755 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:24.211718 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" event={"ID":"95748775-ad15-4a15-8031-a0486a2cf317","Type":"ContainerStarted","Data":"b145bf25487f02697495c258d53fef72f69773b77ab077306d89c9e737318600"} Nov 02 18:07:24.211894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:24.211778 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" event={"ID":"95748775-ad15-4a15-8031-a0486a2cf317","Type":"ContainerStarted","Data":"739f1d6caea4ce1da77e8a051191c6ab54d7f7a8eb113f496fd19df5a39aa378"} Nov 02 18:07:24.212137 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:24.212119 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" Nov 02 18:07:24.229197 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:24.229146 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" podStartSLOduration=2.229130726 podStartE2EDuration="2.229130726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:24.227525293 +0000 UTC m=+2016.016030824" watchObservedRunningTime="2026-11-02 18:07:24.229130726 +0000 UTC m=+2016.017636257" Nov 02 18:07:24.245969 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:24.245912 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" podStartSLOduration=2.245900078 podStartE2EDuration="2.245900078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:24.243439666 +0000 UTC m=+2016.031945197" watchObservedRunningTime="2026-11-02 18:07:24.245900078 +0000 UTC m=+2016.034405605" Nov 02 18:07:24.323543 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:24.323509 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54201446-6418-4b65-b291-9263a4781e30" path="/var/lib/kubelet/pods/54201446-6418-4b65-b291-9263a4781e30/volumes" Nov 02 18:07:24.770122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:24.770082 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:07:25.059986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:25.059747 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:25.741137 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:25.739682 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:26.638977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:26.637783 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:27.446247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:27.438968 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:28.289978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.289594 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:28.316548 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.316348 5549 generic.go:356] "Generic (PLEG): container finished" podID="7095e3de-ee27-40f3-aabc-c7c149a5e66f" containerID="650575ab2dc870660acec8c54713817b64f073630aa09866d76717b5e52fe3fd" exitCode=0 Nov 02 18:07:28.316548 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.316460 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" event={"ID":"7095e3de-ee27-40f3-aabc-c7c149a5e66f","Type":"ContainerDied","Data":"650575ab2dc870660acec8c54713817b64f073630aa09866d76717b5e52fe3fd"} Nov 02 18:07:28.512429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.512402 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 18:07:28.676123 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.676037 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-data\") pod \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " Nov 02 18:07:28.676293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.676126 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-workload-certs\") pod \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " Nov 02 18:07:28.676293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.676220 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istiod-ca-cert\") pod \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " Nov 02 18:07:28.676436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.676389 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-data" pod "7095e3de-ee27-40f3-aabc-c7c149a5e66f" (UID: "7095e3de-ee27-40f3-aabc-c7c149a5e66f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:28.676436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.676412 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-workload-certs" pod "7095e3de-ee27-40f3-aabc-c7c149a5e66f" (UID: "7095e3de-ee27-40f3-aabc-c7c149a5e66f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:28.676559 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.676437 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-token\" (UniqueName: \"kubernetes.io/projected/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-token\") pod \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " Nov 02 18:07:28.676559 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.676491 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7095e3de-ee27-40f3-aabc-c7c149a5e66f-kube-api-access-mm9xz\" (UniqueName: \"kubernetes.io/projected/7095e3de-ee27-40f3-aabc-c7c149a5e66f-kube-api-access-mm9xz\") pod \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " Nov 02 18:07:28.676559 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.676534 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-envoy\") pod \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " Nov 02 18:07:28.676727 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.676596 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istiod-ca-cert" pod "7095e3de-ee27-40f3-aabc-c7c149a5e66f" (UID: "7095e3de-ee27-40f3-aabc-c7c149a5e66f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:28.676727 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.676612 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-podinfo\") pod \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " Nov 02 18:07:28.676727 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.676670 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-workload-socket\") pod \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " Nov 02 18:07:28.676727 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.676713 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-credential-socket\") pod \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\" (UID: \"7095e3de-ee27-40f3-aabc-c7c149a5e66f\") " Nov 02 18:07:28.677031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.676934 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-workload-socket" pod "7095e3de-ee27-40f3-aabc-c7c149a5e66f" (UID: "7095e3de-ee27-40f3-aabc-c7c149a5e66f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:28.677031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.676978 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-credential-socket" pod "7095e3de-ee27-40f3-aabc-c7c149a5e66f" (UID: "7095e3de-ee27-40f3-aabc-c7c149a5e66f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:28.677705 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.677682 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:28.677834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.677708 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:28.677834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.677722 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:28.677834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.677735 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:28.677834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.677747 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:28.679112 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.679083 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-envoy" pod "7095e3de-ee27-40f3-aabc-c7c149a5e66f" (UID: "7095e3de-ee27-40f3-aabc-c7c149a5e66f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:28.679250 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.679226 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-token" pod "7095e3de-ee27-40f3-aabc-c7c149a5e66f" (UID: "7095e3de-ee27-40f3-aabc-c7c149a5e66f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:28.679410 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.679385 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-podinfo" pod "7095e3de-ee27-40f3-aabc-c7c149a5e66f" (UID: "7095e3de-ee27-40f3-aabc-c7c149a5e66f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:07:28.679522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.679409 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7095e3de-ee27-40f3-aabc-c7c149a5e66f-kube-api-access-mm9xz" pod "7095e3de-ee27-40f3-aabc-c7c149a5e66f" (UID: "7095e3de-ee27-40f3-aabc-c7c149a5e66f"). InnerVolumeSpecName "kube-api-access-mm9xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:28.779226 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.779190 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:28.779226 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.779226 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm9xz\" (UniqueName: \"kubernetes.io/projected/7095e3de-ee27-40f3-aabc-c7c149a5e66f-kube-api-access-mm9xz\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:28.779451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.779241 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:28.779451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.779255 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7095e3de-ee27-40f3-aabc-c7c149a5e66f-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:28.943025 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:28.941327 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:29.113792 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:29.113758 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" Nov 02 18:07:29.345339 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:29.345290 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" event={"ID":"7095e3de-ee27-40f3-aabc-c7c149a5e66f","Type":"ContainerDied","Data":"d3fe4b1c897a27bddd54ef132818cd05e0dcf0c806eff3b67d1c41a71e8ddfc2"} Nov 02 18:07:29.345545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:29.345349 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h" Nov 02 18:07:29.345545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:29.345391 5549 scope.go:122] "RemoveContainer" containerID="650575ab2dc870660acec8c54713817b64f073630aa09866d76717b5e52fe3fd" Nov 02 18:07:29.374825 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:29.374781 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h"] Nov 02 18:07:29.379362 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:29.379335 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-rqj8h"] Nov 02 18:07:29.648035 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:29.647337 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:29.851004 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:29.850946 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8"] Nov 02 18:07:30.322723 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:30.322691 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7095e3de-ee27-40f3-aabc-c7c149a5e66f" path="/var/lib/kubelet/pods/7095e3de-ee27-40f3-aabc-c7c149a5e66f/volumes" Nov 02 18:07:30.452039 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:30.439581 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:30.647811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:30.644363 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:30.837976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:30.836592 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:31.257703 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:31.257663 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:31.641604 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:31.641513 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:32.361974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:32.339127 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:33.036815 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:33.036651 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:33.711921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:33.711879 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:34.531016 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:34.530978 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:34.851775 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:34.851682 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" podUID="225611b2-2186-40eb-895c-2219486939af" containerName="payload-processing" containerID="cri-o://f29357c8c643b65c0c55148e63dfd2d60c1c448a81c71b005e1ceafe7eb9583a" gracePeriod=25 Nov 02 18:07:35.154354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.154328 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" Nov 02 18:07:35.211314 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.211278 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:35.226532 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.226503 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" Nov 02 18:07:35.226698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.226584 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" Nov 02 18:07:35.248507 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.248477 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/225611b2-2186-40eb-895c-2219486939af-provider-credentials\" (UniqueName: \"kubernetes.io/projected/225611b2-2186-40eb-895c-2219486939af-provider-credentials\") pod \"225611b2-2186-40eb-895c-2219486939af\" (UID: \"225611b2-2186-40eb-895c-2219486939af\") " Nov 02 18:07:35.248686 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.248524 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/225611b2-2186-40eb-895c-2219486939af-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/225611b2-2186-40eb-895c-2219486939af-routing-overlay-volume\") pod \"225611b2-2186-40eb-895c-2219486939af\" (UID: \"225611b2-2186-40eb-895c-2219486939af\") " Nov 02 18:07:35.248803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.248786 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/225611b2-2186-40eb-895c-2219486939af-kube-api-access-jxdjp\" (UniqueName: \"kubernetes.io/projected/225611b2-2186-40eb-895c-2219486939af-kube-api-access-jxdjp\") pod \"225611b2-2186-40eb-895c-2219486939af\" (UID: \"225611b2-2186-40eb-895c-2219486939af\") " Nov 02 18:07:35.248907 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.248885 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225611b2-2186-40eb-895c-2219486939af-routing-overlay-volume" pod "225611b2-2186-40eb-895c-2219486939af" (UID: "225611b2-2186-40eb-895c-2219486939af"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:35.249000 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.248973 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/225611b2-2186-40eb-895c-2219486939af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225611b2-2186-40eb-895c-2219486939af-plugins-config-volume\") pod \"225611b2-2186-40eb-895c-2219486939af\" (UID: \"225611b2-2186-40eb-895c-2219486939af\") " Nov 02 18:07:35.249441 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.249416 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225611b2-2186-40eb-895c-2219486939af-plugins-config-volume" pod "225611b2-2186-40eb-895c-2219486939af" (UID: "225611b2-2186-40eb-895c-2219486939af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:35.249816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.249796 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225611b2-2186-40eb-895c-2219486939af-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:35.249867 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.249824 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/225611b2-2186-40eb-895c-2219486939af-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:35.251073 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.251051 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225611b2-2186-40eb-895c-2219486939af-kube-api-access-jxdjp" pod "225611b2-2186-40eb-895c-2219486939af" (UID: "225611b2-2186-40eb-895c-2219486939af"). InnerVolumeSpecName "kube-api-access-jxdjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:35.251189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.251175 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225611b2-2186-40eb-895c-2219486939af-provider-credentials" pod "225611b2-2186-40eb-895c-2219486939af" (UID: "225611b2-2186-40eb-895c-2219486939af"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:35.351942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.351901 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxdjp\" (UniqueName: \"kubernetes.io/projected/225611b2-2186-40eb-895c-2219486939af-kube-api-access-jxdjp\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:35.351942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.351931 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/225611b2-2186-40eb-895c-2219486939af-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:35.492719 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.492575 5549 generic.go:356] "Generic (PLEG): container finished" podID="225611b2-2186-40eb-895c-2219486939af" containerID="f29357c8c643b65c0c55148e63dfd2d60c1c448a81c71b005e1ceafe7eb9583a" exitCode=0 Nov 02 18:07:35.492922 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.492794 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" event={"ID":"225611b2-2186-40eb-895c-2219486939af","Type":"ContainerDied","Data":"f29357c8c643b65c0c55148e63dfd2d60c1c448a81c71b005e1ceafe7eb9583a"} Nov 02 18:07:35.492922 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.492833 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" event={"ID":"225611b2-2186-40eb-895c-2219486939af","Type":"ContainerDied","Data":"05b87a9fc1d659882ec096cced9b2a2a7733ae85f0d028cb4adbf23ca8af08f4"} Nov 02 18:07:35.492922 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.492855 5549 scope.go:122] "RemoveContainer" containerID="f29357c8c643b65c0c55148e63dfd2d60c1c448a81c71b005e1ceafe7eb9583a" Nov 02 18:07:35.493119 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.493018 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8" Nov 02 18:07:35.508412 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.508389 5549 scope.go:122] "RemoveContainer" containerID="f29357c8c643b65c0c55148e63dfd2d60c1c448a81c71b005e1ceafe7eb9583a" Nov 02 18:07:35.508714 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:07:35.508693 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f29357c8c643b65c0c55148e63dfd2d60c1c448a81c71b005e1ceafe7eb9583a\": container with ID starting with f29357c8c643b65c0c55148e63dfd2d60c1c448a81c71b005e1ceafe7eb9583a not found: ID does not exist" containerID="f29357c8c643b65c0c55148e63dfd2d60c1c448a81c71b005e1ceafe7eb9583a" Nov 02 18:07:35.508776 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.508722 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f29357c8c643b65c0c55148e63dfd2d60c1c448a81c71b005e1ceafe7eb9583a"} err="failed to get container status \"f29357c8c643b65c0c55148e63dfd2d60c1c448a81c71b005e1ceafe7eb9583a\": rpc error: code = NotFound desc = could not find container \"f29357c8c643b65c0c55148e63dfd2d60c1c448a81c71b005e1ceafe7eb9583a\": container with ID starting with f29357c8c643b65c0c55148e63dfd2d60c1c448a81c71b005e1ceafe7eb9583a not found: ID does not exist" Nov 02 18:07:35.523652 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.523626 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8"] Nov 02 18:07:35.528724 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.528700 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58fflm9s8"] Nov 02 18:07:35.926276 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:35.925734 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:36.324038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:36.324005 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="225611b2-2186-40eb-895c-2219486939af" path="/var/lib/kubelet/pods/225611b2-2186-40eb-895c-2219486939af/volumes" Nov 02 18:07:36.626840 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:36.623481 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:37.355931 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:37.352752 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:37.572340 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:37.572295 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:37.719039 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:37.719001 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:37.860998 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:37.860937 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:38.024975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:38.024852 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:38.821565 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:38.821466 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:39.507399 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:39.507274 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:40.234977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:40.230231 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:40.899489 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:40.899450 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:41.040843 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:41.038944 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:41.807978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:41.800329 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:42.536612 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:42.536494 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:43.426692 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:43.426653 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:44.081707 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:44.081619 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:44.254397 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:44.254358 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:44.402389 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:44.402304 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:44.543245 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:44.543213 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:44.708411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:44.708371 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:44.862348 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:44.862303 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:45.040045 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:45.039593 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:45.324179 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:45.323366 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:45.688242 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:45.688163 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:45.841988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:45.837136 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:46.355453 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.355416 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:46.413984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.413932 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm"] Nov 02 18:07:46.414944 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.414918 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7095e3de-ee27-40f3-aabc-c7c149a5e66f" containerName="istio-proxy" Nov 02 18:07:46.414944 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.414940 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7095e3de-ee27-40f3-aabc-c7c149a5e66f" containerName="istio-proxy" Nov 02 18:07:46.415164 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.415002 5549 cpu_manager.go:414] "containerMap: removing container" podUID="225611b2-2186-40eb-895c-2219486939af" containerName="payload-processing" Nov 02 18:07:46.415164 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.415012 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="225611b2-2186-40eb-895c-2219486939af" containerName="payload-processing" Nov 02 18:07:46.415275 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.415261 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7095e3de-ee27-40f3-aabc-c7c149a5e66f" containerName="istio-proxy" Nov 02 18:07:46.415315 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.415279 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="225611b2-2186-40eb-895c-2219486939af" containerName="payload-processing" Nov 02 18:07:46.419273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.419253 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" Nov 02 18:07:46.428333 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.428194 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"routing-overlay\"" Nov 02 18:07:46.428333 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.428257 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"payload-processing-external-model-e2e--cd90e2f4-dockercfg-7s22q\"" Nov 02 18:07:46.428333 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.428279 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"openshift-service-ca.crt\"" Nov 02 18:07:46.428333 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.428278 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"payload-processing-external-model-plugins-e2e-extmod-a-42d786\"" Nov 02 18:07:46.428333 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.428317 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"kube-root-ca.crt\"" Nov 02 18:07:46.428850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.428828 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"agc-cred-c4e11282498a\"" Nov 02 18:07:46.430238 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.430214 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm"] Nov 02 18:07:46.506342 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.506306 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:46.508929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.508899 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a29bc8bd-1d33-4a87-9239-8ef93b45909e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm\" (UID: \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" Nov 02 18:07:46.509065 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.509026 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p42n\" (UniqueName: \"kubernetes.io/projected/a29bc8bd-1d33-4a87-9239-8ef93b45909e-kube-api-access-9p42n\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm\" (UID: \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" Nov 02 18:07:46.509124 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.509096 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29bc8bd-1d33-4a87-9239-8ef93b45909e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm\" (UID: \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" Nov 02 18:07:46.509218 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.509201 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a29bc8bd-1d33-4a87-9239-8ef93b45909e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm\" (UID: \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" Nov 02 18:07:46.610600 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.610513 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a29bc8bd-1d33-4a87-9239-8ef93b45909e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm\" (UID: \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" Nov 02 18:07:46.610600 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.610579 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p42n\" (UniqueName: \"kubernetes.io/projected/a29bc8bd-1d33-4a87-9239-8ef93b45909e-kube-api-access-9p42n\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm\" (UID: \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" Nov 02 18:07:46.610821 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.610634 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29bc8bd-1d33-4a87-9239-8ef93b45909e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm\" (UID: \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" Nov 02 18:07:46.610821 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.610727 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a29bc8bd-1d33-4a87-9239-8ef93b45909e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm\" (UID: \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" Nov 02 18:07:46.611411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.611392 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a29bc8bd-1d33-4a87-9239-8ef93b45909e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm\" (UID: \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" Nov 02 18:07:46.611476 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.611451 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29bc8bd-1d33-4a87-9239-8ef93b45909e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm\" (UID: \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" Nov 02 18:07:46.613121 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.613096 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a29bc8bd-1d33-4a87-9239-8ef93b45909e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm\" (UID: \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" Nov 02 18:07:46.620884 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.620857 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p42n\" (UniqueName: \"kubernetes.io/projected/a29bc8bd-1d33-4a87-9239-8ef93b45909e-kube-api-access-9p42n\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm\" (UID: \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" Nov 02 18:07:46.648455 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.648418 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:46.743294 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:46.743255 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" Nov 02 18:07:47.171981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.171891 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6"] Nov 02 18:07:47.178322 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.178293 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" Nov 02 18:07:47.187004 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.182644 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"agc-cred-f5112387de64\"" Nov 02 18:07:47.187004 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.186500 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6"] Nov 02 18:07:47.264710 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:07:47.264681 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29bc8bd_1d33_4a87_9239_8ef93b45909e.slice/crio-3112f671a0212ff420d8c9197e2fe8a82e5bc19713e88db86626f55ce790e814 WatchSource:0}: Error finding container 3112f671a0212ff420d8c9197e2fe8a82e5bc19713e88db86626f55ce790e814: Status 404 returned error can't find the container with id 3112f671a0212ff420d8c9197e2fe8a82e5bc19713e88db86626f55ce790e814 Nov 02 18:07:47.265031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.265008 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm"] Nov 02 18:07:47.279050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.278935 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6csfd\" (UniqueName: \"kubernetes.io/projected/2f06f984-c329-4bfc-9be0-5dfb2d44f597-kube-api-access-6csfd\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6\" (UID: \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" Nov 02 18:07:47.279188 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.279168 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f06f984-c329-4bfc-9be0-5dfb2d44f597-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6\" (UID: \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" Nov 02 18:07:47.279255 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.279212 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f06f984-c329-4bfc-9be0-5dfb2d44f597-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6\" (UID: \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" Nov 02 18:07:47.279468 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.279442 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f06f984-c329-4bfc-9be0-5dfb2d44f597-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6\" (UID: \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" Nov 02 18:07:47.381415 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.381376 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f06f984-c329-4bfc-9be0-5dfb2d44f597-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6\" (UID: \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" Nov 02 18:07:47.381581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.381568 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6csfd\" (UniqueName: \"kubernetes.io/projected/2f06f984-c329-4bfc-9be0-5dfb2d44f597-kube-api-access-6csfd\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6\" (UID: \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" Nov 02 18:07:47.381742 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.381721 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f06f984-c329-4bfc-9be0-5dfb2d44f597-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6\" (UID: \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" Nov 02 18:07:47.381785 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.381770 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f06f984-c329-4bfc-9be0-5dfb2d44f597-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6\" (UID: \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" Nov 02 18:07:47.382146 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.382126 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f06f984-c329-4bfc-9be0-5dfb2d44f597-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6\" (UID: \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" Nov 02 18:07:47.382489 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.382469 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f06f984-c329-4bfc-9be0-5dfb2d44f597-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6\" (UID: \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" Nov 02 18:07:47.384231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.384211 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f06f984-c329-4bfc-9be0-5dfb2d44f597-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6\" (UID: \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" Nov 02 18:07:47.390216 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.390198 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6csfd\" (UniqueName: \"kubernetes.io/projected/2f06f984-c329-4bfc-9be0-5dfb2d44f597-kube-api-access-6csfd\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6\" (UID: \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" Nov 02 18:07:47.496579 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.496544 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" Nov 02 18:07:47.608302 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:07:47.608259 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:07:47.628865 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.628824 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:47.730501 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.730461 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" event={"ID":"a29bc8bd-1d33-4a87-9239-8ef93b45909e","Type":"ContainerStarted","Data":"397c4fb2455bd11a6b67ba7236737893299a9e0cf246a610b35579711a3d5bf9"} Nov 02 18:07:47.730691 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.730523 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" event={"ID":"a29bc8bd-1d33-4a87-9239-8ef93b45909e","Type":"ContainerStarted","Data":"3112f671a0212ff420d8c9197e2fe8a82e5bc19713e88db86626f55ce790e814"} Nov 02 18:07:47.730691 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.730555 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" Nov 02 18:07:47.750486 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.750385 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" podStartSLOduration=1.750370749 podStartE2EDuration="1.750370749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:47.749001009 +0000 UTC m=+2039.537506535" watchObservedRunningTime="2026-11-02 18:07:47.750370749 +0000 UTC m=+2039.538876314" Nov 02 18:07:47.781885 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:47.781856 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6"] Nov 02 18:07:47.782047 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:07:47.782009 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f06f984_c329_4bfc_9be0_5dfb2d44f597.slice/crio-a4f1e685c6f05c8504c4a38d6fb29c7408f1accf8299ca14ddabec1b1a6b2480 WatchSource:0}: Error finding container a4f1e685c6f05c8504c4a38d6fb29c7408f1accf8299ca14ddabec1b1a6b2480: Status 404 returned error can't find the container with id a4f1e685c6f05c8504c4a38d6fb29c7408f1accf8299ca14ddabec1b1a6b2480 Nov 02 18:07:48.362151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:48.361994 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:48.742330 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:48.742284 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" event={"ID":"2f06f984-c329-4bfc-9be0-5dfb2d44f597","Type":"ContainerStarted","Data":"7d57c574e5cf1115490fabf003a1a2376a759e16a2b72f5f89c3dbcc9ab6722b"} Nov 02 18:07:48.742625 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:48.742602 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" Nov 02 18:07:48.742758 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:48.742635 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" event={"ID":"2f06f984-c329-4bfc-9be0-5dfb2d44f597","Type":"ContainerStarted","Data":"a4f1e685c6f05c8504c4a38d6fb29c7408f1accf8299ca14ddabec1b1a6b2480"} Nov 02 18:07:48.760712 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:48.760660 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" podStartSLOduration=1.7606456609999999 podStartE2EDuration="1.760645661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:48.758203842 +0000 UTC m=+2040.546709392" watchObservedRunningTime="2026-11-02 18:07:48.760645661 +0000 UTC m=+2040.549151188" Nov 02 18:07:49.018032 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:49.016243 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:49.708918 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:49.708880 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:50.363169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:50.362577 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:51.056992 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:51.056583 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:51.772974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:51.766408 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:52.368029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:52.363633 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:53.047895 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:53.047858 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:53.719130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:53.719090 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:54.411758 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:54.411447 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:55.164178 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:55.164140 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:58.045280 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.045235 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:58.058230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.058193 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz"] Nov 02 18:07:58.070805 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.070776 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.075326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.075300 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-42d786-openshift-default-dockercfg-dkzz6\"" Nov 02 18:07:58.077199 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.077176 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz"] Nov 02 18:07:58.159774 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.159734 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-workload-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.159945 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.159780 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-istio-envoy\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.159945 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.159923 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-istio-data\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.160041 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.159984 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d725b366-553a-416e-bfc1-90a005778f02-istio-token\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.160124 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.160110 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-workload-certs\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.160170 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.160135 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtptf\" (UniqueName: \"kubernetes.io/projected/d725b366-553a-416e-bfc1-90a005778f02-kube-api-access-vtptf\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.160288 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.160265 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d725b366-553a-416e-bfc1-90a005778f02-istiod-ca-cert\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.160353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.160331 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-credential-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.160390 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.160382 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d725b366-553a-416e-bfc1-90a005778f02-istio-podinfo\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.262147 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.262111 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-istio-data\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.262347 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.262181 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d725b366-553a-416e-bfc1-90a005778f02-istio-token\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.262347 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.262271 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-workload-certs\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.262347 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.262293 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtptf\" (UniqueName: \"kubernetes.io/projected/d725b366-553a-416e-bfc1-90a005778f02-kube-api-access-vtptf\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.262535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.262387 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d725b366-553a-416e-bfc1-90a005778f02-istiod-ca-cert\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.262535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.262425 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-credential-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.262535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.262480 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d725b366-553a-416e-bfc1-90a005778f02-istio-podinfo\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.262673 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.262550 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-istio-data\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.262673 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.262653 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-workload-certs\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.262745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.262691 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-workload-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.262745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.262718 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-istio-envoy\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.262850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.262826 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-credential-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.263033 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.263010 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-workload-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.263188 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.263167 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d725b366-553a-416e-bfc1-90a005778f02-istiod-ca-cert\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.265042 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.265011 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-istio-envoy\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.265210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.265194 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d725b366-553a-416e-bfc1-90a005778f02-istio-podinfo\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.271095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.271063 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtptf\" (UniqueName: \"kubernetes.io/projected/d725b366-553a-416e-bfc1-90a005778f02-kube-api-access-vtptf\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.271399 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.271381 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d725b366-553a-416e-bfc1-90a005778f02-istio-token\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.406767 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.406679 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:07:58.586196 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.586157 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:58.787362 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.785506 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" Nov 02 18:07:58.808622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.788360 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz"] Nov 02 18:07:58.813929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.813322 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:07:58.813929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.813413 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:07:58.813929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:58.813458 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:07:59.026404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:59.023280 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" event={"ID":"d725b366-553a-416e-bfc1-90a005778f02","Type":"ContainerStarted","Data":"316060e70f6c4bdf01c8293c93b49598a69e04d3b83bd5f189e92ada22b99a68"} Nov 02 18:07:59.518980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:59.505296 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:59.806938 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:59.806840 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" Nov 02 18:07:59.864248 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:07:59.864205 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm"] Nov 02 18:08:00.033216 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:00.033178 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" event={"ID":"d725b366-553a-416e-bfc1-90a005778f02","Type":"ContainerStarted","Data":"0b7b8745dc15112e555471c60eb8afb0e618953964aa0d1cc7f4179f12e0f145"} Nov 02 18:08:00.066355 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:00.066255 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" podStartSLOduration=2.066237876 podStartE2EDuration="2.066237876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:00.061046968 +0000 UTC m=+2051.849552497" watchObservedRunningTime="2026-11-02 18:08:00.066237876 +0000 UTC m=+2051.854743403" Nov 02 18:08:00.413030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:00.412917 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:08:00.413201 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:00.413102 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:00.416977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:00.415158 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 02 18:08:00.416977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:00.415214 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" podUID="d725b366-553a-416e-bfc1-90a005778f02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 02 18:08:00.762315 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:00.762274 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:01.413984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:01.413396 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 02 18:08:01.413984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:01.413462 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" podUID="d725b366-553a-416e-bfc1-90a005778f02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 02 18:08:02.051976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:02.051377 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:02.408226 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:02.408123 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 02 18:08:02.408226 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:02.408191 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" podUID="d725b366-553a-416e-bfc1-90a005778f02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 02 18:08:03.407514 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:03.407477 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 02 18:08:03.408004 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:03.407543 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" podUID="d725b366-553a-416e-bfc1-90a005778f02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 02 18:08:03.440451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:03.440411 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:04.408029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:04.407990 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 02 18:08:04.408431 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:04.408047 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" podUID="d725b366-553a-416e-bfc1-90a005778f02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 02 18:08:04.682014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:04.681881 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:04.865008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:04.864964 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" podUID="a29bc8bd-1d33-4a87-9239-8ef93b45909e" containerName="payload-processing" containerID="cri-o://397c4fb2455bd11a6b67ba7236737893299a9e0cf246a610b35579711a3d5bf9" gracePeriod=25 Nov 02 18:08:05.227149 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.227074 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" Nov 02 18:08:05.227864 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.227841 5549 generic.go:356] "Generic (PLEG): container finished" podID="a29bc8bd-1d33-4a87-9239-8ef93b45909e" containerID="397c4fb2455bd11a6b67ba7236737893299a9e0cf246a610b35579711a3d5bf9" exitCode=0 Nov 02 18:08:05.227965 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.227883 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" event={"ID":"a29bc8bd-1d33-4a87-9239-8ef93b45909e","Type":"ContainerDied","Data":"397c4fb2455bd11a6b67ba7236737893299a9e0cf246a610b35579711a3d5bf9"} Nov 02 18:08:05.227965 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.227921 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" event={"ID":"a29bc8bd-1d33-4a87-9239-8ef93b45909e","Type":"ContainerDied","Data":"3112f671a0212ff420d8c9197e2fe8a82e5bc19713e88db86626f55ce790e814"} Nov 02 18:08:05.227965 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.227944 5549 scope.go:122] "RemoveContainer" containerID="397c4fb2455bd11a6b67ba7236737893299a9e0cf246a610b35579711a3d5bf9" Nov 02 18:08:05.239644 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.239623 5549 scope.go:122] "RemoveContainer" containerID="397c4fb2455bd11a6b67ba7236737893299a9e0cf246a610b35579711a3d5bf9" Nov 02 18:08:05.240001 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:08:05.239945 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"397c4fb2455bd11a6b67ba7236737893299a9e0cf246a610b35579711a3d5bf9\": container with ID starting with 397c4fb2455bd11a6b67ba7236737893299a9e0cf246a610b35579711a3d5bf9 not found: ID does not exist" containerID="397c4fb2455bd11a6b67ba7236737893299a9e0cf246a610b35579711a3d5bf9" Nov 02 18:08:05.240090 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.240012 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397c4fb2455bd11a6b67ba7236737893299a9e0cf246a610b35579711a3d5bf9"} err="failed to get container status \"397c4fb2455bd11a6b67ba7236737893299a9e0cf246a610b35579711a3d5bf9\": rpc error: code = NotFound desc = could not find container \"397c4fb2455bd11a6b67ba7236737893299a9e0cf246a610b35579711a3d5bf9\": container with ID starting with 397c4fb2455bd11a6b67ba7236737893299a9e0cf246a610b35579711a3d5bf9 not found: ID does not exist" Nov 02 18:08:05.249273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.248097 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29bc8bd-1d33-4a87-9239-8ef93b45909e-kube-api-access-9p42n\" (UniqueName: \"kubernetes.io/projected/a29bc8bd-1d33-4a87-9239-8ef93b45909e-kube-api-access-9p42n\") pod \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\" (UID: \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\") " Nov 02 18:08:05.249273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.248366 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29bc8bd-1d33-4a87-9239-8ef93b45909e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a29bc8bd-1d33-4a87-9239-8ef93b45909e-routing-overlay-volume\") pod \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\" (UID: \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\") " Nov 02 18:08:05.249273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.248415 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29bc8bd-1d33-4a87-9239-8ef93b45909e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a29bc8bd-1d33-4a87-9239-8ef93b45909e-provider-credentials\") pod \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\" (UID: \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\") " Nov 02 18:08:05.249273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.248517 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29bc8bd-1d33-4a87-9239-8ef93b45909e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29bc8bd-1d33-4a87-9239-8ef93b45909e-plugins-config-volume\") pod \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\" (UID: \"a29bc8bd-1d33-4a87-9239-8ef93b45909e\") " Nov 02 18:08:05.249273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.249048 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29bc8bd-1d33-4a87-9239-8ef93b45909e-routing-overlay-volume" pod "a29bc8bd-1d33-4a87-9239-8ef93b45909e" (UID: "a29bc8bd-1d33-4a87-9239-8ef93b45909e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:08:05.249647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.249337 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a29bc8bd-1d33-4a87-9239-8ef93b45909e-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:05.249923 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.249894 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29bc8bd-1d33-4a87-9239-8ef93b45909e-plugins-config-volume" pod "a29bc8bd-1d33-4a87-9239-8ef93b45909e" (UID: "a29bc8bd-1d33-4a87-9239-8ef93b45909e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:08:05.252206 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.252167 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29bc8bd-1d33-4a87-9239-8ef93b45909e-provider-credentials" pod "a29bc8bd-1d33-4a87-9239-8ef93b45909e" (UID: "a29bc8bd-1d33-4a87-9239-8ef93b45909e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:05.253603 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.253577 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29bc8bd-1d33-4a87-9239-8ef93b45909e-kube-api-access-9p42n" pod "a29bc8bd-1d33-4a87-9239-8ef93b45909e" (UID: "a29bc8bd-1d33-4a87-9239-8ef93b45909e"). InnerVolumeSpecName "kube-api-access-9p42n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:05.350750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.350585 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29bc8bd-1d33-4a87-9239-8ef93b45909e-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:05.350750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.350743 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p42n\" (UniqueName: \"kubernetes.io/projected/a29bc8bd-1d33-4a87-9239-8ef93b45909e-kube-api-access-9p42n\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:05.351180 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.350759 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a29bc8bd-1d33-4a87-9239-8ef93b45909e-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:05.407207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.407168 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 02 18:08:05.407389 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.407269 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" podUID="d725b366-553a-416e-bfc1-90a005778f02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 02 18:08:05.561031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:05.554694 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:06.237566 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:06.237533 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm" Nov 02 18:08:06.269575 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:06.269538 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm"] Nov 02 18:08:06.280096 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:06.280059 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588gr2lm"] Nov 02 18:08:06.324757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:06.324722 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29bc8bd-1d33-4a87-9239-8ef93b45909e" path="/var/lib/kubelet/pods/a29bc8bd-1d33-4a87-9239-8ef93b45909e/volumes" Nov 02 18:08:06.463977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:06.456968 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:06.864511 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:06.864477 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:08:06.865553 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:06.865527 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:08:06.868166 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:06.868145 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:08:06.957054 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:06.957018 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:07.358175 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:07.358075 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:08.126059 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:08.126026 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:08.280981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:08.280923 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:08.445032 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:08.437067 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:08.751113 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:08.744135 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:08.921427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:08.921371 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:09.337943 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:09.337904 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:09.529986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:09.529925 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:10.097058 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.097021 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl"] Nov 02 18:08:10.098079 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.098056 5549 cpu_manager.go:414] "containerMap: removing container" podUID="a29bc8bd-1d33-4a87-9239-8ef93b45909e" containerName="payload-processing" Nov 02 18:08:10.098079 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.098076 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29bc8bd-1d33-4a87-9239-8ef93b45909e" containerName="payload-processing" Nov 02 18:08:10.098342 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.098327 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29bc8bd-1d33-4a87-9239-8ef93b45909e" containerName="payload-processing" Nov 02 18:08:10.104451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.104431 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" Nov 02 18:08:10.108576 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.108549 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:10.111092 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.111065 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl"] Nov 02 18:08:10.111240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.111189 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"kube-root-ca.crt\"" Nov 02 18:08:10.111240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.111189 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"openshift-service-ca.crt\"" Nov 02 18:08:10.111240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.111208 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"agc-cred-5ce29a06f184\"" Nov 02 18:08:10.111240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.111189 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"routing-overlay\"" Nov 02 18:08:10.111445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.111186 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2db162\"" Nov 02 18:08:10.111445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.111337 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"payload-processing-external-model-e2e--96b2e561-dockercfg-jmnf4\"" Nov 02 18:08:10.233977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.225088 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl\" (UID: \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" Nov 02 18:08:10.233977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.225228 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl\" (UID: \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" Nov 02 18:08:10.233977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.225377 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xszs\" (UniqueName: \"kubernetes.io/projected/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-kube-api-access-8xszs\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl\" (UID: \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" Nov 02 18:08:10.233977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.225438 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl\" (UID: \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" Nov 02 18:08:10.341976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.338847 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl\" (UID: \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" Nov 02 18:08:10.341976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.339114 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl\" (UID: \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" Nov 02 18:08:10.341976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.339222 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl\" (UID: \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" Nov 02 18:08:10.341976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.339367 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xszs\" (UniqueName: \"kubernetes.io/projected/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-kube-api-access-8xszs\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl\" (UID: \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" Nov 02 18:08:10.358020 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.357068 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl\" (UID: \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" Nov 02 18:08:10.362986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.362152 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl\" (UID: \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" Nov 02 18:08:10.404984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.404620 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xszs\" (UniqueName: \"kubernetes.io/projected/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-kube-api-access-8xszs\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl\" (UID: \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" Nov 02 18:08:10.436986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.435039 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl\" (UID: \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" Nov 02 18:08:10.436986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.435397 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" Nov 02 18:08:10.853169 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:08:10.853129 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c9626d_d20e_47cb_b78c_88f1aeccdc4a.slice/crio-927b938ed2207e91b5fcd19c93bb22c0ccec207208749ace4c6196bf81d02b6d WatchSource:0}: Error finding container 927b938ed2207e91b5fcd19c93bb22c0ccec207208749ace4c6196bf81d02b6d: Status 404 returned error can't find the container with id 927b938ed2207e91b5fcd19c93bb22c0ccec207208749ace4c6196bf81d02b6d Nov 02 18:08:10.853553 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.853530 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl"] Nov 02 18:08:10.911215 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:10.909223 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:11.363809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:11.363765 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" event={"ID":"34c9626d-d20e-47cb-b78c-88f1aeccdc4a","Type":"ContainerStarted","Data":"7df5703b7b6ffe9595baec022e70aff955967eee00e8ba2a21567a81b2cc2669"} Nov 02 18:08:11.364275 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:11.363829 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" event={"ID":"34c9626d-d20e-47cb-b78c-88f1aeccdc4a","Type":"ContainerStarted","Data":"927b938ed2207e91b5fcd19c93bb22c0ccec207208749ace4c6196bf81d02b6d"} Nov 02 18:08:11.364275 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:11.363897 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" Nov 02 18:08:11.384012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:11.383931 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" podStartSLOduration=1.383915536 podStartE2EDuration="1.383915536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:11.381411845 +0000 UTC m=+2063.169917374" watchObservedRunningTime="2026-11-02 18:08:11.383915536 +0000 UTC m=+2063.172421064" Nov 02 18:08:11.792104 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:11.792061 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:12.535164 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:12.535112 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:13.175136 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:13.175073 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:13.345302 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:13.345267 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:13.963433 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:13.963394 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:14.718868 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:14.718832 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:15.453945 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:15.453858 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:16.226747 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:16.226663 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:17.076561 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:17.076520 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:17.591187 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:08:17.591155 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:08:17.996821 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:17.995024 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:18.845149 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:18.830537 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:19.555830 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:19.555783 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:20.406682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:20.406006 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:21.274978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:21.270716 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:21.931972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:21.931922 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:22.375346 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:22.375311 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" Nov 02 18:08:22.757608 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:22.757565 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:23.423913 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:23.423874 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:24.224279 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:24.224232 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:25.018081 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:25.016025 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:25.048217 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:25.022826 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82"] Nov 02 18:08:25.061435 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:25.061357 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" podUID="40e3fde9-2048-40f8-8a84-5004a426c81f" containerName="istio-proxy" containerID="cri-o://544c86e6cd5ceae36bae075be436ab64478ecd91afef5957834ff94e476000d0" gracePeriod=30 Nov 02 18:08:25.746978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:25.746748 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:26.479361 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:26.479300 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:27.145604 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:27.145567 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:27.851859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:27.851808 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:28.515977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:28.506907 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:29.357415 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:29.357278 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:30.064763 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.064717 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:30.383502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.383471 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 18:08:30.540479 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.540445 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e3fde9-2048-40f8-8a84-5004a426c81f-kube-api-access-s9xqv\" (UniqueName: \"kubernetes.io/projected/40e3fde9-2048-40f8-8a84-5004a426c81f-kube-api-access-s9xqv\") pod \"40e3fde9-2048-40f8-8a84-5004a426c81f\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " Nov 02 18:08:30.540699 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.540530 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-token\" (UniqueName: \"kubernetes.io/projected/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-token\") pod \"40e3fde9-2048-40f8-8a84-5004a426c81f\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " Nov 02 18:08:30.540699 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.540677 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-workload-socket\") pod \"40e3fde9-2048-40f8-8a84-5004a426c81f\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " Nov 02 18:08:30.540833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.540727 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-workload-certs\") pod \"40e3fde9-2048-40f8-8a84-5004a426c81f\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " Nov 02 18:08:30.540833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.540770 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-envoy\") pod \"40e3fde9-2048-40f8-8a84-5004a426c81f\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " Nov 02 18:08:30.540970 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.540840 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-credential-socket\") pod \"40e3fde9-2048-40f8-8a84-5004a426c81f\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " Nov 02 18:08:30.540970 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.540872 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-podinfo\") pod \"40e3fde9-2048-40f8-8a84-5004a426c81f\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " Nov 02 18:08:30.541103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.540968 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-workload-socket" pod "40e3fde9-2048-40f8-8a84-5004a426c81f" (UID: "40e3fde9-2048-40f8-8a84-5004a426c81f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:30.541103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.540976 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e3fde9-2048-40f8-8a84-5004a426c81f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e3fde9-2048-40f8-8a84-5004a426c81f-istiod-ca-cert\") pod \"40e3fde9-2048-40f8-8a84-5004a426c81f\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " Nov 02 18:08:30.541103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.541081 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-data\") pod \"40e3fde9-2048-40f8-8a84-5004a426c81f\" (UID: \"40e3fde9-2048-40f8-8a84-5004a426c81f\") " Nov 02 18:08:30.541279 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.541117 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-workload-certs" pod "40e3fde9-2048-40f8-8a84-5004a426c81f" (UID: "40e3fde9-2048-40f8-8a84-5004a426c81f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:30.541279 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.541173 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-credential-socket" pod "40e3fde9-2048-40f8-8a84-5004a426c81f" (UID: "40e3fde9-2048-40f8-8a84-5004a426c81f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:30.541379 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.541307 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e3fde9-2048-40f8-8a84-5004a426c81f-istiod-ca-cert" pod "40e3fde9-2048-40f8-8a84-5004a426c81f" (UID: "40e3fde9-2048-40f8-8a84-5004a426c81f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:08:30.541503 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.541475 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-data" pod "40e3fde9-2048-40f8-8a84-5004a426c81f" (UID: "40e3fde9-2048-40f8-8a84-5004a426c81f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:30.542110 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.542087 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e3fde9-2048-40f8-8a84-5004a426c81f-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:30.542167 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.542115 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:30.542167 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.542132 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:30.542167 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.542146 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:30.542167 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.542160 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:30.543257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.543221 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-podinfo" pod "40e3fde9-2048-40f8-8a84-5004a426c81f" (UID: "40e3fde9-2048-40f8-8a84-5004a426c81f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:08:30.543378 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.543346 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-envoy" pod "40e3fde9-2048-40f8-8a84-5004a426c81f" (UID: "40e3fde9-2048-40f8-8a84-5004a426c81f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:30.544751 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.544727 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e3fde9-2048-40f8-8a84-5004a426c81f-kube-api-access-s9xqv" pod "40e3fde9-2048-40f8-8a84-5004a426c81f" (UID: "40e3fde9-2048-40f8-8a84-5004a426c81f"). InnerVolumeSpecName "kube-api-access-s9xqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:30.544846 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.544832 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-token" pod "40e3fde9-2048-40f8-8a84-5004a426c81f" (UID: "40e3fde9-2048-40f8-8a84-5004a426c81f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:30.599814 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.599735 5549 generic.go:356] "Generic (PLEG): container finished" podID="40e3fde9-2048-40f8-8a84-5004a426c81f" containerID="544c86e6cd5ceae36bae075be436ab64478ecd91afef5957834ff94e476000d0" exitCode=0 Nov 02 18:08:30.599814 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.599783 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" Nov 02 18:08:30.600014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.599818 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" event={"ID":"40e3fde9-2048-40f8-8a84-5004a426c81f","Type":"ContainerDied","Data":"544c86e6cd5ceae36bae075be436ab64478ecd91afef5957834ff94e476000d0"} Nov 02 18:08:30.600014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.599875 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82" event={"ID":"40e3fde9-2048-40f8-8a84-5004a426c81f","Type":"ContainerDied","Data":"ff90c870fec21a406449a68adf48f050d0b9c05d9dd57878e14e7e08b912aa77"} Nov 02 18:08:30.600014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.599897 5549 scope.go:122] "RemoveContainer" containerID="544c86e6cd5ceae36bae075be436ab64478ecd91afef5957834ff94e476000d0" Nov 02 18:08:30.612521 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.612506 5549 scope.go:122] "RemoveContainer" containerID="544c86e6cd5ceae36bae075be436ab64478ecd91afef5957834ff94e476000d0" Nov 02 18:08:30.612818 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:08:30.612794 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"544c86e6cd5ceae36bae075be436ab64478ecd91afef5957834ff94e476000d0\": container with ID starting with 544c86e6cd5ceae36bae075be436ab64478ecd91afef5957834ff94e476000d0 not found: ID does not exist" containerID="544c86e6cd5ceae36bae075be436ab64478ecd91afef5957834ff94e476000d0" Nov 02 18:08:30.612901 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.612829 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"544c86e6cd5ceae36bae075be436ab64478ecd91afef5957834ff94e476000d0"} err="failed to get container status \"544c86e6cd5ceae36bae075be436ab64478ecd91afef5957834ff94e476000d0\": rpc error: code = NotFound desc = could not find container \"544c86e6cd5ceae36bae075be436ab64478ecd91afef5957834ff94e476000d0\": container with ID starting with 544c86e6cd5ceae36bae075be436ab64478ecd91afef5957834ff94e476000d0 not found: ID does not exist" Nov 02 18:08:30.627769 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.627718 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82"] Nov 02 18:08:30.632749 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.632720 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8qb82"] Nov 02 18:08:30.643990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.643919 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9xqv\" (UniqueName: \"kubernetes.io/projected/40e3fde9-2048-40f8-8a84-5004a426c81f-kube-api-access-s9xqv\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:30.643990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.643966 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:30.643990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.643983 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:30.644184 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.643996 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e3fde9-2048-40f8-8a84-5004a426c81f-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:30.821094 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:30.818794 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:31.556979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:31.554581 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:32.297974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:32.295379 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:32.380492 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:32.380455 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e3fde9-2048-40f8-8a84-5004a426c81f" path="/var/lib/kubelet/pods/40e3fde9-2048-40f8-8a84-5004a426c81f/volumes" Nov 02 18:08:32.951422 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:32.951383 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:33.653826 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:33.653771 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:34.372926 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:34.372889 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:35.117930 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:35.117892 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:35.864524 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:35.864491 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:36.002367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:36.002332 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:08:36.733320 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:36.733288 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:37.747140 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:37.747102 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:38.652572 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:38.652530 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:39.312759 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:39.312719 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:39.371782 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:39.371742 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj"] Nov 02 18:08:39.372016 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:39.371996 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" podUID="852a217c-3306-4428-a34c-0414d04eaba4" containerName="istio-proxy" containerID="cri-o://531a6762dc62bb1c952c8e83f2260673b80da9c7463fea90a69ae20187d13f0f" gracePeriod=30 Nov 02 18:08:39.376454 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:39.376430 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v"] Nov 02 18:08:39.376620 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:39.376602 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" podUID="47851f7c-7870-4a63-bb42-6681e40bf144" containerName="istio-proxy" containerID="cri-o://82e08c1c0d8c5f64f2992ed635b56b2855a7f41e759d919ddf67974580a3ba54" gracePeriod=30 Nov 02 18:08:39.476689 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:39.476640 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:39.624600 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:39.624509 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:40.036302 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.036270 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg"] Nov 02 18:08:40.037282 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.037265 5549 cpu_manager.go:414] "containerMap: removing container" podUID="40e3fde9-2048-40f8-8a84-5004a426c81f" containerName="istio-proxy" Nov 02 18:08:40.037336 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.037284 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e3fde9-2048-40f8-8a84-5004a426c81f" containerName="istio-proxy" Nov 02 18:08:40.037445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.037436 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e3fde9-2048-40f8-8a84-5004a426c81f" containerName="istio-proxy" Nov 02 18:08:40.044217 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.044197 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" Nov 02 18:08:40.050260 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.050230 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"routing-overlay\"" Nov 02 18:08:40.050717 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.050696 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg"] Nov 02 18:08:40.051090 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.051067 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"kube-root-ca.crt\"" Nov 02 18:08:40.051218 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.051160 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"agc-cred-270450402597\"" Nov 02 18:08:40.051218 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.051170 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"e2e-cfl-879b22-provider-cred\"" Nov 02 18:08:40.051345 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.051226 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a8e574\"" Nov 02 18:08:40.051345 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.051252 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"openshift-service-ca.crt\"" Nov 02 18:08:40.051478 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.051348 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"payload-processing-external-model-e2e--a4661749-dockercfg-vq976\"" Nov 02 18:08:40.195807 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.195773 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd876e5-3e6e-45de-9f5b-4148388c34cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg\" (UID: \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" Nov 02 18:08:40.196068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.196021 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s86t8\" (UniqueName: \"kubernetes.io/projected/7dd876e5-3e6e-45de-9f5b-4148388c34cd-kube-api-access-s86t8\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg\" (UID: \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" Nov 02 18:08:40.196220 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.196097 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dd876e5-3e6e-45de-9f5b-4148388c34cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg\" (UID: \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" Nov 02 18:08:40.196220 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.196192 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dd876e5-3e6e-45de-9f5b-4148388c34cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg\" (UID: \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" Nov 02 18:08:40.298238 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.298159 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s86t8\" (UniqueName: \"kubernetes.io/projected/7dd876e5-3e6e-45de-9f5b-4148388c34cd-kube-api-access-s86t8\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg\" (UID: \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" Nov 02 18:08:40.298411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.298244 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dd876e5-3e6e-45de-9f5b-4148388c34cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg\" (UID: \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" Nov 02 18:08:40.298411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.298324 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dd876e5-3e6e-45de-9f5b-4148388c34cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg\" (UID: \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" Nov 02 18:08:40.298411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.298394 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd876e5-3e6e-45de-9f5b-4148388c34cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg\" (UID: \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" Nov 02 18:08:40.299082 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.299054 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dd876e5-3e6e-45de-9f5b-4148388c34cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg\" (UID: \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" Nov 02 18:08:40.299223 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.299166 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd876e5-3e6e-45de-9f5b-4148388c34cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg\" (UID: \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" Nov 02 18:08:40.300876 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.300856 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dd876e5-3e6e-45de-9f5b-4148388c34cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg\" (UID: \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" Nov 02 18:08:40.306569 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.306549 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s86t8\" (UniqueName: \"kubernetes.io/projected/7dd876e5-3e6e-45de-9f5b-4148388c34cd-kube-api-access-s86t8\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg\" (UID: \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" Nov 02 18:08:40.364420 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.364387 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" Nov 02 18:08:40.532774 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.524587 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:40.672564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.672294 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg"] Nov 02 18:08:40.825984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.825904 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" event={"ID":"7dd876e5-3e6e-45de-9f5b-4148388c34cd","Type":"ContainerStarted","Data":"7cacd033612bebf9abbfbc695c3f724c0a3aa4a0a8d25744b8ce521f9b09fcf7"} Nov 02 18:08:40.826168 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.826018 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" event={"ID":"7dd876e5-3e6e-45de-9f5b-4148388c34cd","Type":"ContainerStarted","Data":"842080d5d06e1d265c596404023802bcd9cc788cfdf7f162f9f9b390c21d7268"} Nov 02 18:08:40.826168 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.826056 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" Nov 02 18:08:40.847488 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.847436 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" podStartSLOduration=0.847422183 podStartE2EDuration="847.422183ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:40.84324146 +0000 UTC m=+2092.631746988" watchObservedRunningTime="2026-11-02 18:08:40.847422183 +0000 UTC m=+2092.635927705" Nov 02 18:08:40.864858 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:40.864826 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:41.035974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.026014 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:41.534797 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.534762 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk"] Nov 02 18:08:41.540377 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.540348 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" Nov 02 18:08:41.547112 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.547087 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk"] Nov 02 18:08:41.549256 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.549231 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"kube-root-ca.crt\"" Nov 02 18:08:41.549409 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.549320 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"payload-processing-external-model-plugins-e2e-extmod-a-807c08\"" Nov 02 18:08:41.549487 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.549427 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"openshift-service-ca.crt\"" Nov 02 18:08:41.549487 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.549446 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"agc-cred-5b43ce884121\"" Nov 02 18:08:41.549487 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.549428 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"agc-cred-a01632a9a9c9\"" Nov 02 18:08:41.549487 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.549470 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"routing-overlay\"" Nov 02 18:08:41.549686 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.549515 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"payload-processing-external-model-e2e--d2178d1f-dockercfg-lmw2m\"" Nov 02 18:08:41.573030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.572993 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/725576cc-826f-4e80-8a9a-7e0deef9a840-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk\" (UID: \"725576cc-826f-4e80-8a9a-7e0deef9a840\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" Nov 02 18:08:41.573235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.573144 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725576cc-826f-4e80-8a9a-7e0deef9a840-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk\" (UID: \"725576cc-826f-4e80-8a9a-7e0deef9a840\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" Nov 02 18:08:41.573358 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.573229 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/725576cc-826f-4e80-8a9a-7e0deef9a840-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk\" (UID: \"725576cc-826f-4e80-8a9a-7e0deef9a840\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" Nov 02 18:08:41.573358 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.573275 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g479f\" (UniqueName: \"kubernetes.io/projected/725576cc-826f-4e80-8a9a-7e0deef9a840-kube-api-access-g479f\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk\" (UID: \"725576cc-826f-4e80-8a9a-7e0deef9a840\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" Nov 02 18:08:41.675176 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.675078 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/725576cc-826f-4e80-8a9a-7e0deef9a840-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk\" (UID: \"725576cc-826f-4e80-8a9a-7e0deef9a840\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" Nov 02 18:08:41.675176 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.675167 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725576cc-826f-4e80-8a9a-7e0deef9a840-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk\" (UID: \"725576cc-826f-4e80-8a9a-7e0deef9a840\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" Nov 02 18:08:41.675453 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.675222 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/725576cc-826f-4e80-8a9a-7e0deef9a840-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk\" (UID: \"725576cc-826f-4e80-8a9a-7e0deef9a840\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" Nov 02 18:08:41.675453 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.675258 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g479f\" (UniqueName: \"kubernetes.io/projected/725576cc-826f-4e80-8a9a-7e0deef9a840-kube-api-access-g479f\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk\" (UID: \"725576cc-826f-4e80-8a9a-7e0deef9a840\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" Nov 02 18:08:41.676051 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.676023 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/725576cc-826f-4e80-8a9a-7e0deef9a840-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk\" (UID: \"725576cc-826f-4e80-8a9a-7e0deef9a840\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" Nov 02 18:08:41.676186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.676078 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725576cc-826f-4e80-8a9a-7e0deef9a840-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk\" (UID: \"725576cc-826f-4e80-8a9a-7e0deef9a840\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" Nov 02 18:08:41.677871 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.677842 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/725576cc-826f-4e80-8a9a-7e0deef9a840-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk\" (UID: \"725576cc-826f-4e80-8a9a-7e0deef9a840\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" Nov 02 18:08:41.686262 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.686237 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g479f\" (UniqueName: \"kubernetes.io/projected/725576cc-826f-4e80-8a9a-7e0deef9a840-kube-api-access-g479f\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk\" (UID: \"725576cc-826f-4e80-8a9a-7e0deef9a840\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" Nov 02 18:08:41.861181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:41.861096 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" Nov 02 18:08:42.134036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:42.133617 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:42.222015 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:42.221745 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk"] Nov 02 18:08:42.845224 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:42.845183 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" event={"ID":"725576cc-826f-4e80-8a9a-7e0deef9a840","Type":"ContainerStarted","Data":"27c70cdcf5891e7cc6fb2b8bf8f3b15091c2cc48b05eb2ace87b562ab9835691"} Nov 02 18:08:42.845642 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:42.845245 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" event={"ID":"725576cc-826f-4e80-8a9a-7e0deef9a840","Type":"ContainerStarted","Data":"e6a8e8ed144316918043cd1daa1714c73e337ca401abf6e0852beba2d8dd444b"} Nov 02 18:08:42.845642 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:42.845274 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" Nov 02 18:08:42.876843 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:42.876786 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" podStartSLOduration=1.8767711459999998 podStartE2EDuration="1.876771146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:42.871406961 +0000 UTC m=+2094.659912490" watchObservedRunningTime="2026-11-02 18:08:42.876771146 +0000 UTC m=+2094.665276674" Nov 02 18:08:43.014319 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:43.008576 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:43.846979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:43.844865 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:44.732045 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.730923 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:44.759682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.759650 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 18:08:44.765776 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.765744 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 18:08:44.861318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.861284 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/852a217c-3306-4428-a34c-0414d04eaba4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852a217c-3306-4428-a34c-0414d04eaba4-istio-podinfo\") pod \"852a217c-3306-4428-a34c-0414d04eaba4\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " Nov 02 18:08:44.861777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.861335 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-workload-certs\") pod \"47851f7c-7870-4a63-bb42-6681e40bf144\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " Nov 02 18:08:44.861777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.861385 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-workload-certs\") pod \"852a217c-3306-4428-a34c-0414d04eaba4\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " Nov 02 18:08:44.861777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.861415 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-istio-data\") pod \"47851f7c-7870-4a63-bb42-6681e40bf144\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " Nov 02 18:08:44.861777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.861448 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-istio-data\") pod \"852a217c-3306-4428-a34c-0414d04eaba4\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " Nov 02 18:08:44.861777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.861594 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47851f7c-7870-4a63-bb42-6681e40bf144-kube-api-access-cfr9w\" (UniqueName: \"kubernetes.io/projected/47851f7c-7870-4a63-bb42-6681e40bf144-kube-api-access-cfr9w\") pod \"47851f7c-7870-4a63-bb42-6681e40bf144\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " Nov 02 18:08:44.861777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.861659 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852a217c-3306-4428-a34c-0414d04eaba4-istio-token\" (UniqueName: \"kubernetes.io/projected/852a217c-3306-4428-a34c-0414d04eaba4-istio-token\") pod \"852a217c-3306-4428-a34c-0414d04eaba4\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " Nov 02 18:08:44.861777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.861711 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-workload-certs" pod "47851f7c-7870-4a63-bb42-6681e40bf144" (UID: "47851f7c-7870-4a63-bb42-6681e40bf144"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:44.861777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.861722 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-istio-data" pod "47851f7c-7870-4a63-bb42-6681e40bf144" (UID: "47851f7c-7870-4a63-bb42-6681e40bf144"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:44.861777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.861751 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-workload-certs" pod "852a217c-3306-4428-a34c-0414d04eaba4" (UID: "852a217c-3306-4428-a34c-0414d04eaba4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:44.861777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.861744 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-credential-socket\") pod \"852a217c-3306-4428-a34c-0414d04eaba4\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " Nov 02 18:08:44.862587 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.861787 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-istio-data" pod "852a217c-3306-4428-a34c-0414d04eaba4" (UID: "852a217c-3306-4428-a34c-0414d04eaba4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:44.862587 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.861920 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47851f7c-7870-4a63-bb42-6681e40bf144-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47851f7c-7870-4a63-bb42-6681e40bf144-istiod-ca-cert\") pod \"47851f7c-7870-4a63-bb42-6681e40bf144\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " Nov 02 18:08:44.862587 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.861936 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-credential-socket" pod "852a217c-3306-4428-a34c-0414d04eaba4" (UID: "852a217c-3306-4428-a34c-0414d04eaba4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:44.862587 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.861947 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-istio-envoy\") pod \"47851f7c-7870-4a63-bb42-6681e40bf144\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " Nov 02 18:08:44.862587 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.862280 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47851f7c-7870-4a63-bb42-6681e40bf144-istiod-ca-cert" pod "47851f7c-7870-4a63-bb42-6681e40bf144" (UID: "47851f7c-7870-4a63-bb42-6681e40bf144"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:08:44.862587 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.862151 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-istio-envoy\") pod \"852a217c-3306-4428-a34c-0414d04eaba4\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " Nov 02 18:08:44.862587 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.862501 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-workload-socket\") pod \"852a217c-3306-4428-a34c-0414d04eaba4\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " Nov 02 18:08:44.862587 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.862561 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852a217c-3306-4428-a34c-0414d04eaba4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852a217c-3306-4428-a34c-0414d04eaba4-istiod-ca-cert\") pod \"852a217c-3306-4428-a34c-0414d04eaba4\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " Nov 02 18:08:44.863062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.862600 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-workload-socket\") pod \"47851f7c-7870-4a63-bb42-6681e40bf144\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " Nov 02 18:08:44.863062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.862630 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852a217c-3306-4428-a34c-0414d04eaba4-kube-api-access-cgwl9\" (UniqueName: \"kubernetes.io/projected/852a217c-3306-4428-a34c-0414d04eaba4-kube-api-access-cgwl9\") pod \"852a217c-3306-4428-a34c-0414d04eaba4\" (UID: \"852a217c-3306-4428-a34c-0414d04eaba4\") " Nov 02 18:08:44.863062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.862687 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47851f7c-7870-4a63-bb42-6681e40bf144-istio-token\" (UniqueName: \"kubernetes.io/projected/47851f7c-7870-4a63-bb42-6681e40bf144-istio-token\") pod \"47851f7c-7870-4a63-bb42-6681e40bf144\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " Nov 02 18:08:44.863062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.862734 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-credential-socket\") pod \"47851f7c-7870-4a63-bb42-6681e40bf144\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " Nov 02 18:08:44.863062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.862779 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47851f7c-7870-4a63-bb42-6681e40bf144-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47851f7c-7870-4a63-bb42-6681e40bf144-istio-podinfo\") pod \"47851f7c-7870-4a63-bb42-6681e40bf144\" (UID: \"47851f7c-7870-4a63-bb42-6681e40bf144\") " Nov 02 18:08:44.863062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.862913 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-workload-socket" pod "47851f7c-7870-4a63-bb42-6681e40bf144" (UID: "47851f7c-7870-4a63-bb42-6681e40bf144"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:44.863456 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.863069 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852a217c-3306-4428-a34c-0414d04eaba4-istiod-ca-cert" pod "852a217c-3306-4428-a34c-0414d04eaba4" (UID: "852a217c-3306-4428-a34c-0414d04eaba4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:08:44.863456 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.863308 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-workload-socket" pod "852a217c-3306-4428-a34c-0414d04eaba4" (UID: "852a217c-3306-4428-a34c-0414d04eaba4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:44.863810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.863784 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852a217c-3306-4428-a34c-0414d04eaba4-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:44.863929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.863816 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:44.863929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.863833 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:44.863929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.863846 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:44.863929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.863861 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:44.863929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.863875 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:44.863929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.863890 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:44.863929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.863904 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47851f7c-7870-4a63-bb42-6681e40bf144-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:44.863929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.863919 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:44.865799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.865772 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-istio-envoy" pod "47851f7c-7870-4a63-bb42-6681e40bf144" (UID: "47851f7c-7870-4a63-bb42-6681e40bf144"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:44.865919 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.865853 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47851f7c-7870-4a63-bb42-6681e40bf144-kube-api-access-cfr9w" pod "47851f7c-7870-4a63-bb42-6681e40bf144" (UID: "47851f7c-7870-4a63-bb42-6681e40bf144"). InnerVolumeSpecName "kube-api-access-cfr9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:44.865919 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.865883 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852a217c-3306-4428-a34c-0414d04eaba4-istio-token" pod "852a217c-3306-4428-a34c-0414d04eaba4" (UID: "852a217c-3306-4428-a34c-0414d04eaba4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:44.866131 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.866109 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-credential-socket" pod "47851f7c-7870-4a63-bb42-6681e40bf144" (UID: "47851f7c-7870-4a63-bb42-6681e40bf144"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:44.866215 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.866196 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/852a217c-3306-4428-a34c-0414d04eaba4-istio-podinfo" pod "852a217c-3306-4428-a34c-0414d04eaba4" (UID: "852a217c-3306-4428-a34c-0414d04eaba4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:08:44.866378 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.866353 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-istio-envoy" pod "852a217c-3306-4428-a34c-0414d04eaba4" (UID: "852a217c-3306-4428-a34c-0414d04eaba4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:44.866495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.866393 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47851f7c-7870-4a63-bb42-6681e40bf144-istio-token" pod "47851f7c-7870-4a63-bb42-6681e40bf144" (UID: "47851f7c-7870-4a63-bb42-6681e40bf144"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:44.866495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.866419 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852a217c-3306-4428-a34c-0414d04eaba4-kube-api-access-cgwl9" pod "852a217c-3306-4428-a34c-0414d04eaba4" (UID: "852a217c-3306-4428-a34c-0414d04eaba4"). InnerVolumeSpecName "kube-api-access-cgwl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:44.866495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.866469 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47851f7c-7870-4a63-bb42-6681e40bf144-istio-podinfo" pod "47851f7c-7870-4a63-bb42-6681e40bf144" (UID: "47851f7c-7870-4a63-bb42-6681e40bf144"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:08:44.894641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.894606 5549 generic.go:356] "Generic (PLEG): container finished" podID="47851f7c-7870-4a63-bb42-6681e40bf144" containerID="82e08c1c0d8c5f64f2992ed635b56b2855a7f41e759d919ddf67974580a3ba54" exitCode=0 Nov 02 18:08:44.894800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.894659 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" Nov 02 18:08:44.894800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.894682 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" event={"ID":"47851f7c-7870-4a63-bb42-6681e40bf144","Type":"ContainerDied","Data":"82e08c1c0d8c5f64f2992ed635b56b2855a7f41e759d919ddf67974580a3ba54"} Nov 02 18:08:44.894800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.894737 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v" event={"ID":"47851f7c-7870-4a63-bb42-6681e40bf144","Type":"ContainerDied","Data":"149242ff41fd6f1a0106b5fbe1ce4ba90c64803c5e91c35431b03dadc6ccc5f9"} Nov 02 18:08:44.894800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.894759 5549 scope.go:122] "RemoveContainer" containerID="82e08c1c0d8c5f64f2992ed635b56b2855a7f41e759d919ddf67974580a3ba54" Nov 02 18:08:44.896520 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.896495 5549 generic.go:356] "Generic (PLEG): container finished" podID="852a217c-3306-4428-a34c-0414d04eaba4" containerID="531a6762dc62bb1c952c8e83f2260673b80da9c7463fea90a69ae20187d13f0f" exitCode=0 Nov 02 18:08:44.896661 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.896539 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" Nov 02 18:08:44.896661 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.896552 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" event={"ID":"852a217c-3306-4428-a34c-0414d04eaba4","Type":"ContainerDied","Data":"531a6762dc62bb1c952c8e83f2260673b80da9c7463fea90a69ae20187d13f0f"} Nov 02 18:08:44.896661 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.896594 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj" event={"ID":"852a217c-3306-4428-a34c-0414d04eaba4","Type":"ContainerDied","Data":"84bbdb2c07230d8f585e7923e05ca0c823bf0f6f485ba2267c701b8e39ccebd4"} Nov 02 18:08:44.908482 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.908465 5549 scope.go:122] "RemoveContainer" containerID="82e08c1c0d8c5f64f2992ed635b56b2855a7f41e759d919ddf67974580a3ba54" Nov 02 18:08:44.908761 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:08:44.908742 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e08c1c0d8c5f64f2992ed635b56b2855a7f41e759d919ddf67974580a3ba54\": container with ID starting with 82e08c1c0d8c5f64f2992ed635b56b2855a7f41e759d919ddf67974580a3ba54 not found: ID does not exist" containerID="82e08c1c0d8c5f64f2992ed635b56b2855a7f41e759d919ddf67974580a3ba54" Nov 02 18:08:44.908843 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.908768 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e08c1c0d8c5f64f2992ed635b56b2855a7f41e759d919ddf67974580a3ba54"} err="failed to get container status \"82e08c1c0d8c5f64f2992ed635b56b2855a7f41e759d919ddf67974580a3ba54\": rpc error: code = NotFound desc = could not find container \"82e08c1c0d8c5f64f2992ed635b56b2855a7f41e759d919ddf67974580a3ba54\": container with ID starting with 82e08c1c0d8c5f64f2992ed635b56b2855a7f41e759d919ddf67974580a3ba54 not found: ID does not exist" Nov 02 18:08:44.908843 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.908784 5549 scope.go:122] "RemoveContainer" containerID="531a6762dc62bb1c952c8e83f2260673b80da9c7463fea90a69ae20187d13f0f" Nov 02 18:08:44.939829 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.937251 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v"] Nov 02 18:08:44.943673 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.943637 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-zkp5v"] Nov 02 18:08:44.944764 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.944740 5549 scope.go:122] "RemoveContainer" containerID="531a6762dc62bb1c952c8e83f2260673b80da9c7463fea90a69ae20187d13f0f" Nov 02 18:08:44.945762 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:08:44.945727 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531a6762dc62bb1c952c8e83f2260673b80da9c7463fea90a69ae20187d13f0f\": container with ID starting with 531a6762dc62bb1c952c8e83f2260673b80da9c7463fea90a69ae20187d13f0f not found: ID does not exist" containerID="531a6762dc62bb1c952c8e83f2260673b80da9c7463fea90a69ae20187d13f0f" Nov 02 18:08:44.945894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.945760 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531a6762dc62bb1c952c8e83f2260673b80da9c7463fea90a69ae20187d13f0f"} err="failed to get container status \"531a6762dc62bb1c952c8e83f2260673b80da9c7463fea90a69ae20187d13f0f\": rpc error: code = NotFound desc = could not find container \"531a6762dc62bb1c952c8e83f2260673b80da9c7463fea90a69ae20187d13f0f\": container with ID starting with 531a6762dc62bb1c952c8e83f2260673b80da9c7463fea90a69ae20187d13f0f not found: ID does not exist" Nov 02 18:08:44.950564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.950532 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj"] Nov 02 18:08:44.954686 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.954656 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-cmgcj"] Nov 02 18:08:44.965110 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.965086 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:44.965110 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.965109 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852a217c-3306-4428-a34c-0414d04eaba4-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:44.965238 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.965121 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgwl9\" (UniqueName: \"kubernetes.io/projected/852a217c-3306-4428-a34c-0414d04eaba4-kube-api-access-cgwl9\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:44.965238 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.965130 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47851f7c-7870-4a63-bb42-6681e40bf144-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:44.965238 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.965139 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47851f7c-7870-4a63-bb42-6681e40bf144-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:44.965238 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.965147 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47851f7c-7870-4a63-bb42-6681e40bf144-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:44.965238 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.965156 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852a217c-3306-4428-a34c-0414d04eaba4-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:44.965238 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.965164 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfr9w\" (UniqueName: \"kubernetes.io/projected/47851f7c-7870-4a63-bb42-6681e40bf144-kube-api-access-cfr9w\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:44.965238 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:44.965173 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852a217c-3306-4428-a34c-0414d04eaba4-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:45.530669 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:45.530533 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:46.324979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:46.323389 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:46.386055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:46.386018 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47851f7c-7870-4a63-bb42-6681e40bf144" path="/var/lib/kubelet/pods/47851f7c-7870-4a63-bb42-6681e40bf144/volumes" Nov 02 18:08:46.386655 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:46.386638 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="852a217c-3306-4428-a34c-0414d04eaba4" path="/var/lib/kubelet/pods/852a217c-3306-4428-a34c-0414d04eaba4/volumes" Nov 02 18:08:47.068120 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:47.064318 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:47.590767 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:08:47.590725 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:08:47.857645 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:47.857516 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:48.596041 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:48.596002 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:49.443989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:49.443741 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:49.733784 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:49.733755 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:08:49.949532 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:49.949500 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:08:50.211304 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:50.211271 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:50.981772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:50.981719 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:51.729661 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:51.729607 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:51.836696 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:51.836664 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" Nov 02 18:08:52.454347 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:52.454288 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:53.206121 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:53.206076 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:53.886111 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:53.886078 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" Nov 02 18:08:53.939579 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:53.939537 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:54.632986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:54.624542 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:55.309010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:55.299898 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:56.022763 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:56.022727 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:56.740983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:56.738595 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:57.452161 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:57.452127 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:58.329818 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:58.329459 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:59.005719 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:59.005676 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:59.701770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:08:59.701694 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:00.734730 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:00.734692 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:01.008272 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:01.008193 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:01.627977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:01.624873 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:02.737904 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:02.737847 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:03.541009 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:03.540278 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:04.251470 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:04.251418 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:05.055590 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:05.054594 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:05.697719 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:05.697679 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:06.436597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:06.436558 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:07.266757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:07.266716 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:08.101140 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:08.101109 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:08.809490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:08.809456 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:09.534666 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:09.534623 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:10.241264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:10.241219 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:11.033900 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:11.033860 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:11.687414 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:11.687360 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:12.365898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:12.365846 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:13.245973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:13.241578 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:13.532041 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:13.531926 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:13.695983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:13.695555 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:13.701748 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:13.700171 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9"] Nov 02 18:09:13.701941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:13.701754 5549 cpu_manager.go:414] "containerMap: removing container" podUID="47851f7c-7870-4a63-bb42-6681e40bf144" containerName="istio-proxy" Nov 02 18:09:13.701941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:13.701780 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47851f7c-7870-4a63-bb42-6681e40bf144" containerName="istio-proxy" Nov 02 18:09:13.701941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:13.701866 5549 cpu_manager.go:414] "containerMap: removing container" podUID="852a217c-3306-4428-a34c-0414d04eaba4" containerName="istio-proxy" Nov 02 18:09:13.701941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:13.701875 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852a217c-3306-4428-a34c-0414d04eaba4" containerName="istio-proxy" Nov 02 18:09:13.702230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:13.702125 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="852a217c-3306-4428-a34c-0414d04eaba4" containerName="istio-proxy" Nov 02 18:09:13.702230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:13.702148 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="47851f7c-7870-4a63-bb42-6681e40bf144" containerName="istio-proxy" Nov 02 18:09:13.849262 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:13.849176 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" Nov 02 18:09:13.850561 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:13.850366 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9"] Nov 02 18:09:13.989793 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:13.989754 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8027f9f1-260a-427f-bc4d-c228611e91e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9\" (UID: \"8027f9f1-260a-427f-bc4d-c228611e91e0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" Nov 02 18:09:13.990013 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:13.989875 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8027f9f1-260a-427f-bc4d-c228611e91e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9\" (UID: \"8027f9f1-260a-427f-bc4d-c228611e91e0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" Nov 02 18:09:13.990013 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:13.989990 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8027f9f1-260a-427f-bc4d-c228611e91e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9\" (UID: \"8027f9f1-260a-427f-bc4d-c228611e91e0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" Nov 02 18:09:13.990257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:13.990228 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjhjv\" (UniqueName: \"kubernetes.io/projected/8027f9f1-260a-427f-bc4d-c228611e91e0-kube-api-access-jjhjv\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9\" (UID: \"8027f9f1-260a-427f-bc4d-c228611e91e0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" Nov 02 18:09:14.091650 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:14.091610 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8027f9f1-260a-427f-bc4d-c228611e91e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9\" (UID: \"8027f9f1-260a-427f-bc4d-c228611e91e0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" Nov 02 18:09:14.091867 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:14.091752 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjhjv\" (UniqueName: \"kubernetes.io/projected/8027f9f1-260a-427f-bc4d-c228611e91e0-kube-api-access-jjhjv\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9\" (UID: \"8027f9f1-260a-427f-bc4d-c228611e91e0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" Nov 02 18:09:14.091867 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:14.091825 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8027f9f1-260a-427f-bc4d-c228611e91e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9\" (UID: \"8027f9f1-260a-427f-bc4d-c228611e91e0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" Nov 02 18:09:14.092031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:14.091883 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8027f9f1-260a-427f-bc4d-c228611e91e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9\" (UID: \"8027f9f1-260a-427f-bc4d-c228611e91e0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" Nov 02 18:09:14.092581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:14.092558 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8027f9f1-260a-427f-bc4d-c228611e91e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9\" (UID: \"8027f9f1-260a-427f-bc4d-c228611e91e0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" Nov 02 18:09:14.092666 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:14.092615 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8027f9f1-260a-427f-bc4d-c228611e91e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9\" (UID: \"8027f9f1-260a-427f-bc4d-c228611e91e0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" Nov 02 18:09:14.094189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:14.094164 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8027f9f1-260a-427f-bc4d-c228611e91e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9\" (UID: \"8027f9f1-260a-427f-bc4d-c228611e91e0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" Nov 02 18:09:14.100563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:14.100507 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjhjv\" (UniqueName: \"kubernetes.io/projected/8027f9f1-260a-427f-bc4d-c228611e91e0-kube-api-access-jjhjv\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9\" (UID: \"8027f9f1-260a-427f-bc4d-c228611e91e0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" Nov 02 18:09:14.125259 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:14.125202 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:14.172317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:14.172276 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" Nov 02 18:09:14.345533 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:14.345477 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:14.836064 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:14.833133 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd"] Nov 02 18:09:14.840576 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:14.840532 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd"] Nov 02 18:09:14.840766 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:14.840632 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" Nov 02 18:09:14.996925 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:14.996635 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:15.016697 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.016666 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9"] Nov 02 18:09:15.016912 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:09:15.016875 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8027f9f1_260a_427f_bc4d_c228611e91e0.slice/crio-2e57ac8be69a5b04010a805870864a70ccd5badea249acf3f2360618d69adba8 WatchSource:0}: Error finding container 2e57ac8be69a5b04010a805870864a70ccd5badea249acf3f2360618d69adba8: Status 404 returned error can't find the container with id 2e57ac8be69a5b04010a805870864a70ccd5badea249acf3f2360618d69adba8 Nov 02 18:09:15.038344 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.038304 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v99tp\" (UniqueName: \"kubernetes.io/projected/660b29d1-be18-4a4c-ae47-1e38d196388c-kube-api-access-v99tp\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd\" (UID: \"660b29d1-be18-4a4c-ae47-1e38d196388c\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" Nov 02 18:09:15.038540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.038428 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/660b29d1-be18-4a4c-ae47-1e38d196388c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd\" (UID: \"660b29d1-be18-4a4c-ae47-1e38d196388c\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" Nov 02 18:09:15.038663 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.038643 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660b29d1-be18-4a4c-ae47-1e38d196388c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd\" (UID: \"660b29d1-be18-4a4c-ae47-1e38d196388c\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" Nov 02 18:09:15.038760 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.038742 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/660b29d1-be18-4a4c-ae47-1e38d196388c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd\" (UID: \"660b29d1-be18-4a4c-ae47-1e38d196388c\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" Nov 02 18:09:15.139901 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.139859 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v99tp\" (UniqueName: \"kubernetes.io/projected/660b29d1-be18-4a4c-ae47-1e38d196388c-kube-api-access-v99tp\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd\" (UID: \"660b29d1-be18-4a4c-ae47-1e38d196388c\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" Nov 02 18:09:15.140092 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.139985 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/660b29d1-be18-4a4c-ae47-1e38d196388c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd\" (UID: \"660b29d1-be18-4a4c-ae47-1e38d196388c\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" Nov 02 18:09:15.140143 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.140112 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660b29d1-be18-4a4c-ae47-1e38d196388c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd\" (UID: \"660b29d1-be18-4a4c-ae47-1e38d196388c\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" Nov 02 18:09:15.140196 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.140178 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/660b29d1-be18-4a4c-ae47-1e38d196388c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd\" (UID: \"660b29d1-be18-4a4c-ae47-1e38d196388c\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" Nov 02 18:09:15.140695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.140671 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/660b29d1-be18-4a4c-ae47-1e38d196388c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd\" (UID: \"660b29d1-be18-4a4c-ae47-1e38d196388c\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" Nov 02 18:09:15.140917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.140897 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660b29d1-be18-4a4c-ae47-1e38d196388c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd\" (UID: \"660b29d1-be18-4a4c-ae47-1e38d196388c\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" Nov 02 18:09:15.142670 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.142648 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/660b29d1-be18-4a4c-ae47-1e38d196388c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd\" (UID: \"660b29d1-be18-4a4c-ae47-1e38d196388c\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" Nov 02 18:09:15.148454 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.148432 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v99tp\" (UniqueName: \"kubernetes.io/projected/660b29d1-be18-4a4c-ae47-1e38d196388c-kube-api-access-v99tp\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd\" (UID: \"660b29d1-be18-4a4c-ae47-1e38d196388c\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" Nov 02 18:09:15.172451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.172417 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" Nov 02 18:09:15.403499 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.403459 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" event={"ID":"8027f9f1-260a-427f-bc4d-c228611e91e0","Type":"ContainerStarted","Data":"2b80b76e81e27a242b5d00fdb7d82f6c403170e591310e36e18ef72efe258b17"} Nov 02 18:09:15.403936 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.403520 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" event={"ID":"8027f9f1-260a-427f-bc4d-c228611e91e0","Type":"ContainerStarted","Data":"2e57ac8be69a5b04010a805870864a70ccd5badea249acf3f2360618d69adba8"} Nov 02 18:09:15.403936 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.403552 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" Nov 02 18:09:15.423900 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.423846 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" podStartSLOduration=2.423831931 podStartE2EDuration="2.423831931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:09:15.420120802 +0000 UTC m=+2127.208626358" watchObservedRunningTime="2026-11-02 18:09:15.423831931 +0000 UTC m=+2127.212337461" Nov 02 18:09:15.463013 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:09:15.462981 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660b29d1_be18_4a4c_ae47_1e38d196388c.slice/crio-841b8c9ace2dbf9ebfcd1f3dcf3d53123dc4d9b648346641726014a36c1ec1a1 WatchSource:0}: Error finding container 841b8c9ace2dbf9ebfcd1f3dcf3d53123dc4d9b648346641726014a36c1ec1a1: Status 404 returned error can't find the container with id 841b8c9ace2dbf9ebfcd1f3dcf3d53123dc4d9b648346641726014a36c1ec1a1 Nov 02 18:09:15.463206 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.463176 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd"] Nov 02 18:09:15.620389 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:15.617810 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:16.280280 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:16.272857 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:16.413738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:16.413698 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" event={"ID":"660b29d1-be18-4a4c-ae47-1e38d196388c","Type":"ContainerStarted","Data":"e703ddd1798a5e1af2993994c1858b6b5ac00925cec90c4ec32df66e9efdf445"} Nov 02 18:09:16.414212 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:16.413764 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" event={"ID":"660b29d1-be18-4a4c-ae47-1e38d196388c","Type":"ContainerStarted","Data":"841b8c9ace2dbf9ebfcd1f3dcf3d53123dc4d9b648346641726014a36c1ec1a1"} Nov 02 18:09:16.414212 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:16.413795 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" Nov 02 18:09:16.438090 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:16.438032 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" podStartSLOduration=2.438016906 podStartE2EDuration="2.438016906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:09:16.433627399 +0000 UTC m=+2128.222132927" watchObservedRunningTime="2026-11-02 18:09:16.438016906 +0000 UTC m=+2128.226522434" Nov 02 18:09:17.013178 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:17.013009 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:17.590370 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:17.590333 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:09:17.798095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:17.795064 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:18.519493 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:18.517545 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:19.286996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:19.286746 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:19.957137 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:19.957087 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:20.623988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:20.619995 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:21.354174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:21.354117 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:22.037977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:22.022822 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:22.686518 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:22.686408 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:23.436789 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:23.434324 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:24.125034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:24.124994 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:24.842013 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:24.834734 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:25.628615 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:25.628575 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:25.795038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:25.794988 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd"] Nov 02 18:09:25.804994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:25.804947 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" Nov 02 18:09:25.807197 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:25.807167 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg"] Nov 02 18:09:25.900772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:25.900666 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:26.108517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:26.108480 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:26.174663 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:26.174577 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9"] Nov 02 18:09:26.177781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:26.177751 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" Nov 02 18:09:26.179564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:26.179534 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk"] Nov 02 18:09:26.253002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:26.252938 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:26.445940 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:26.445724 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:26.891763 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:26.891675 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:27.031936 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.031901 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45"] Nov 02 18:09:27.038210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.038152 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" Nov 02 18:09:27.043546 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.043518 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"payload-processing-external-model-e2e--a4661749-dockercfg-7nzv9\"" Nov 02 18:09:27.043665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.043597 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"agc-cred-c7aa0fd5ea16\"" Nov 02 18:09:27.046139 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.046114 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45"] Nov 02 18:09:27.083545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.083511 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:27.204765 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.204722 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec036f2-6fac-41b3-9152-bd79fc6a2195-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45\" (UID: \"4ec036f2-6fac-41b3-9152-bd79fc6a2195\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" Nov 02 18:09:27.205739 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.205705 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk4x6\" (UniqueName: \"kubernetes.io/projected/4ec036f2-6fac-41b3-9152-bd79fc6a2195-kube-api-access-kk4x6\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45\" (UID: \"4ec036f2-6fac-41b3-9152-bd79fc6a2195\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" Nov 02 18:09:27.206406 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.206381 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ec036f2-6fac-41b3-9152-bd79fc6a2195-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45\" (UID: \"4ec036f2-6fac-41b3-9152-bd79fc6a2195\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" Nov 02 18:09:27.206715 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.206691 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ec036f2-6fac-41b3-9152-bd79fc6a2195-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45\" (UID: \"4ec036f2-6fac-41b3-9152-bd79fc6a2195\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" Nov 02 18:09:27.311247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.311119 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ec036f2-6fac-41b3-9152-bd79fc6a2195-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45\" (UID: \"4ec036f2-6fac-41b3-9152-bd79fc6a2195\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" Nov 02 18:09:27.311620 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.311599 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ec036f2-6fac-41b3-9152-bd79fc6a2195-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45\" (UID: \"4ec036f2-6fac-41b3-9152-bd79fc6a2195\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" Nov 02 18:09:27.311985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.311945 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec036f2-6fac-41b3-9152-bd79fc6a2195-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45\" (UID: \"4ec036f2-6fac-41b3-9152-bd79fc6a2195\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" Nov 02 18:09:27.312297 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.312276 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk4x6\" (UniqueName: \"kubernetes.io/projected/4ec036f2-6fac-41b3-9152-bd79fc6a2195-kube-api-access-kk4x6\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45\" (UID: \"4ec036f2-6fac-41b3-9152-bd79fc6a2195\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" Nov 02 18:09:27.319973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.313555 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ec036f2-6fac-41b3-9152-bd79fc6a2195-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45\" (UID: \"4ec036f2-6fac-41b3-9152-bd79fc6a2195\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" Nov 02 18:09:27.320259 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:27.320238 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a8e574/agc-cred-c7aa0fd5ea16: secret "agc-cred-c7aa0fd5ea16" not found Nov 02 18:09:27.320372 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:27.320361 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45: secret "agc-cred-c7aa0fd5ea16" not found Nov 02 18:09:27.320524 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:27.320513 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ec036f2-6fac-41b3-9152-bd79fc6a2195-provider-credentials podName:4ec036f2-6fac-41b3-9152-bd79fc6a2195 nodeName:}" failed. No retries permitted until 2026-11-02 18:09:27.820489163 +0000 UTC m=+2139.608994693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4ec036f2-6fac-41b3-9152-bd79fc6a2195-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" (UID: "4ec036f2-6fac-41b3-9152-bd79fc6a2195") : secret "agc-cred-c7aa0fd5ea16" not found Nov 02 18:09:27.321735 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.321710 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec036f2-6fac-41b3-9152-bd79fc6a2195-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45\" (UID: \"4ec036f2-6fac-41b3-9152-bd79fc6a2195\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" Nov 02 18:09:27.380858 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.380809 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk4x6\" (UniqueName: \"kubernetes.io/projected/4ec036f2-6fac-41b3-9152-bd79fc6a2195-kube-api-access-kk4x6\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45\" (UID: \"4ec036f2-6fac-41b3-9152-bd79fc6a2195\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" Nov 02 18:09:27.537911 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.537832 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45"] Nov 02 18:09:27.538384 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:27.538359 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" podUID="4ec036f2-6fac-41b3-9152-bd79fc6a2195" Nov 02 18:09:27.581675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.581483 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:27.586154 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.586130 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" Nov 02 18:09:27.597147 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.597119 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" Nov 02 18:09:27.737904 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.730663 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ec036f2-6fac-41b3-9152-bd79fc6a2195-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec036f2-6fac-41b3-9152-bd79fc6a2195-plugins-config-volume\") pod \"4ec036f2-6fac-41b3-9152-bd79fc6a2195\" (UID: \"4ec036f2-6fac-41b3-9152-bd79fc6a2195\") " Nov 02 18:09:27.737904 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.730911 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ec036f2-6fac-41b3-9152-bd79fc6a2195-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ec036f2-6fac-41b3-9152-bd79fc6a2195-routing-overlay-volume\") pod \"4ec036f2-6fac-41b3-9152-bd79fc6a2195\" (UID: \"4ec036f2-6fac-41b3-9152-bd79fc6a2195\") " Nov 02 18:09:27.737904 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.731165 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec036f2-6fac-41b3-9152-bd79fc6a2195-kube-api-access-kk4x6\" (UniqueName: \"kubernetes.io/projected/4ec036f2-6fac-41b3-9152-bd79fc6a2195-kube-api-access-kk4x6\") pod \"4ec036f2-6fac-41b3-9152-bd79fc6a2195\" (UID: \"4ec036f2-6fac-41b3-9152-bd79fc6a2195\") " Nov 02 18:09:27.746552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.746515 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec036f2-6fac-41b3-9152-bd79fc6a2195-plugins-config-volume" pod "4ec036f2-6fac-41b3-9152-bd79fc6a2195" (UID: "4ec036f2-6fac-41b3-9152-bd79fc6a2195"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:27.746830 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.746799 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:27.747778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.747207 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec036f2-6fac-41b3-9152-bd79fc6a2195-routing-overlay-volume" pod "4ec036f2-6fac-41b3-9152-bd79fc6a2195" (UID: "4ec036f2-6fac-41b3-9152-bd79fc6a2195"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:27.792075 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.785043 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec036f2-6fac-41b3-9152-bd79fc6a2195-kube-api-access-kk4x6" pod "4ec036f2-6fac-41b3-9152-bd79fc6a2195" (UID: "4ec036f2-6fac-41b3-9152-bd79fc6a2195"). InnerVolumeSpecName "kube-api-access-kk4x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:27.837978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.834454 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ec036f2-6fac-41b3-9152-bd79fc6a2195-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45\" (UID: \"4ec036f2-6fac-41b3-9152-bd79fc6a2195\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" Nov 02 18:09:27.837978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.834666 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ec036f2-6fac-41b3-9152-bd79fc6a2195-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:27.837978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.834686 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk4x6\" (UniqueName: \"kubernetes.io/projected/4ec036f2-6fac-41b3-9152-bd79fc6a2195-kube-api-access-kk4x6\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:27.837978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:27.834702 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec036f2-6fac-41b3-9152-bd79fc6a2195-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:27.837978 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:27.834813 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a8e574/agc-cred-c7aa0fd5ea16: secret "agc-cred-c7aa0fd5ea16" not found Nov 02 18:09:27.837978 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:27.834828 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45: secret "agc-cred-c7aa0fd5ea16" not found Nov 02 18:09:27.837978 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:27.834906 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ec036f2-6fac-41b3-9152-bd79fc6a2195-provider-credentials podName:4ec036f2-6fac-41b3-9152-bd79fc6a2195 nodeName:}" failed. No retries permitted until 2026-11-02 18:09:28.834878977 +0000 UTC m=+2140.623384486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4ec036f2-6fac-41b3-9152-bd79fc6a2195-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" (UID: "4ec036f2-6fac-41b3-9152-bd79fc6a2195") : secret "agc-cred-c7aa0fd5ea16" not found Nov 02 18:09:28.593717 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:28.593639 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45" Nov 02 18:09:28.655980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:28.641011 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:28.687502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:28.687444 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45"] Nov 02 18:09:28.695384 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:28.695354 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd9nw45"] Nov 02 18:09:28.865308 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:28.865218 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ec036f2-6fac-41b3-9152-bd79fc6a2195-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:29.328207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:29.328170 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:30.018973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:30.007539 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:30.323364 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:30.323276 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ec036f2-6fac-41b3-9152-bd79fc6a2195" path="/var/lib/kubelet/pods/4ec036f2-6fac-41b3-9152-bd79fc6a2195/volumes" Nov 02 18:09:30.756992 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:30.754047 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:30.805894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:30.805849 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" podUID="660b29d1-be18-4a4c-ae47-1e38d196388c" containerName="payload-processing" containerID="cri-o://e703ddd1798a5e1af2993994c1858b6b5ac00925cec90c4ec32df66e9efdf445" gracePeriod=25 Nov 02 18:09:30.807523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:30.807492 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" podUID="7dd876e5-3e6e-45de-9f5b-4148388c34cd" containerName="payload-processing" containerID="cri-o://7cacd033612bebf9abbfbc695c3f724c0a3aa4a0a8d25744b8ce521f9b09fcf7" gracePeriod=25 Nov 02 18:09:31.175001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.174939 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" podUID="8027f9f1-260a-427f-bc4d-c228611e91e0" containerName="payload-processing" containerID="cri-o://2b80b76e81e27a242b5d00fdb7d82f6c403170e591310e36e18ef72efe258b17" gracePeriod=25 Nov 02 18:09:31.180391 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.180243 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" Nov 02 18:09:31.180538 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.180413 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" podUID="725576cc-826f-4e80-8a9a-7e0deef9a840" containerName="payload-processing" containerID="cri-o://27c70cdcf5891e7cc6fb2b8bf8f3b15091c2cc48b05eb2ace87b562ab9835691" gracePeriod=25 Nov 02 18:09:31.185447 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.185416 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" Nov 02 18:09:31.250168 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.250140 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:31.265200 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.265128 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660b29d1-be18-4a4c-ae47-1e38d196388c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660b29d1-be18-4a4c-ae47-1e38d196388c-plugins-config-volume\") pod \"660b29d1-be18-4a4c-ae47-1e38d196388c\" (UID: \"660b29d1-be18-4a4c-ae47-1e38d196388c\") " Nov 02 18:09:31.265346 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.265278 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660b29d1-be18-4a4c-ae47-1e38d196388c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/660b29d1-be18-4a4c-ae47-1e38d196388c-provider-credentials\") pod \"660b29d1-be18-4a4c-ae47-1e38d196388c\" (UID: \"660b29d1-be18-4a4c-ae47-1e38d196388c\") " Nov 02 18:09:31.265346 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.265327 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dd876e5-3e6e-45de-9f5b-4148388c34cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd876e5-3e6e-45de-9f5b-4148388c34cd-plugins-config-volume\") pod \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\" (UID: \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\") " Nov 02 18:09:31.265430 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.265398 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660b29d1-be18-4a4c-ae47-1e38d196388c-kube-api-access-v99tp\" (UniqueName: \"kubernetes.io/projected/660b29d1-be18-4a4c-ae47-1e38d196388c-kube-api-access-v99tp\") pod \"660b29d1-be18-4a4c-ae47-1e38d196388c\" (UID: \"660b29d1-be18-4a4c-ae47-1e38d196388c\") " Nov 02 18:09:31.265489 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.265474 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660b29d1-be18-4a4c-ae47-1e38d196388c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/660b29d1-be18-4a4c-ae47-1e38d196388c-routing-overlay-volume\") pod \"660b29d1-be18-4a4c-ae47-1e38d196388c\" (UID: \"660b29d1-be18-4a4c-ae47-1e38d196388c\") " Nov 02 18:09:31.265631 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.265612 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd876e5-3e6e-45de-9f5b-4148388c34cd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dd876e5-3e6e-45de-9f5b-4148388c34cd-provider-credentials\") pod \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\" (UID: \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\") " Nov 02 18:09:31.265730 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.265697 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd876e5-3e6e-45de-9f5b-4148388c34cd-kube-api-access-s86t8\" (UniqueName: \"kubernetes.io/projected/7dd876e5-3e6e-45de-9f5b-4148388c34cd-kube-api-access-s86t8\") pod \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\" (UID: \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\") " Nov 02 18:09:31.265795 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.265755 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dd876e5-3e6e-45de-9f5b-4148388c34cd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dd876e5-3e6e-45de-9f5b-4148388c34cd-routing-overlay-volume\") pod \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\" (UID: \"7dd876e5-3e6e-45de-9f5b-4148388c34cd\") " Nov 02 18:09:31.266199 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.265847 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660b29d1-be18-4a4c-ae47-1e38d196388c-plugins-config-volume" pod "660b29d1-be18-4a4c-ae47-1e38d196388c" (UID: "660b29d1-be18-4a4c-ae47-1e38d196388c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:31.266199 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.265867 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd876e5-3e6e-45de-9f5b-4148388c34cd-plugins-config-volume" pod "7dd876e5-3e6e-45de-9f5b-4148388c34cd" (UID: "7dd876e5-3e6e-45de-9f5b-4148388c34cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:31.266372 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.266312 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd876e5-3e6e-45de-9f5b-4148388c34cd-routing-overlay-volume" pod "7dd876e5-3e6e-45de-9f5b-4148388c34cd" (UID: "7dd876e5-3e6e-45de-9f5b-4148388c34cd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:31.266486 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.266451 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660b29d1-be18-4a4c-ae47-1e38d196388c-routing-overlay-volume" pod "660b29d1-be18-4a4c-ae47-1e38d196388c" (UID: "660b29d1-be18-4a4c-ae47-1e38d196388c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:31.266791 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.266769 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/660b29d1-be18-4a4c-ae47-1e38d196388c-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:31.266889 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.266796 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dd876e5-3e6e-45de-9f5b-4148388c34cd-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:31.266889 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.266813 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660b29d1-be18-4a4c-ae47-1e38d196388c-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:31.266889 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.266826 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd876e5-3e6e-45de-9f5b-4148388c34cd-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:31.267994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.267966 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660b29d1-be18-4a4c-ae47-1e38d196388c-kube-api-access-v99tp" pod "660b29d1-be18-4a4c-ae47-1e38d196388c" (UID: "660b29d1-be18-4a4c-ae47-1e38d196388c"). InnerVolumeSpecName "kube-api-access-v99tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:31.268199 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.268179 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660b29d1-be18-4a4c-ae47-1e38d196388c-provider-credentials" pod "660b29d1-be18-4a4c-ae47-1e38d196388c" (UID: "660b29d1-be18-4a4c-ae47-1e38d196388c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:31.268312 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.268295 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd876e5-3e6e-45de-9f5b-4148388c34cd-provider-credentials" pod "7dd876e5-3e6e-45de-9f5b-4148388c34cd" (UID: "7dd876e5-3e6e-45de-9f5b-4148388c34cd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:31.268375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.268351 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd876e5-3e6e-45de-9f5b-4148388c34cd-kube-api-access-s86t8" pod "7dd876e5-3e6e-45de-9f5b-4148388c34cd" (UID: "7dd876e5-3e6e-45de-9f5b-4148388c34cd"). InnerVolumeSpecName "kube-api-access-s86t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:31.371517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.368671 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v99tp\" (UniqueName: \"kubernetes.io/projected/660b29d1-be18-4a4c-ae47-1e38d196388c-kube-api-access-v99tp\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:31.371517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.368702 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dd876e5-3e6e-45de-9f5b-4148388c34cd-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:31.371517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.368713 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s86t8\" (UniqueName: \"kubernetes.io/projected/7dd876e5-3e6e-45de-9f5b-4148388c34cd-kube-api-access-s86t8\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:31.371517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.368723 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/660b29d1-be18-4a4c-ae47-1e38d196388c-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:31.510053 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.506791 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" Nov 02 18:09:31.573100 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.571695 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:31.594038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.591160 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8027f9f1-260a-427f-bc4d-c228611e91e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8027f9f1-260a-427f-bc4d-c228611e91e0-plugins-config-volume\") pod \"8027f9f1-260a-427f-bc4d-c228611e91e0\" (UID: \"8027f9f1-260a-427f-bc4d-c228611e91e0\") " Nov 02 18:09:31.594038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.591214 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8027f9f1-260a-427f-bc4d-c228611e91e0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8027f9f1-260a-427f-bc4d-c228611e91e0-provider-credentials\") pod \"8027f9f1-260a-427f-bc4d-c228611e91e0\" (UID: \"8027f9f1-260a-427f-bc4d-c228611e91e0\") " Nov 02 18:09:31.594038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.591436 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8027f9f1-260a-427f-bc4d-c228611e91e0-kube-api-access-jjhjv\" (UniqueName: \"kubernetes.io/projected/8027f9f1-260a-427f-bc4d-c228611e91e0-kube-api-access-jjhjv\") pod \"8027f9f1-260a-427f-bc4d-c228611e91e0\" (UID: \"8027f9f1-260a-427f-bc4d-c228611e91e0\") " Nov 02 18:09:31.594038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.591600 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8027f9f1-260a-427f-bc4d-c228611e91e0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8027f9f1-260a-427f-bc4d-c228611e91e0-routing-overlay-volume\") pod \"8027f9f1-260a-427f-bc4d-c228611e91e0\" (UID: \"8027f9f1-260a-427f-bc4d-c228611e91e0\") " Nov 02 18:09:31.607007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.600220 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8027f9f1-260a-427f-bc4d-c228611e91e0-plugins-config-volume" pod "8027f9f1-260a-427f-bc4d-c228611e91e0" (UID: "8027f9f1-260a-427f-bc4d-c228611e91e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:31.616354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.616223 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8027f9f1-260a-427f-bc4d-c228611e91e0-routing-overlay-volume" pod "8027f9f1-260a-427f-bc4d-c228611e91e0" (UID: "8027f9f1-260a-427f-bc4d-c228611e91e0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:31.622270 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.622212 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8027f9f1-260a-427f-bc4d-c228611e91e0-kube-api-access-jjhjv" pod "8027f9f1-260a-427f-bc4d-c228611e91e0" (UID: "8027f9f1-260a-427f-bc4d-c228611e91e0"). InnerVolumeSpecName "kube-api-access-jjhjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:31.623766 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.623738 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8027f9f1-260a-427f-bc4d-c228611e91e0-provider-credentials" pod "8027f9f1-260a-427f-bc4d-c228611e91e0" (UID: "8027f9f1-260a-427f-bc4d-c228611e91e0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:31.628994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.628779 5549 generic.go:356] "Generic (PLEG): container finished" podID="7dd876e5-3e6e-45de-9f5b-4148388c34cd" containerID="7cacd033612bebf9abbfbc695c3f724c0a3aa4a0a8d25744b8ce521f9b09fcf7" exitCode=0 Nov 02 18:09:31.628994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.628908 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" event={"ID":"7dd876e5-3e6e-45de-9f5b-4148388c34cd","Type":"ContainerDied","Data":"7cacd033612bebf9abbfbc695c3f724c0a3aa4a0a8d25744b8ce521f9b09fcf7"} Nov 02 18:09:31.628994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.628945 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" event={"ID":"7dd876e5-3e6e-45de-9f5b-4148388c34cd","Type":"ContainerDied","Data":"842080d5d06e1d265c596404023802bcd9cc788cfdf7f162f9f9b390c21d7268"} Nov 02 18:09:31.629239 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.629012 5549 scope.go:122] "RemoveContainer" containerID="7cacd033612bebf9abbfbc695c3f724c0a3aa4a0a8d25744b8ce521f9b09fcf7" Nov 02 18:09:31.629239 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.629183 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg" Nov 02 18:09:31.634582 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.632451 5549 generic.go:356] "Generic (PLEG): container finished" podID="660b29d1-be18-4a4c-ae47-1e38d196388c" containerID="e703ddd1798a5e1af2993994c1858b6b5ac00925cec90c4ec32df66e9efdf445" exitCode=0 Nov 02 18:09:31.634582 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.632528 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" event={"ID":"660b29d1-be18-4a4c-ae47-1e38d196388c","Type":"ContainerDied","Data":"e703ddd1798a5e1af2993994c1858b6b5ac00925cec90c4ec32df66e9efdf445"} Nov 02 18:09:31.634582 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.632550 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" Nov 02 18:09:31.634582 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.632581 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd" event={"ID":"660b29d1-be18-4a4c-ae47-1e38d196388c","Type":"ContainerDied","Data":"841b8c9ace2dbf9ebfcd1f3dcf3d53123dc4d9b648346641726014a36c1ec1a1"} Nov 02 18:09:31.636278 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.636251 5549 generic.go:356] "Generic (PLEG): container finished" podID="8027f9f1-260a-427f-bc4d-c228611e91e0" containerID="2b80b76e81e27a242b5d00fdb7d82f6c403170e591310e36e18ef72efe258b17" exitCode=0 Nov 02 18:09:31.636386 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.636357 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" event={"ID":"8027f9f1-260a-427f-bc4d-c228611e91e0","Type":"ContainerDied","Data":"2b80b76e81e27a242b5d00fdb7d82f6c403170e591310e36e18ef72efe258b17"} Nov 02 18:09:31.636450 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.636378 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" Nov 02 18:09:31.636538 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.636398 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9" event={"ID":"8027f9f1-260a-427f-bc4d-c228611e91e0","Type":"ContainerDied","Data":"2e57ac8be69a5b04010a805870864a70ccd5badea249acf3f2360618d69adba8"} Nov 02 18:09:31.639999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.639976 5549 generic.go:356] "Generic (PLEG): container finished" podID="725576cc-826f-4e80-8a9a-7e0deef9a840" containerID="27c70cdcf5891e7cc6fb2b8bf8f3b15091c2cc48b05eb2ace87b562ab9835691" exitCode=0 Nov 02 18:09:31.640164 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.640040 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" event={"ID":"725576cc-826f-4e80-8a9a-7e0deef9a840","Type":"ContainerDied","Data":"27c70cdcf5891e7cc6fb2b8bf8f3b15091c2cc48b05eb2ace87b562ab9835691"} Nov 02 18:09:31.641738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.641706 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" Nov 02 18:09:31.645038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.645017 5549 scope.go:122] "RemoveContainer" containerID="7cacd033612bebf9abbfbc695c3f724c0a3aa4a0a8d25744b8ce521f9b09fcf7" Nov 02 18:09:31.645538 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:31.645517 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cacd033612bebf9abbfbc695c3f724c0a3aa4a0a8d25744b8ce521f9b09fcf7\": container with ID starting with 7cacd033612bebf9abbfbc695c3f724c0a3aa4a0a8d25744b8ce521f9b09fcf7 not found: ID does not exist" containerID="7cacd033612bebf9abbfbc695c3f724c0a3aa4a0a8d25744b8ce521f9b09fcf7" Nov 02 18:09:31.645634 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.645616 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cacd033612bebf9abbfbc695c3f724c0a3aa4a0a8d25744b8ce521f9b09fcf7"} err="failed to get container status \"7cacd033612bebf9abbfbc695c3f724c0a3aa4a0a8d25744b8ce521f9b09fcf7\": rpc error: code = NotFound desc = could not find container \"7cacd033612bebf9abbfbc695c3f724c0a3aa4a0a8d25744b8ce521f9b09fcf7\": container with ID starting with 7cacd033612bebf9abbfbc695c3f724c0a3aa4a0a8d25744b8ce521f9b09fcf7 not found: ID does not exist" Nov 02 18:09:31.645704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.645696 5549 scope.go:122] "RemoveContainer" containerID="e703ddd1798a5e1af2993994c1858b6b5ac00925cec90c4ec32df66e9efdf445" Nov 02 18:09:31.659620 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.659600 5549 scope.go:122] "RemoveContainer" containerID="e703ddd1798a5e1af2993994c1858b6b5ac00925cec90c4ec32df66e9efdf445" Nov 02 18:09:31.659916 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:31.659895 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e703ddd1798a5e1af2993994c1858b6b5ac00925cec90c4ec32df66e9efdf445\": container with ID starting with e703ddd1798a5e1af2993994c1858b6b5ac00925cec90c4ec32df66e9efdf445 not found: ID does not exist" containerID="e703ddd1798a5e1af2993994c1858b6b5ac00925cec90c4ec32df66e9efdf445" Nov 02 18:09:31.660055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.659928 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e703ddd1798a5e1af2993994c1858b6b5ac00925cec90c4ec32df66e9efdf445"} err="failed to get container status \"e703ddd1798a5e1af2993994c1858b6b5ac00925cec90c4ec32df66e9efdf445\": rpc error: code = NotFound desc = could not find container \"e703ddd1798a5e1af2993994c1858b6b5ac00925cec90c4ec32df66e9efdf445\": container with ID starting with e703ddd1798a5e1af2993994c1858b6b5ac00925cec90c4ec32df66e9efdf445 not found: ID does not exist" Nov 02 18:09:31.660055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.659987 5549 scope.go:122] "RemoveContainer" containerID="2b80b76e81e27a242b5d00fdb7d82f6c403170e591310e36e18ef72efe258b17" Nov 02 18:09:31.683011 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.682936 5549 scope.go:122] "RemoveContainer" containerID="2b80b76e81e27a242b5d00fdb7d82f6c403170e591310e36e18ef72efe258b17" Nov 02 18:09:31.683372 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:31.683348 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b80b76e81e27a242b5d00fdb7d82f6c403170e591310e36e18ef72efe258b17\": container with ID starting with 2b80b76e81e27a242b5d00fdb7d82f6c403170e591310e36e18ef72efe258b17 not found: ID does not exist" containerID="2b80b76e81e27a242b5d00fdb7d82f6c403170e591310e36e18ef72efe258b17" Nov 02 18:09:31.683496 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.683384 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b80b76e81e27a242b5d00fdb7d82f6c403170e591310e36e18ef72efe258b17"} err="failed to get container status \"2b80b76e81e27a242b5d00fdb7d82f6c403170e591310e36e18ef72efe258b17\": rpc error: code = NotFound desc = could not find container \"2b80b76e81e27a242b5d00fdb7d82f6c403170e591310e36e18ef72efe258b17\": container with ID starting with 2b80b76e81e27a242b5d00fdb7d82f6c403170e591310e36e18ef72efe258b17 not found: ID does not exist" Nov 02 18:09:31.689085 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.689063 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9"] Nov 02 18:09:31.693115 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.693090 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/725576cc-826f-4e80-8a9a-7e0deef9a840-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/725576cc-826f-4e80-8a9a-7e0deef9a840-routing-overlay-volume\") pod \"725576cc-826f-4e80-8a9a-7e0deef9a840\" (UID: \"725576cc-826f-4e80-8a9a-7e0deef9a840\") " Nov 02 18:09:31.693623 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.693468 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/725576cc-826f-4e80-8a9a-7e0deef9a840-kube-api-access-g479f\" (UniqueName: \"kubernetes.io/projected/725576cc-826f-4e80-8a9a-7e0deef9a840-kube-api-access-g479f\") pod \"725576cc-826f-4e80-8a9a-7e0deef9a840\" (UID: \"725576cc-826f-4e80-8a9a-7e0deef9a840\") " Nov 02 18:09:31.693623 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.693586 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/725576cc-826f-4e80-8a9a-7e0deef9a840-routing-overlay-volume" pod "725576cc-826f-4e80-8a9a-7e0deef9a840" (UID: "725576cc-826f-4e80-8a9a-7e0deef9a840"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:31.694293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.694130 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/725576cc-826f-4e80-8a9a-7e0deef9a840-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725576cc-826f-4e80-8a9a-7e0deef9a840-plugins-config-volume\") pod \"725576cc-826f-4e80-8a9a-7e0deef9a840\" (UID: \"725576cc-826f-4e80-8a9a-7e0deef9a840\") " Nov 02 18:09:31.694293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.694275 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/725576cc-826f-4e80-8a9a-7e0deef9a840-provider-credentials\" (UniqueName: \"kubernetes.io/projected/725576cc-826f-4e80-8a9a-7e0deef9a840-provider-credentials\") pod \"725576cc-826f-4e80-8a9a-7e0deef9a840\" (UID: \"725576cc-826f-4e80-8a9a-7e0deef9a840\") " Nov 02 18:09:31.694438 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.694306 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/725576cc-826f-4e80-8a9a-7e0deef9a840-plugins-config-volume" pod "725576cc-826f-4e80-8a9a-7e0deef9a840" (UID: "725576cc-826f-4e80-8a9a-7e0deef9a840"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:31.695209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.695179 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjhjv\" (UniqueName: \"kubernetes.io/projected/8027f9f1-260a-427f-bc4d-c228611e91e0-kube-api-access-jjhjv\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:31.695318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.695213 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725576cc-826f-4e80-8a9a-7e0deef9a840-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:31.695318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.695229 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8027f9f1-260a-427f-bc4d-c228611e91e0-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:31.695318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.695243 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/725576cc-826f-4e80-8a9a-7e0deef9a840-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:31.695318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.695255 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8027f9f1-260a-427f-bc4d-c228611e91e0-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:31.695318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.695269 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8027f9f1-260a-427f-bc4d-c228611e91e0-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:31.696381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.696359 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725576cc-826f-4e80-8a9a-7e0deef9a840-kube-api-access-g479f" pod "725576cc-826f-4e80-8a9a-7e0deef9a840" (UID: "725576cc-826f-4e80-8a9a-7e0deef9a840"). InnerVolumeSpecName "kube-api-access-g479f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:31.696854 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.696822 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725576cc-826f-4e80-8a9a-7e0deef9a840-provider-credentials" pod "725576cc-826f-4e80-8a9a-7e0deef9a840" (UID: "725576cc-826f-4e80-8a9a-7e0deef9a840"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:31.696991 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.696919 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7755k9sp9"] Nov 02 18:09:31.704783 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.704744 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg"] Nov 02 18:09:31.711640 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.711619 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-f8b57mszg"] Nov 02 18:09:31.716881 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.716861 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd"] Nov 02 18:09:31.720591 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.720574 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdr9lkd"] Nov 02 18:09:31.796730 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.796696 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g479f\" (UniqueName: \"kubernetes.io/projected/725576cc-826f-4e80-8a9a-7e0deef9a840-kube-api-access-g479f\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:31.796730 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:31.796728 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/725576cc-826f-4e80-8a9a-7e0deef9a840-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:32.159036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:32.158912 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:09:32.329670 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:32.329629 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660b29d1-be18-4a4c-ae47-1e38d196388c" path="/var/lib/kubelet/pods/660b29d1-be18-4a4c-ae47-1e38d196388c/volumes" Nov 02 18:09:32.330338 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:32.330318 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dd876e5-3e6e-45de-9f5b-4148388c34cd" path="/var/lib/kubelet/pods/7dd876e5-3e6e-45de-9f5b-4148388c34cd/volumes" Nov 02 18:09:32.330883 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:32.330864 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8027f9f1-260a-427f-bc4d-c228611e91e0" path="/var/lib/kubelet/pods/8027f9f1-260a-427f-bc4d-c228611e91e0/volumes" Nov 02 18:09:32.446267 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:32.446230 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:32.650906 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:32.650872 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" Nov 02 18:09:32.651120 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:32.650910 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk" event={"ID":"725576cc-826f-4e80-8a9a-7e0deef9a840","Type":"ContainerDied","Data":"e6a8e8ed144316918043cd1daa1714c73e337ca401abf6e0852beba2d8dd444b"} Nov 02 18:09:32.651120 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:32.650992 5549 scope.go:122] "RemoveContainer" containerID="27c70cdcf5891e7cc6fb2b8bf8f3b15091c2cc48b05eb2ace87b562ab9835691" Nov 02 18:09:32.674072 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:32.674037 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk"] Nov 02 18:09:32.678039 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:32.678006 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-5599lp9kk"] Nov 02 18:09:32.964051 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:32.964008 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:33.771980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:33.770544 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:34.000976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:33.993490 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:34.395978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:34.395228 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="725576cc-826f-4e80-8a9a-7e0deef9a840" path="/var/lib/kubelet/pods/725576cc-826f-4e80-8a9a-7e0deef9a840/volumes" Nov 02 18:09:34.499979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:34.493186 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:34.673039 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:34.672538 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:34.847979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:34.847723 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:35.521975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:35.520599 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:36.206973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:36.203687 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:37.040977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:37.036985 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:37.810980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:37.810707 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:37.964577 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:37.964540 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:38.099207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.099124 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:38.543847 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.543811 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk"] Nov 02 18:09:38.545158 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.545133 5549 cpu_manager.go:414] "containerMap: removing container" podUID="725576cc-826f-4e80-8a9a-7e0deef9a840" containerName="payload-processing" Nov 02 18:09:38.545158 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.545157 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="725576cc-826f-4e80-8a9a-7e0deef9a840" containerName="payload-processing" Nov 02 18:09:38.545435 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.545176 5549 cpu_manager.go:414] "containerMap: removing container" podUID="660b29d1-be18-4a4c-ae47-1e38d196388c" containerName="payload-processing" Nov 02 18:09:38.545435 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.545184 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660b29d1-be18-4a4c-ae47-1e38d196388c" containerName="payload-processing" Nov 02 18:09:38.545435 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.545250 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7dd876e5-3e6e-45de-9f5b-4148388c34cd" containerName="payload-processing" Nov 02 18:09:38.545435 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.545259 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd876e5-3e6e-45de-9f5b-4148388c34cd" containerName="payload-processing" Nov 02 18:09:38.545435 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.545276 5549 cpu_manager.go:414] "containerMap: removing container" podUID="8027f9f1-260a-427f-bc4d-c228611e91e0" containerName="payload-processing" Nov 02 18:09:38.545435 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.545281 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8027f9f1-260a-427f-bc4d-c228611e91e0" containerName="payload-processing" Nov 02 18:09:38.545435 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.545431 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="725576cc-826f-4e80-8a9a-7e0deef9a840" containerName="payload-processing" Nov 02 18:09:38.545435 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.545442 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="660b29d1-be18-4a4c-ae47-1e38d196388c" containerName="payload-processing" Nov 02 18:09:38.546177 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.545456 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd876e5-3e6e-45de-9f5b-4148388c34cd" containerName="payload-processing" Nov 02 18:09:38.546177 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.545464 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8027f9f1-260a-427f-bc4d-c228611e91e0" containerName="payload-processing" Nov 02 18:09:38.552093 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.552066 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" Nov 02 18:09:38.556163 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.556135 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk"] Nov 02 18:09:38.556466 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.556446 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"agc-cred-441aad9490a2\"" Nov 02 18:09:38.687811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.687770 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk\" (UID: \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" Nov 02 18:09:38.687998 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.687853 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk\" (UID: \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" Nov 02 18:09:38.688163 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.688140 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk\" (UID: \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" Nov 02 18:09:38.688267 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.688243 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dqpz\" (UniqueName: \"kubernetes.io/projected/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-kube-api-access-6dqpz\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk\" (UID: \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" Nov 02 18:09:38.794232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.793064 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk\" (UID: \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" Nov 02 18:09:38.794232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.793182 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dqpz\" (UniqueName: \"kubernetes.io/projected/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-kube-api-access-6dqpz\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk\" (UID: \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" Nov 02 18:09:38.794232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.793243 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk\" (UID: \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" Nov 02 18:09:38.794232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.793325 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk\" (UID: \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" Nov 02 18:09:38.796473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.795151 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk\" (UID: \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" Nov 02 18:09:38.796473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.795422 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk\" (UID: \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" Nov 02 18:09:38.798556 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.797556 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk\" (UID: \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" Nov 02 18:09:38.809448 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.809419 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dqpz\" (UniqueName: \"kubernetes.io/projected/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-kube-api-access-6dqpz\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk\" (UID: \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" Nov 02 18:09:38.875544 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.875510 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" Nov 02 18:09:38.906153 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.906118 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg"] Nov 02 18:09:38.913220 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.913195 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" Nov 02 18:09:38.915116 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.914645 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg"] Nov 02 18:09:38.947518 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.940596 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"openshift-service-ca.crt\"" Nov 02 18:09:38.947518 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.941166 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"kube-root-ca.crt\"" Nov 02 18:09:38.947518 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.941244 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"agc-cred-db1d42f7323f\"" Nov 02 18:09:38.947518 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.941317 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"payload-processing-external-model-plugins-e2e-extmod-a-36a991\"" Nov 02 18:09:38.947518 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.941327 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"payload-processing-external-model-e2e--59b59218-dockercfg-mwhk2\"" Nov 02 18:09:38.947518 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.941391 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"routing-overlay\"" Nov 02 18:09:38.967379 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.967295 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:38.996845 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.996152 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmgfn\" (UniqueName: \"kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-kube-api-access-zmgfn\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg\" (UID: \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" Nov 02 18:09:38.997550 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.997086 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg\" (UID: \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" Nov 02 18:09:38.997550 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.997277 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg\" (UID: \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" Nov 02 18:09:38.997550 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:38.997370 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg\" (UID: \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" Nov 02 18:09:39.104025 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:39.103324 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg\" (UID: \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" Nov 02 18:09:39.104025 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:39.103500 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg\" (UID: \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" Nov 02 18:09:39.104025 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:39.103654 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmgfn\" (UniqueName: \"kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-kube-api-access-zmgfn\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg\" (UID: \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" Nov 02 18:09:39.104025 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:39.103696 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg\" (UID: \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" Nov 02 18:09:39.112379 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:39.107784 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg\" (UID: \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" Nov 02 18:09:39.112379 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:39.108460 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg\" (UID: \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" Nov 02 18:09:39.218974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:39.218729 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:39.355095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:39.314645 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmgfn\" (UniqueName: \"kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-kube-api-access-zmgfn\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg\" (UID: \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" Nov 02 18:09:39.355095 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:09:39.335942 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47bd69ca_a6f7_4399_a5b8_b3a27e7a3086.slice/crio-e08167d41665fa0f63a4cb1d7dd15808623cce07516f99a0ccdb7e634849c359 WatchSource:0}: Error finding container e08167d41665fa0f63a4cb1d7dd15808623cce07516f99a0ccdb7e634849c359: Status 404 returned error can't find the container with id e08167d41665fa0f63a4cb1d7dd15808623cce07516f99a0ccdb7e634849c359 Nov 02 18:09:39.426217 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:39.426167 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk"] Nov 02 18:09:39.428390 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:39.428352 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg\" (UID: \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" Nov 02 18:09:39.573637 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:39.573603 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" Nov 02 18:09:39.797885 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:39.797847 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" event={"ID":"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086","Type":"ContainerStarted","Data":"03c6ff1a5d909e923397ea57adba0339e0ec37a7f8592b731931d40b975dd051"} Nov 02 18:09:39.798080 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:39.797915 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" event={"ID":"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086","Type":"ContainerStarted","Data":"e08167d41665fa0f63a4cb1d7dd15808623cce07516f99a0ccdb7e634849c359"} Nov 02 18:09:39.798080 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:39.797946 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" Nov 02 18:09:39.823390 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:39.823339 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" podStartSLOduration=1.823324234 podStartE2EDuration="1.823324234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:09:39.821807135 +0000 UTC m=+2151.610312663" watchObservedRunningTime="2026-11-02 18:09:39.823324234 +0000 UTC m=+2151.611829811" Nov 02 18:09:39.863862 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:09:39.863830 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85cfd2f9_6e96_4933_ba22_f8e0ae1297c0.slice/crio-cf13ffa14107a100bf88e63768f142bb196588021e747ae96d0c692f0e3c6512 WatchSource:0}: Error finding container cf13ffa14107a100bf88e63768f142bb196588021e747ae96d0c692f0e3c6512: Status 404 returned error can't find the container with id cf13ffa14107a100bf88e63768f142bb196588021e747ae96d0c692f0e3c6512 Nov 02 18:09:39.864074 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:39.864051 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg"] Nov 02 18:09:39.966000 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:39.945813 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:40.217647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:40.217608 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg"] Nov 02 18:09:40.333999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:40.333947 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:40.811516 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:40.811468 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" event={"ID":"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0","Type":"ContainerStarted","Data":"09297dee90f472c48fc5f3725af3809128ccc60d290e46ba9ea98e6800dade69"} Nov 02 18:09:40.811812 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:40.811540 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" event={"ID":"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0","Type":"ContainerStarted","Data":"cf13ffa14107a100bf88e63768f142bb196588021e747ae96d0c692f0e3c6512"} Nov 02 18:09:40.811900 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:40.811859 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" Nov 02 18:09:40.812055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:40.811930 5549 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" secret="" err="secret \"payload-processing-external-model-e2e--59b59218-dockercfg-mwhk2\" not found" Nov 02 18:09:40.835242 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:40.835183 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" podStartSLOduration=2.835163959 podStartE2EDuration="2.835163959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:09:40.82913495 +0000 UTC m=+2152.617640479" watchObservedRunningTime="2026-11-02 18:09:40.835163959 +0000 UTC m=+2152.623669522" Nov 02 18:09:40.966232 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:40.966194 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-36a991/agc-cred-db1d42f7323f: secret "agc-cred-db1d42f7323f" not found Nov 02 18:09:40.966232 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:40.966227 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg: secret "agc-cred-db1d42f7323f" not found Nov 02 18:09:40.966698 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:40.966262 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-plugins-e2e-extmod-a-36a991: configmap "payload-processing-external-model-plugins-e2e-extmod-a-36a991" not found Nov 02 18:09:40.966698 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:40.966291 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-provider-credentials podName:85cfd2f9-6e96-4933-ba22-f8e0ae1297c0 nodeName:}" failed. No retries permitted until 2026-11-02 18:09:41.466271218 +0000 UTC m=+2153.254776738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" (UID: "85cfd2f9-6e96-4933-ba22-f8e0ae1297c0") : secret "agc-cred-db1d42f7323f" not found Nov 02 18:09:40.966698 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:40.966312 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-plugins-config-volume podName:85cfd2f9-6e96-4933-ba22-f8e0ae1297c0 nodeName:}" failed. No retries permitted until 2026-11-02 18:09:41.466303507 +0000 UTC m=+2153.254809017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" (UID: "85cfd2f9-6e96-4933-ba22-f8e0ae1297c0") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-36a991" not found Nov 02 18:09:41.157876 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:41.157789 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:41.301270 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:41.301236 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:41.475531 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:41.475497 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-36a991/agc-cred-db1d42f7323f: secret "agc-cred-db1d42f7323f" not found Nov 02 18:09:41.475531 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:41.475529 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg: secret "agc-cred-db1d42f7323f" not found Nov 02 18:09:41.475813 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:41.475581 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-provider-credentials podName:85cfd2f9-6e96-4933-ba22-f8e0ae1297c0 nodeName:}" failed. No retries permitted until 2026-11-02 18:09:42.475563976 +0000 UTC m=+2154.264069483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" (UID: "85cfd2f9-6e96-4933-ba22-f8e0ae1297c0") : secret "agc-cred-db1d42f7323f" not found Nov 02 18:09:41.475813 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:41.475599 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-plugins-e2e-extmod-a-36a991: configmap "payload-processing-external-model-plugins-e2e-extmod-a-36a991" not found Nov 02 18:09:41.475813 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:41.475649 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-plugins-config-volume podName:85cfd2f9-6e96-4933-ba22-f8e0ae1297c0 nodeName:}" failed. No retries permitted until 2026-11-02 18:09:42.475634202 +0000 UTC m=+2154.264139708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" (UID: "85cfd2f9-6e96-4933-ba22-f8e0ae1297c0") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-36a991" not found Nov 02 18:09:42.048137 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:42.048105 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:42.229975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:42.224036 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:42.400031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:42.399517 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:42.508146 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:42.508104 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-36a991/agc-cred-db1d42f7323f: secret "agc-cred-db1d42f7323f" not found Nov 02 18:09:42.508146 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:42.508136 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg: secret "agc-cred-db1d42f7323f" not found Nov 02 18:09:42.508393 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:42.508213 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-provider-credentials podName:85cfd2f9-6e96-4933-ba22-f8e0ae1297c0 nodeName:}" failed. No retries permitted until 2026-11-02 18:09:44.508192929 +0000 UTC m=+2156.296698436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" (UID: "85cfd2f9-6e96-4933-ba22-f8e0ae1297c0") : secret "agc-cred-db1d42f7323f" not found Nov 02 18:09:42.508633 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:42.508610 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-plugins-e2e-extmod-a-36a991: configmap "payload-processing-external-model-plugins-e2e-extmod-a-36a991" not found Nov 02 18:09:42.508744 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:42.508672 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-plugins-config-volume podName:85cfd2f9-6e96-4933-ba22-f8e0ae1297c0 nodeName:}" failed. No retries permitted until 2026-11-02 18:09:44.508656793 +0000 UTC m=+2156.297162300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" (UID: "85cfd2f9-6e96-4933-ba22-f8e0ae1297c0") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-36a991" not found Nov 02 18:09:42.576145 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:42.576097 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:42.811778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:42.811738 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f"] Nov 02 18:09:42.817427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:42.817402 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" Nov 02 18:09:42.822027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:42.821847 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 18:09:42.822027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:42.821890 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 18:09:42.822027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:42.821922 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-68e7dc8d1887\"" Nov 02 18:09:42.822330 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:42.822087 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-kpkqq\"" Nov 02 18:09:42.822392 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:42.822369 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 18:09:42.823616 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:42.823571 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 18:09:42.827186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:42.827155 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f"] Nov 02 18:09:42.843471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:42.843437 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:42.915382 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:42.915350 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/689de785-ba56-465d-a8a2-29a28c4d4a13-provider-credentials\") pod \"payload-processing-external-model-cbbcc59d-kwr9f\" (UID: \"689de785-ba56-465d-a8a2-29a28c4d4a13\") " pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" Nov 02 18:09:42.915635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:42.915399 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llxwf\" (UniqueName: \"kubernetes.io/projected/689de785-ba56-465d-a8a2-29a28c4d4a13-kube-api-access-llxwf\") pod \"payload-processing-external-model-cbbcc59d-kwr9f\" (UID: \"689de785-ba56-465d-a8a2-29a28c4d4a13\") " pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" Nov 02 18:09:42.915635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:42.915553 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/689de785-ba56-465d-a8a2-29a28c4d4a13-routing-overlay-volume\") pod \"payload-processing-external-model-cbbcc59d-kwr9f\" (UID: \"689de785-ba56-465d-a8a2-29a28c4d4a13\") " pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" Nov 02 18:09:42.915737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:42.915642 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/689de785-ba56-465d-a8a2-29a28c4d4a13-plugins-config-volume\") pod \"payload-processing-external-model-cbbcc59d-kwr9f\" (UID: \"689de785-ba56-465d-a8a2-29a28c4d4a13\") " pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" Nov 02 18:09:43.017594 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:43.017537 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/689de785-ba56-465d-a8a2-29a28c4d4a13-routing-overlay-volume\") pod \"payload-processing-external-model-cbbcc59d-kwr9f\" (UID: \"689de785-ba56-465d-a8a2-29a28c4d4a13\") " pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" Nov 02 18:09:43.017815 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:43.017636 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/689de785-ba56-465d-a8a2-29a28c4d4a13-plugins-config-volume\") pod \"payload-processing-external-model-cbbcc59d-kwr9f\" (UID: \"689de785-ba56-465d-a8a2-29a28c4d4a13\") " pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" Nov 02 18:09:43.018037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:43.017990 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/689de785-ba56-465d-a8a2-29a28c4d4a13-provider-credentials\") pod \"payload-processing-external-model-cbbcc59d-kwr9f\" (UID: \"689de785-ba56-465d-a8a2-29a28c4d4a13\") " pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" Nov 02 18:09:43.018145 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:43.018077 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llxwf\" (UniqueName: \"kubernetes.io/projected/689de785-ba56-465d-a8a2-29a28c4d4a13-kube-api-access-llxwf\") pod \"payload-processing-external-model-cbbcc59d-kwr9f\" (UID: \"689de785-ba56-465d-a8a2-29a28c4d4a13\") " pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" Nov 02 18:09:43.018461 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:43.018433 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/689de785-ba56-465d-a8a2-29a28c4d4a13-routing-overlay-volume\") pod \"payload-processing-external-model-cbbcc59d-kwr9f\" (UID: \"689de785-ba56-465d-a8a2-29a28c4d4a13\") " pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" Nov 02 18:09:43.019636 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:43.019612 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/689de785-ba56-465d-a8a2-29a28c4d4a13-plugins-config-volume\") pod \"payload-processing-external-model-cbbcc59d-kwr9f\" (UID: \"689de785-ba56-465d-a8a2-29a28c4d4a13\") " pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" Nov 02 18:09:43.024600 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:43.024545 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/689de785-ba56-465d-a8a2-29a28c4d4a13-provider-credentials\") pod \"payload-processing-external-model-cbbcc59d-kwr9f\" (UID: \"689de785-ba56-465d-a8a2-29a28c4d4a13\") " pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" Nov 02 18:09:43.063734 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:43.063201 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llxwf\" (UniqueName: \"kubernetes.io/projected/689de785-ba56-465d-a8a2-29a28c4d4a13-kube-api-access-llxwf\") pod \"payload-processing-external-model-cbbcc59d-kwr9f\" (UID: \"689de785-ba56-465d-a8a2-29a28c4d4a13\") " pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" Nov 02 18:09:43.151574 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:43.151271 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" Nov 02 18:09:43.395765 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:43.387317 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:43.567917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:43.566383 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:43.712425 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:43.712394 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f"] Nov 02 18:09:43.712540 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:09:43.712403 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689de785_ba56_465d_a8a2_29a28c4d4a13.slice/crio-04c8ce8b62b127791ccdbe479268fbe2b06f371c9dc148c7c075e7fa14ff4ca3 WatchSource:0}: Error finding container 04c8ce8b62b127791ccdbe479268fbe2b06f371c9dc148c7c075e7fa14ff4ca3: Status 404 returned error can't find the container with id 04c8ce8b62b127791ccdbe479268fbe2b06f371c9dc148c7c075e7fa14ff4ca3 Nov 02 18:09:43.788154 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:43.788124 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:43.844054 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:43.844013 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" event={"ID":"689de785-ba56-465d-a8a2-29a28c4d4a13","Type":"ContainerStarted","Data":"e3a91df7aeae3bb01eb7ad83cd2d39627a630644e57d02bd4b43c013226b8608"} Nov 02 18:09:43.844230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:43.844077 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" event={"ID":"689de785-ba56-465d-a8a2-29a28c4d4a13","Type":"ContainerStarted","Data":"04c8ce8b62b127791ccdbe479268fbe2b06f371c9dc148c7c075e7fa14ff4ca3"} Nov 02 18:09:43.844230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:43.844106 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" Nov 02 18:09:43.975026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:43.969881 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" podStartSLOduration=1.969859483 podStartE2EDuration="1.969859483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:09:43.872413603 +0000 UTC m=+2155.660919132" watchObservedRunningTime="2026-11-02 18:09:43.969859483 +0000 UTC m=+2155.758365011" Nov 02 18:09:44.398795 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:44.398695 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-769b787cbf-h9x98"] Nov 02 18:09:44.404214 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:44.404189 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" Nov 02 18:09:44.408196 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:44.408167 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-041be4a981fc\"" Nov 02 18:09:44.412076 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:44.412051 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-769b787cbf-h9x98"] Nov 02 18:09:44.467150 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:44.467103 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73f075da-0542-4e75-9b04-ae4c5c3879e1-routing-overlay-volume\") pod \"payload-processing-external-model-769b787cbf-h9x98\" (UID: \"73f075da-0542-4e75-9b04-ae4c5c3879e1\") " pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" Nov 02 18:09:44.467343 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:44.467223 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f075da-0542-4e75-9b04-ae4c5c3879e1-plugins-config-volume\") pod \"payload-processing-external-model-769b787cbf-h9x98\" (UID: \"73f075da-0542-4e75-9b04-ae4c5c3879e1\") " pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" Nov 02 18:09:44.467435 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:44.467343 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/73f075da-0542-4e75-9b04-ae4c5c3879e1-provider-credentials\") pod \"payload-processing-external-model-769b787cbf-h9x98\" (UID: \"73f075da-0542-4e75-9b04-ae4c5c3879e1\") " pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" Nov 02 18:09:44.467485 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:44.467445 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m7cc\" (UniqueName: \"kubernetes.io/projected/73f075da-0542-4e75-9b04-ae4c5c3879e1-kube-api-access-7m7cc\") pod \"payload-processing-external-model-769b787cbf-h9x98\" (UID: \"73f075da-0542-4e75-9b04-ae4c5c3879e1\") " pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" Nov 02 18:09:44.569266 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:44.569224 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73f075da-0542-4e75-9b04-ae4c5c3879e1-routing-overlay-volume\") pod \"payload-processing-external-model-769b787cbf-h9x98\" (UID: \"73f075da-0542-4e75-9b04-ae4c5c3879e1\") " pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" Nov 02 18:09:44.569449 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:44.569305 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-36a991/agc-cred-db1d42f7323f: secret "agc-cred-db1d42f7323f" not found Nov 02 18:09:44.569449 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:44.569328 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg: secret "agc-cred-db1d42f7323f" not found Nov 02 18:09:44.569449 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:44.569337 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f075da-0542-4e75-9b04-ae4c5c3879e1-plugins-config-volume\") pod \"payload-processing-external-model-769b787cbf-h9x98\" (UID: \"73f075da-0542-4e75-9b04-ae4c5c3879e1\") " pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" Nov 02 18:09:44.569449 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:44.569392 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-provider-credentials podName:85cfd2f9-6e96-4933-ba22-f8e0ae1297c0 nodeName:}" failed. No retries permitted until 2026-11-02 18:09:48.569371103 +0000 UTC m=+2160.357876636 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" (UID: "85cfd2f9-6e96-4933-ba22-f8e0ae1297c0") : secret "agc-cred-db1d42f7323f" not found Nov 02 18:09:44.569449 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:44.569415 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-plugins-e2e-extmod-a-36a991: configmap "payload-processing-external-model-plugins-e2e-extmod-a-36a991" not found Nov 02 18:09:44.569763 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:44.569477 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-plugins-config-volume podName:85cfd2f9-6e96-4933-ba22-f8e0ae1297c0 nodeName:}" failed. No retries permitted until 2026-11-02 18:09:48.569460554 +0000 UTC m=+2160.357966078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" (UID: "85cfd2f9-6e96-4933-ba22-f8e0ae1297c0") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-36a991" not found Nov 02 18:09:44.569763 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:44.569550 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/73f075da-0542-4e75-9b04-ae4c5c3879e1-provider-credentials\") pod \"payload-processing-external-model-769b787cbf-h9x98\" (UID: \"73f075da-0542-4e75-9b04-ae4c5c3879e1\") " pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" Nov 02 18:09:44.569763 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:44.569708 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m7cc\" (UniqueName: \"kubernetes.io/projected/73f075da-0542-4e75-9b04-ae4c5c3879e1-kube-api-access-7m7cc\") pod \"payload-processing-external-model-769b787cbf-h9x98\" (UID: \"73f075da-0542-4e75-9b04-ae4c5c3879e1\") " pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" Nov 02 18:09:44.570084 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:44.570059 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73f075da-0542-4e75-9b04-ae4c5c3879e1-routing-overlay-volume\") pod \"payload-processing-external-model-769b787cbf-h9x98\" (UID: \"73f075da-0542-4e75-9b04-ae4c5c3879e1\") " pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" Nov 02 18:09:44.570295 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:44.570275 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f075da-0542-4e75-9b04-ae4c5c3879e1-plugins-config-volume\") pod \"payload-processing-external-model-769b787cbf-h9x98\" (UID: \"73f075da-0542-4e75-9b04-ae4c5c3879e1\") " pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" Nov 02 18:09:44.572770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:44.572743 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/73f075da-0542-4e75-9b04-ae4c5c3879e1-provider-credentials\") pod \"payload-processing-external-model-769b787cbf-h9x98\" (UID: \"73f075da-0542-4e75-9b04-ae4c5c3879e1\") " pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" Nov 02 18:09:44.580664 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:44.580639 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m7cc\" (UniqueName: \"kubernetes.io/projected/73f075da-0542-4e75-9b04-ae4c5c3879e1-kube-api-access-7m7cc\") pod \"payload-processing-external-model-769b787cbf-h9x98\" (UID: \"73f075da-0542-4e75-9b04-ae4c5c3879e1\") " pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" Nov 02 18:09:44.725020 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:44.724940 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" Nov 02 18:09:44.957977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:44.948494 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:45.016351 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:09:45.015899 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f075da_0542_4e75_9b04_ae4c5c3879e1.slice/crio-77fb73e980cf62ae084ef70141b1fb214d113999c8a77fa88b3703bd9c7afb5f WatchSource:0}: Error finding container 77fb73e980cf62ae084ef70141b1fb214d113999c8a77fa88b3703bd9c7afb5f: Status 404 returned error can't find the container with id 77fb73e980cf62ae084ef70141b1fb214d113999c8a77fa88b3703bd9c7afb5f Nov 02 18:09:45.016578 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:45.016547 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-769b787cbf-h9x98"] Nov 02 18:09:45.923571 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:45.923534 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:45.985468 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:45.984150 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" event={"ID":"73f075da-0542-4e75-9b04-ae4c5c3879e1","Type":"ContainerStarted","Data":"e999eae5d6f042dde19fef815b0c307800d4b0fb65b1c8b018ad910b5068c3a0"} Nov 02 18:09:45.985468 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:45.984227 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" event={"ID":"73f075da-0542-4e75-9b04-ae4c5c3879e1","Type":"ContainerStarted","Data":"77fb73e980cf62ae084ef70141b1fb214d113999c8a77fa88b3703bd9c7afb5f"} Nov 02 18:09:45.985468 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:45.984879 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" Nov 02 18:09:46.013560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:46.013502 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" podStartSLOduration=2.013488367 podStartE2EDuration="2.013488367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:09:46.011528211 +0000 UTC m=+2157.800033739" watchObservedRunningTime="2026-11-02 18:09:46.013488367 +0000 UTC m=+2157.801993895" Nov 02 18:09:46.755991 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:46.748027 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:46.823575 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:46.823518 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" podUID="85cfd2f9-6e96-4933-ba22-f8e0ae1297c0" containerName="payload-processing" containerID="cri-o://09297dee90f472c48fc5f3725af3809128ccc60d290e46ba9ea98e6800dade69" gracePeriod=25 Nov 02 18:09:46.995821 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:46.995782 5549 generic.go:356] "Generic (PLEG): container finished" podID="85cfd2f9-6e96-4933-ba22-f8e0ae1297c0" containerID="09297dee90f472c48fc5f3725af3809128ccc60d290e46ba9ea98e6800dade69" exitCode=0 Nov 02 18:09:46.996238 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:46.995844 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" event={"ID":"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0","Type":"ContainerDied","Data":"09297dee90f472c48fc5f3725af3809128ccc60d290e46ba9ea98e6800dade69"} Nov 02 18:09:47.124040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:47.124010 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" Nov 02 18:09:47.211429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:47.211387 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-provider-credentials\") pod \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\" (UID: \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\") " Nov 02 18:09:47.211578 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:47.211538 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-plugins-config-volume\") pod \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\" (UID: \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\") " Nov 02 18:09:47.211653 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:47.211640 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-routing-overlay-volume\") pod \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\" (UID: \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\") " Nov 02 18:09:47.211720 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:47.211710 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-kube-api-access-zmgfn\" (UniqueName: \"kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-kube-api-access-zmgfn\") pod \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\" (UID: \"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0\") " Nov 02 18:09:47.212032 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:47.212002 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-routing-overlay-volume" pod "85cfd2f9-6e96-4933-ba22-f8e0ae1297c0" (UID: "85cfd2f9-6e96-4933-ba22-f8e0ae1297c0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:47.212032 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:47.212021 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-plugins-config-volume" pod "85cfd2f9-6e96-4933-ba22-f8e0ae1297c0" (UID: "85cfd2f9-6e96-4933-ba22-f8e0ae1297c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:47.212379 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:47.212358 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:47.212441 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:47.212384 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:47.213824 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:47.213801 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-provider-credentials" pod "85cfd2f9-6e96-4933-ba22-f8e0ae1297c0" (UID: "85cfd2f9-6e96-4933-ba22-f8e0ae1297c0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:47.213941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:47.213860 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-kube-api-access-zmgfn" pod "85cfd2f9-6e96-4933-ba22-f8e0ae1297c0" (UID: "85cfd2f9-6e96-4933-ba22-f8e0ae1297c0"). InnerVolumeSpecName "kube-api-access-zmgfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:47.313463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:47.313428 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmgfn\" (UniqueName: \"kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-kube-api-access-zmgfn\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:47.313463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:47.313460 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:47.611975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:47.605839 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:47.661999 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:47.656271 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:09:48.005581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:48.005553 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" Nov 02 18:09:48.006050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:48.005580 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg" event={"ID":"85cfd2f9-6e96-4933-ba22-f8e0ae1297c0","Type":"ContainerDied","Data":"cf13ffa14107a100bf88e63768f142bb196588021e747ae96d0c692f0e3c6512"} Nov 02 18:09:48.006050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:48.005641 5549 scope.go:122] "RemoveContainer" containerID="09297dee90f472c48fc5f3725af3809128ccc60d290e46ba9ea98e6800dade69" Nov 02 18:09:48.036732 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:48.036703 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg"] Nov 02 18:09:48.041038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:48.041012 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-9658sgggg"] Nov 02 18:09:48.322935 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:48.322855 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85cfd2f9-6e96-4933-ba22-f8e0ae1297c0" path="/var/lib/kubelet/pods/85cfd2f9-6e96-4933-ba22-f8e0ae1297c0/volumes" Nov 02 18:09:48.525401 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:48.521975 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:49.182559 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:49.182512 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:49.970982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:49.963669 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:50.630551 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:50.630436 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:50.815405 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:50.815375 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" Nov 02 18:09:50.873993 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:50.873937 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl"] Nov 02 18:09:51.321625 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:51.321541 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:52.071778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:52.071737 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:52.823404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:52.823363 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:53.556630 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:53.556580 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:54.322982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:54.322021 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:54.935585 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:54.935555 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" Nov 02 18:09:54.988627 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:54.988590 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:55.804833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:55.804798 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:56.043043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:56.043003 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" podUID="34c9626d-d20e-47cb-b78c-88f1aeccdc4a" containerName="payload-processing" containerID="cri-o://7df5703b7b6ffe9595baec022e70aff955967eee00e8ba2a21567a81b2cc2669" gracePeriod=25 Nov 02 18:09:56.338587 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:56.338560 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" Nov 02 18:09:56.411072 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:56.411033 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-kube-api-access-8xszs\" (UniqueName: \"kubernetes.io/projected/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-kube-api-access-8xszs\") pod \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\" (UID: \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\") " Nov 02 18:09:56.411439 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:56.411409 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-provider-credentials\") pod \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\" (UID: \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\") " Nov 02 18:09:56.411574 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:56.411462 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-routing-overlay-volume\") pod \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\" (UID: \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\") " Nov 02 18:09:56.411574 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:56.411515 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-plugins-config-volume\") pod \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\" (UID: \"34c9626d-d20e-47cb-b78c-88f1aeccdc4a\") " Nov 02 18:09:56.412155 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:56.412126 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-routing-overlay-volume" pod "34c9626d-d20e-47cb-b78c-88f1aeccdc4a" (UID: "34c9626d-d20e-47cb-b78c-88f1aeccdc4a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:56.412273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:56.412179 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-plugins-config-volume" pod "34c9626d-d20e-47cb-b78c-88f1aeccdc4a" (UID: "34c9626d-d20e-47cb-b78c-88f1aeccdc4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:56.413222 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:56.413195 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:56.413341 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:56.413320 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:56.414069 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:56.414046 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-kube-api-access-8xszs" pod "34c9626d-d20e-47cb-b78c-88f1aeccdc4a" (UID: "34c9626d-d20e-47cb-b78c-88f1aeccdc4a"). InnerVolumeSpecName "kube-api-access-8xszs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:56.414281 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:56.414265 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-provider-credentials" pod "34c9626d-d20e-47cb-b78c-88f1aeccdc4a" (UID: "34c9626d-d20e-47cb-b78c-88f1aeccdc4a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:56.541977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:56.540540 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:56.541977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:56.540581 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xszs\" (UniqueName: \"kubernetes.io/projected/34c9626d-d20e-47cb-b78c-88f1aeccdc4a-kube-api-access-8xszs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:56.560170 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:56.560074 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:56.999014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:56.998982 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" Nov 02 18:09:57.058374 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:57.058335 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f"] Nov 02 18:09:57.227598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:57.227564 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:57.263272 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:57.263175 5549 generic.go:356] "Generic (PLEG): container finished" podID="34c9626d-d20e-47cb-b78c-88f1aeccdc4a" containerID="7df5703b7b6ffe9595baec022e70aff955967eee00e8ba2a21567a81b2cc2669" exitCode=0 Nov 02 18:09:57.263767 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:57.263645 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" Nov 02 18:09:57.266075 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:57.263667 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" event={"ID":"34c9626d-d20e-47cb-b78c-88f1aeccdc4a","Type":"ContainerDied","Data":"7df5703b7b6ffe9595baec022e70aff955967eee00e8ba2a21567a81b2cc2669"} Nov 02 18:09:57.266874 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:57.266784 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl" event={"ID":"34c9626d-d20e-47cb-b78c-88f1aeccdc4a","Type":"ContainerDied","Data":"927b938ed2207e91b5fcd19c93bb22c0ccec207208749ace4c6196bf81d02b6d"} Nov 02 18:09:57.267106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:57.267035 5549 scope.go:122] "RemoveContainer" containerID="7df5703b7b6ffe9595baec022e70aff955967eee00e8ba2a21567a81b2cc2669" Nov 02 18:09:57.290627 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:57.290605 5549 scope.go:122] "RemoveContainer" containerID="7df5703b7b6ffe9595baec022e70aff955967eee00e8ba2a21567a81b2cc2669" Nov 02 18:09:57.290986 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:09:57.290936 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df5703b7b6ffe9595baec022e70aff955967eee00e8ba2a21567a81b2cc2669\": container with ID starting with 7df5703b7b6ffe9595baec022e70aff955967eee00e8ba2a21567a81b2cc2669 not found: ID does not exist" containerID="7df5703b7b6ffe9595baec022e70aff955967eee00e8ba2a21567a81b2cc2669" Nov 02 18:09:57.291058 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:57.290984 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df5703b7b6ffe9595baec022e70aff955967eee00e8ba2a21567a81b2cc2669"} err="failed to get container status \"7df5703b7b6ffe9595baec022e70aff955967eee00e8ba2a21567a81b2cc2669\": rpc error: code = NotFound desc = could not find container \"7df5703b7b6ffe9595baec022e70aff955967eee00e8ba2a21567a81b2cc2669\": container with ID starting with 7df5703b7b6ffe9595baec022e70aff955967eee00e8ba2a21567a81b2cc2669 not found: ID does not exist" Nov 02 18:09:57.313487 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:57.313457 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl"] Nov 02 18:09:57.317648 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:57.317624 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9frltsl"] Nov 02 18:09:58.323786 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:58.323750 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34c9626d-d20e-47cb-b78c-88f1aeccdc4a" path="/var/lib/kubelet/pods/34c9626d-d20e-47cb-b78c-88f1aeccdc4a/volumes" Nov 02 18:09:58.419343 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:58.419307 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:59.465101 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.464946 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:59.647974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.637864 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:59.823307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.823222 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh"] Nov 02 18:09:59.824872 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.824847 5549 cpu_manager.go:414] "containerMap: removing container" podUID="34c9626d-d20e-47cb-b78c-88f1aeccdc4a" containerName="payload-processing" Nov 02 18:09:59.824959 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.824878 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c9626d-d20e-47cb-b78c-88f1aeccdc4a" containerName="payload-processing" Nov 02 18:09:59.824959 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.824942 5549 cpu_manager.go:414] "containerMap: removing container" podUID="85cfd2f9-6e96-4933-ba22-f8e0ae1297c0" containerName="payload-processing" Nov 02 18:09:59.825040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.824967 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85cfd2f9-6e96-4933-ba22-f8e0ae1297c0" containerName="payload-processing" Nov 02 18:09:59.825225 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.825207 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="85cfd2f9-6e96-4933-ba22-f8e0ae1297c0" containerName="payload-processing" Nov 02 18:09:59.825268 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.825246 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c9626d-d20e-47cb-b78c-88f1aeccdc4a" containerName="payload-processing" Nov 02 18:09:59.834904 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.834867 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" Nov 02 18:09:59.835675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.834872 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh"] Nov 02 18:09:59.839848 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.839813 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5ec436\"/\"routing-overlay\"" Nov 02 18:09:59.840095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.839847 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5ec436\"/\"payload-processing-external-model-plugins-e2e-extmod-a-5ec436\"" Nov 02 18:09:59.840095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.839899 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5ec436\"/\"payload-processing-external-model-e2e--96dec6bd-dockercfg-qbg29\"" Nov 02 18:09:59.840289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.839912 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5ec436\"/\"agc-cred-c13e2e0d36b2\"" Nov 02 18:09:59.841283 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.841262 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5ec436\"/\"kube-root-ca.crt\"" Nov 02 18:09:59.841439 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.841361 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5ec436\"/\"openshift-service-ca.crt\"" Nov 02 18:09:59.938462 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.938413 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d45b1e63-976c-4dc3-86e1-2f824ae345be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh\" (UID: \"d45b1e63-976c-4dc3-86e1-2f824ae345be\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" Nov 02 18:09:59.938692 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.938674 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d45b1e63-976c-4dc3-86e1-2f824ae345be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh\" (UID: \"d45b1e63-976c-4dc3-86e1-2f824ae345be\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" Nov 02 18:09:59.938777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.938724 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkplx\" (UniqueName: \"kubernetes.io/projected/d45b1e63-976c-4dc3-86e1-2f824ae345be-kube-api-access-mkplx\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh\" (UID: \"d45b1e63-976c-4dc3-86e1-2f824ae345be\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" Nov 02 18:09:59.938850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:09:59.938787 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d45b1e63-976c-4dc3-86e1-2f824ae345be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh\" (UID: \"d45b1e63-976c-4dc3-86e1-2f824ae345be\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" Nov 02 18:10:00.040502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:00.040446 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d45b1e63-976c-4dc3-86e1-2f824ae345be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh\" (UID: \"d45b1e63-976c-4dc3-86e1-2f824ae345be\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" Nov 02 18:10:00.040721 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:00.040702 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d45b1e63-976c-4dc3-86e1-2f824ae345be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh\" (UID: \"d45b1e63-976c-4dc3-86e1-2f824ae345be\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" Nov 02 18:10:00.040801 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:00.040755 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkplx\" (UniqueName: \"kubernetes.io/projected/d45b1e63-976c-4dc3-86e1-2f824ae345be-kube-api-access-mkplx\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh\" (UID: \"d45b1e63-976c-4dc3-86e1-2f824ae345be\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" Nov 02 18:10:00.040865 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:00.040831 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d45b1e63-976c-4dc3-86e1-2f824ae345be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh\" (UID: \"d45b1e63-976c-4dc3-86e1-2f824ae345be\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" Nov 02 18:10:00.041648 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:00.041611 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d45b1e63-976c-4dc3-86e1-2f824ae345be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh\" (UID: \"d45b1e63-976c-4dc3-86e1-2f824ae345be\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" Nov 02 18:10:00.041798 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:00.041691 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d45b1e63-976c-4dc3-86e1-2f824ae345be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh\" (UID: \"d45b1e63-976c-4dc3-86e1-2f824ae345be\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" Nov 02 18:10:00.043500 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:00.043473 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d45b1e63-976c-4dc3-86e1-2f824ae345be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh\" (UID: \"d45b1e63-976c-4dc3-86e1-2f824ae345be\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" Nov 02 18:10:00.051716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:00.051679 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkplx\" (UniqueName: \"kubernetes.io/projected/d45b1e63-976c-4dc3-86e1-2f824ae345be-kube-api-access-mkplx\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh\" (UID: \"d45b1e63-976c-4dc3-86e1-2f824ae345be\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" Nov 02 18:10:00.182716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:00.182610 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" Nov 02 18:10:00.406336 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:00.406298 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh"] Nov 02 18:10:00.576026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:00.569488 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:00.576026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:00.570441 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh"] Nov 02 18:10:00.726975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:00.726321 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:01.069173 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:01.069136 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:01.354540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:01.354500 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" event={"ID":"d45b1e63-976c-4dc3-86e1-2f824ae345be","Type":"ContainerStarted","Data":"b04fcf2d52d5b6cecbfa691411a6a7f31cc3f6f507ac98c78b2beaff585d6766"} Nov 02 18:10:01.433977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:01.432779 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:01.901978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:01.900692 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5lphd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:10:01.901978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:01.900772 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" podUID="781f545f-9659-4d21-9e53-85d4b82a7214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:10:02.058755 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.058716 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" podUID="689de785-ba56-465d-a8a2-29a28c4d4a13" containerName="payload-processing" containerID="cri-o://e3a91df7aeae3bb01eb7ad83cd2d39627a630644e57d02bd4b43c013226b8608" gracePeriod=25 Nov 02 18:10:02.370756 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.370723 5549 generic.go:356] "Generic (PLEG): container finished" podID="689de785-ba56-465d-a8a2-29a28c4d4a13" containerID="e3a91df7aeae3bb01eb7ad83cd2d39627a630644e57d02bd4b43c013226b8608" exitCode=0 Nov 02 18:10:02.370966 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.370794 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" event={"ID":"689de785-ba56-465d-a8a2-29a28c4d4a13","Type":"ContainerDied","Data":"e3a91df7aeae3bb01eb7ad83cd2d39627a630644e57d02bd4b43c013226b8608"} Nov 02 18:10:02.372670 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.372644 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" event={"ID":"d45b1e63-976c-4dc3-86e1-2f824ae345be","Type":"ContainerStarted","Data":"437a67e8eab069e80d9e6b80fe176edfdd0b28f23d20b59956a3b41216ac0380"} Nov 02 18:10:02.372788 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.372715 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" Nov 02 18:10:02.700557 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.700464 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" podStartSLOduration=3.700428179 podStartE2EDuration="3.700428179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:02.583261961 +0000 UTC m=+2174.371767527" watchObservedRunningTime="2026-11-02 18:10:02.700428179 +0000 UTC m=+2174.488933708" Nov 02 18:10:02.721199 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.721169 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" Nov 02 18:10:02.878711 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.778046 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:02.878711 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.815311 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/689de785-ba56-465d-a8a2-29a28c4d4a13-provider-credentials\" (UniqueName: \"kubernetes.io/projected/689de785-ba56-465d-a8a2-29a28c4d4a13-provider-credentials\") pod \"689de785-ba56-465d-a8a2-29a28c4d4a13\" (UID: \"689de785-ba56-465d-a8a2-29a28c4d4a13\") " Nov 02 18:10:02.878711 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.856861 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/689de785-ba56-465d-a8a2-29a28c4d4a13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/689de785-ba56-465d-a8a2-29a28c4d4a13-plugins-config-volume\") pod \"689de785-ba56-465d-a8a2-29a28c4d4a13\" (UID: \"689de785-ba56-465d-a8a2-29a28c4d4a13\") " Nov 02 18:10:02.878711 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.857017 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/689de785-ba56-465d-a8a2-29a28c4d4a13-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/689de785-ba56-465d-a8a2-29a28c4d4a13-routing-overlay-volume\") pod \"689de785-ba56-465d-a8a2-29a28c4d4a13\" (UID: \"689de785-ba56-465d-a8a2-29a28c4d4a13\") " Nov 02 18:10:02.878711 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.857089 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/689de785-ba56-465d-a8a2-29a28c4d4a13-kube-api-access-llxwf\" (UniqueName: \"kubernetes.io/projected/689de785-ba56-465d-a8a2-29a28c4d4a13-kube-api-access-llxwf\") pod \"689de785-ba56-465d-a8a2-29a28c4d4a13\" (UID: \"689de785-ba56-465d-a8a2-29a28c4d4a13\") " Nov 02 18:10:02.878711 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.858604 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/689de785-ba56-465d-a8a2-29a28c4d4a13-plugins-config-volume" pod "689de785-ba56-465d-a8a2-29a28c4d4a13" (UID: "689de785-ba56-465d-a8a2-29a28c4d4a13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:02.878711 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.859018 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/689de785-ba56-465d-a8a2-29a28c4d4a13-routing-overlay-volume" pod "689de785-ba56-465d-a8a2-29a28c4d4a13" (UID: "689de785-ba56-465d-a8a2-29a28c4d4a13"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:02.915261 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.887887 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689de785-ba56-465d-a8a2-29a28c4d4a13-kube-api-access-llxwf" pod "689de785-ba56-465d-a8a2-29a28c4d4a13" (UID: "689de785-ba56-465d-a8a2-29a28c4d4a13"). InnerVolumeSpecName "kube-api-access-llxwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:02.915261 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.888274 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689de785-ba56-465d-a8a2-29a28c4d4a13-provider-credentials" pod "689de785-ba56-465d-a8a2-29a28c4d4a13" (UID: "689de785-ba56-465d-a8a2-29a28c4d4a13"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:02.960818 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.959906 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/689de785-ba56-465d-a8a2-29a28c4d4a13-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:02.960818 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.959945 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/689de785-ba56-465d-a8a2-29a28c4d4a13-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:02.960818 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.960004 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/689de785-ba56-465d-a8a2-29a28c4d4a13-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:02.960818 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:02.960019 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llxwf\" (UniqueName: \"kubernetes.io/projected/689de785-ba56-465d-a8a2-29a28c4d4a13-kube-api-access-llxwf\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:03.382064 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:03.382036 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" Nov 02 18:10:03.382255 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:03.382078 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f" event={"ID":"689de785-ba56-465d-a8a2-29a28c4d4a13","Type":"ContainerDied","Data":"04c8ce8b62b127791ccdbe479268fbe2b06f371c9dc148c7c075e7fa14ff4ca3"} Nov 02 18:10:03.382255 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:03.382136 5549 scope.go:122] "RemoveContainer" containerID="e3a91df7aeae3bb01eb7ad83cd2d39627a630644e57d02bd4b43c013226b8608" Nov 02 18:10:03.409197 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:03.409166 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f"] Nov 02 18:10:03.413489 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:03.413466 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-cbbcc59d-kwr9f"] Nov 02 18:10:03.647488 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:03.641893 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:04.344971 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:04.343492 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:04.390270 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:04.388504 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="689de785-ba56-465d-a8a2-29a28c4d4a13" path="/var/lib/kubelet/pods/689de785-ba56-465d-a8a2-29a28c4d4a13/volumes" Nov 02 18:10:05.036973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:05.026655 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:05.684975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:05.671668 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:06.533816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:06.531574 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:07.250644 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:07.231794 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:07.372931 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:07.372886 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" podUID="d45b1e63-976c-4dc3-86e1-2f824ae345be" containerName="payload-processing" containerID="cri-o://437a67e8eab069e80d9e6b80fe176edfdd0b28f23d20b59956a3b41216ac0380" gracePeriod=25 Nov 02 18:10:07.430081 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:07.429365 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:07.578923 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:07.578742 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:07.684598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:07.684576 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" Nov 02 18:10:07.754848 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:07.754808 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d45b1e63-976c-4dc3-86e1-2f824ae345be-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d45b1e63-976c-4dc3-86e1-2f824ae345be-provider-credentials\") pod \"d45b1e63-976c-4dc3-86e1-2f824ae345be\" (UID: \"d45b1e63-976c-4dc3-86e1-2f824ae345be\") " Nov 02 18:10:07.755106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:07.754866 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d45b1e63-976c-4dc3-86e1-2f824ae345be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d45b1e63-976c-4dc3-86e1-2f824ae345be-plugins-config-volume\") pod \"d45b1e63-976c-4dc3-86e1-2f824ae345be\" (UID: \"d45b1e63-976c-4dc3-86e1-2f824ae345be\") " Nov 02 18:10:07.755106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:07.754985 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d45b1e63-976c-4dc3-86e1-2f824ae345be-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d45b1e63-976c-4dc3-86e1-2f824ae345be-routing-overlay-volume\") pod \"d45b1e63-976c-4dc3-86e1-2f824ae345be\" (UID: \"d45b1e63-976c-4dc3-86e1-2f824ae345be\") " Nov 02 18:10:07.755236 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:07.755195 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d45b1e63-976c-4dc3-86e1-2f824ae345be-kube-api-access-mkplx\" (UniqueName: \"kubernetes.io/projected/d45b1e63-976c-4dc3-86e1-2f824ae345be-kube-api-access-mkplx\") pod \"d45b1e63-976c-4dc3-86e1-2f824ae345be\" (UID: \"d45b1e63-976c-4dc3-86e1-2f824ae345be\") " Nov 02 18:10:07.755416 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:07.755380 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d45b1e63-976c-4dc3-86e1-2f824ae345be-routing-overlay-volume" pod "d45b1e63-976c-4dc3-86e1-2f824ae345be" (UID: "d45b1e63-976c-4dc3-86e1-2f824ae345be"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:07.755568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:07.755389 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d45b1e63-976c-4dc3-86e1-2f824ae345be-plugins-config-volume" pod "d45b1e63-976c-4dc3-86e1-2f824ae345be" (UID: "d45b1e63-976c-4dc3-86e1-2f824ae345be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:07.756145 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:07.756116 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d45b1e63-976c-4dc3-86e1-2f824ae345be-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:07.756250 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:07.756149 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d45b1e63-976c-4dc3-86e1-2f824ae345be-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:07.757362 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:07.757339 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45b1e63-976c-4dc3-86e1-2f824ae345be-provider-credentials" pod "d45b1e63-976c-4dc3-86e1-2f824ae345be" (UID: "d45b1e63-976c-4dc3-86e1-2f824ae345be"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:07.757450 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:07.757376 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45b1e63-976c-4dc3-86e1-2f824ae345be-kube-api-access-mkplx" pod "d45b1e63-976c-4dc3-86e1-2f824ae345be" (UID: "d45b1e63-976c-4dc3-86e1-2f824ae345be"). InnerVolumeSpecName "kube-api-access-mkplx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:07.858082 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:07.857999 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d45b1e63-976c-4dc3-86e1-2f824ae345be-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:07.858082 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:07.858028 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkplx\" (UniqueName: \"kubernetes.io/projected/d45b1e63-976c-4dc3-86e1-2f824ae345be-kube-api-access-mkplx\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:07.938328 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:07.938277 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:08.447423 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:08.447386 5549 generic.go:356] "Generic (PLEG): container finished" podID="d45b1e63-976c-4dc3-86e1-2f824ae345be" containerID="437a67e8eab069e80d9e6b80fe176edfdd0b28f23d20b59956a3b41216ac0380" exitCode=0 Nov 02 18:10:08.447880 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:08.447435 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" Nov 02 18:10:08.447880 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:08.447461 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" event={"ID":"d45b1e63-976c-4dc3-86e1-2f824ae345be","Type":"ContainerDied","Data":"437a67e8eab069e80d9e6b80fe176edfdd0b28f23d20b59956a3b41216ac0380"} Nov 02 18:10:08.447880 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:08.447513 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh" event={"ID":"d45b1e63-976c-4dc3-86e1-2f824ae345be","Type":"ContainerDied","Data":"b04fcf2d52d5b6cecbfa691411a6a7f31cc3f6f507ac98c78b2beaff585d6766"} Nov 02 18:10:08.447880 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:08.447534 5549 scope.go:122] "RemoveContainer" containerID="437a67e8eab069e80d9e6b80fe176edfdd0b28f23d20b59956a3b41216ac0380" Nov 02 18:10:08.459574 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:08.459553 5549 scope.go:122] "RemoveContainer" containerID="437a67e8eab069e80d9e6b80fe176edfdd0b28f23d20b59956a3b41216ac0380" Nov 02 18:10:08.459843 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:08.459820 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"437a67e8eab069e80d9e6b80fe176edfdd0b28f23d20b59956a3b41216ac0380\": container with ID starting with 437a67e8eab069e80d9e6b80fe176edfdd0b28f23d20b59956a3b41216ac0380 not found: ID does not exist" containerID="437a67e8eab069e80d9e6b80fe176edfdd0b28f23d20b59956a3b41216ac0380" Nov 02 18:10:08.459928 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:08.459853 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437a67e8eab069e80d9e6b80fe176edfdd0b28f23d20b59956a3b41216ac0380"} err="failed to get container status \"437a67e8eab069e80d9e6b80fe176edfdd0b28f23d20b59956a3b41216ac0380\": rpc error: code = NotFound desc = could not find container \"437a67e8eab069e80d9e6b80fe176edfdd0b28f23d20b59956a3b41216ac0380\": container with ID starting with 437a67e8eab069e80d9e6b80fe176edfdd0b28f23d20b59956a3b41216ac0380 not found: ID does not exist" Nov 02 18:10:08.474531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:08.474506 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh"] Nov 02 18:10:08.479941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:08.479917 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-58ffpxhnh"] Nov 02 18:10:08.686070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:08.679129 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:09.462007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.461969 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t"] Nov 02 18:10:09.463685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.463541 5549 cpu_manager.go:414] "containerMap: removing container" podUID="d45b1e63-976c-4dc3-86e1-2f824ae345be" containerName="payload-processing" Nov 02 18:10:09.463685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.463573 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d45b1e63-976c-4dc3-86e1-2f824ae345be" containerName="payload-processing" Nov 02 18:10:09.463685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.463608 5549 cpu_manager.go:414] "containerMap: removing container" podUID="689de785-ba56-465d-a8a2-29a28c4d4a13" containerName="payload-processing" Nov 02 18:10:09.463685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.463616 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="689de785-ba56-465d-a8a2-29a28c4d4a13" containerName="payload-processing" Nov 02 18:10:09.464028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.463903 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d45b1e63-976c-4dc3-86e1-2f824ae345be" containerName="payload-processing" Nov 02 18:10:09.464028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.463929 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="689de785-ba56-465d-a8a2-29a28c4d4a13" containerName="payload-processing" Nov 02 18:10:09.474522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.474480 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:09.482069 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.482037 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.484434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.484407 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t"] Nov 02 18:10:09.600905 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.600872 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk72s\" (UniqueName: \"kubernetes.io/projected/ca0043f1-fb84-4f38-b637-2315256e0c46-kube-api-access-gk72s\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.601096 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.601024 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.601096 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.601066 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.601180 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.601117 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.601217 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.601201 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.601269 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.601254 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0043f1-fb84-4f38-b637-2315256e0c46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.601418 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.601401 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.601459 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.601442 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.601604 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.601578 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.703316 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.703276 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.703522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.703370 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0043f1-fb84-4f38-b637-2315256e0c46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.703522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.703515 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.703627 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.703563 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.703690 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.703646 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.703750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.703701 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.703812 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.703772 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk72s\" (UniqueName: \"kubernetes.io/projected/ca0043f1-fb84-4f38-b637-2315256e0c46-kube-api-access-gk72s\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.703879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.703859 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.703936 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.703894 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.703936 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.703914 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.703936 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.703929 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.704132 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.704035 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.704132 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.704057 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.704479 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.704458 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0043f1-fb84-4f38-b637-2315256e0c46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.706156 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.706122 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.707466 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.707437 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.712303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.712252 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk72s\" (UniqueName: \"kubernetes.io/projected/ca0043f1-fb84-4f38-b637-2315256e0c46-kube-api-access-gk72s\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.712303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.712250 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-msm5t\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:09.832900 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:09.832869 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:10.126076 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:10.124411 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:10.183402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:10.183370 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t"] Nov 02 18:10:10.184878 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:10:10.184852 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0043f1_fb84_4f38_b637_2315256e0c46.slice/crio-291d829d2649c92b73ee7549731c3a123dc8bffa44e0b667a2db61be016c4204 WatchSource:0}: Error finding container 291d829d2649c92b73ee7549731c3a123dc8bffa44e0b667a2db61be016c4204: Status 404 returned error can't find the container with id 291d829d2649c92b73ee7549731c3a123dc8bffa44e0b667a2db61be016c4204 Nov 02 18:10:10.188605 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:10.188574 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:10:10.188685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:10.188641 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:10:10.188685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:10.188668 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:10:10.324685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:10.324652 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d45b1e63-976c-4dc3-86e1-2f824ae345be" path="/var/lib/kubelet/pods/d45b1e63-976c-4dc3-86e1-2f824ae345be/volumes" Nov 02 18:10:10.494583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:10.494545 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" event={"ID":"ca0043f1-fb84-4f38-b637-2315256e0c46","Type":"ContainerStarted","Data":"52aa504a43c50ad1cd12420fa399ab6a81cdd39423c82e42c208f1297a4456ce"} Nov 02 18:10:10.495031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:10.494602 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" event={"ID":"ca0043f1-fb84-4f38-b637-2315256e0c46","Type":"ContainerStarted","Data":"291d829d2649c92b73ee7549731c3a123dc8bffa44e0b667a2db61be016c4204"} Nov 02 18:10:10.522158 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:10.522107 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" podStartSLOduration=1.522094632 podStartE2EDuration="1.522094632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:10.51811137 +0000 UTC m=+2182.306616899" watchObservedRunningTime="2026-11-02 18:10:10.522094632 +0000 UTC m=+2182.310600160" Nov 02 18:10:10.837043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:10.836030 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:10.852081 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:10.850887 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:11.613047 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:11.612537 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:11.835381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:11.835345 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-msm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 02 18:10:11.835568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:11.835409 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" podUID="ca0043f1-fb84-4f38-b637-2315256e0c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 02 18:10:12.417977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:12.412285 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:12.834071 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:12.834038 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-msm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 02 18:10:12.834443 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:12.834088 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" podUID="ca0043f1-fb84-4f38-b637-2315256e0c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 02 18:10:13.003978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:13.003582 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:13.812981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:13.810156 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:13.838976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:13.837576 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-msm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 02 18:10:13.838976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:13.837639 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" podUID="ca0043f1-fb84-4f38-b637-2315256e0c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 02 18:10:14.425698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:14.425661 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:14.599977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:14.595815 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:14.833271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:14.833232 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-msm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 02 18:10:14.833531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:14.833291 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" podUID="ca0043f1-fb84-4f38-b637-2315256e0c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 02 18:10:15.305631 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:15.305589 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:15.590432 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:15.590342 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:15.833729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:15.833692 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-msm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 02 18:10:15.833928 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:15.833755 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" podUID="ca0043f1-fb84-4f38-b637-2315256e0c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 02 18:10:16.035606 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.035567 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj"] Nov 02 18:10:16.061814 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.061776 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:16.065251 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.065211 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" Nov 02 18:10:16.067468 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.067443 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj"] Nov 02 18:10:16.070108 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.069198 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fc0a0665a23b\"" Nov 02 18:10:16.150911 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.150043 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t2fl\" (UniqueName: \"kubernetes.io/projected/99704f7e-ab00-4874-a248-ddbbbc88a1b5-kube-api-access-7t2fl\") pod \"payload-processing-external-model-6fc469f668-5rmnj\" (UID: \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\") " pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" Nov 02 18:10:16.150911 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.150101 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99704f7e-ab00-4874-a248-ddbbbc88a1b5-provider-credentials\") pod \"payload-processing-external-model-6fc469f668-5rmnj\" (UID: \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\") " pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" Nov 02 18:10:16.150911 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.150312 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99704f7e-ab00-4874-a248-ddbbbc88a1b5-routing-overlay-volume\") pod \"payload-processing-external-model-6fc469f668-5rmnj\" (UID: \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\") " pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" Nov 02 18:10:16.150911 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.150372 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99704f7e-ab00-4874-a248-ddbbbc88a1b5-plugins-config-volume\") pod \"payload-processing-external-model-6fc469f668-5rmnj\" (UID: \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\") " pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" Nov 02 18:10:16.252696 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.252655 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99704f7e-ab00-4874-a248-ddbbbc88a1b5-routing-overlay-volume\") pod \"payload-processing-external-model-6fc469f668-5rmnj\" (UID: \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\") " pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" Nov 02 18:10:16.252903 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.252711 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99704f7e-ab00-4874-a248-ddbbbc88a1b5-plugins-config-volume\") pod \"payload-processing-external-model-6fc469f668-5rmnj\" (UID: \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\") " pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" Nov 02 18:10:16.253036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.252932 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2fl\" (UniqueName: \"kubernetes.io/projected/99704f7e-ab00-4874-a248-ddbbbc88a1b5-kube-api-access-7t2fl\") pod \"payload-processing-external-model-6fc469f668-5rmnj\" (UID: \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\") " pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" Nov 02 18:10:16.253036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.253000 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99704f7e-ab00-4874-a248-ddbbbc88a1b5-provider-credentials\") pod \"payload-processing-external-model-6fc469f668-5rmnj\" (UID: \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\") " pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" Nov 02 18:10:16.253495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.253464 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99704f7e-ab00-4874-a248-ddbbbc88a1b5-routing-overlay-volume\") pod \"payload-processing-external-model-6fc469f668-5rmnj\" (UID: \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\") " pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" Nov 02 18:10:16.253701 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.253679 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99704f7e-ab00-4874-a248-ddbbbc88a1b5-plugins-config-volume\") pod \"payload-processing-external-model-6fc469f668-5rmnj\" (UID: \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\") " pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" Nov 02 18:10:16.255641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.255611 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99704f7e-ab00-4874-a248-ddbbbc88a1b5-provider-credentials\") pod \"payload-processing-external-model-6fc469f668-5rmnj\" (UID: \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\") " pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" Nov 02 18:10:16.261862 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.261837 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t2fl\" (UniqueName: \"kubernetes.io/projected/99704f7e-ab00-4874-a248-ddbbbc88a1b5-kube-api-access-7t2fl\") pod \"payload-processing-external-model-6fc469f668-5rmnj\" (UID: \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\") " pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" Nov 02 18:10:16.485983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.483305 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" Nov 02 18:10:16.794037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.794004 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj"] Nov 02 18:10:16.794223 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:10:16.794028 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99704f7e_ab00_4874_a248_ddbbbc88a1b5.slice/crio-1e7c2f8addb7c7cb51ef1e26628b13ebbfd8ad69ee84604f8dd0230c86a48cec WatchSource:0}: Error finding container 1e7c2f8addb7c7cb51ef1e26628b13ebbfd8ad69ee84604f8dd0230c86a48cec: Status 404 returned error can't find the container with id 1e7c2f8addb7c7cb51ef1e26628b13ebbfd8ad69ee84604f8dd0230c86a48cec Nov 02 18:10:16.825581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.825546 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:16.853879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.853372 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-msm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 02 18:10:16.853879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:16.853443 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" podUID="ca0043f1-fb84-4f38-b637-2315256e0c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 02 18:10:17.593700 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:17.593666 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:10:17.639294 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:17.639244 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" event={"ID":"99704f7e-ab00-4874-a248-ddbbbc88a1b5","Type":"ContainerStarted","Data":"24b906a834783a0015226dc3b8ff693145d801f9e620d6d94bcd4118efb37593"} Nov 02 18:10:17.639488 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:17.639354 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" event={"ID":"99704f7e-ab00-4874-a248-ddbbbc88a1b5","Type":"ContainerStarted","Data":"1e7c2f8addb7c7cb51ef1e26628b13ebbfd8ad69ee84604f8dd0230c86a48cec"} Nov 02 18:10:17.640942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:17.640914 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" Nov 02 18:10:17.678987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:17.672130 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:17.728505 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:17.701048 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" podStartSLOduration=1.70102446 podStartE2EDuration="1.70102446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:17.6572964 +0000 UTC m=+2189.445801930" watchObservedRunningTime="2026-11-02 18:10:17.70102446 +0000 UTC m=+2189.489529989" Nov 02 18:10:17.834230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:17.834195 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-msm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 02 18:10:17.834395 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:17.834244 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" podUID="ca0043f1-fb84-4f38-b637-2315256e0c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 02 18:10:18.420980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:18.418547 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:19.069979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:19.061661 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:19.466456 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:19.466419 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:19.466941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:19.466915 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:19.467688 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:19.467668 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:10:19.862617 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:19.860882 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:20.621860 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:20.618932 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:21.324707 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:21.324671 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:21.974486 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:21.974452 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:22.615662 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:22.615610 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:23.414976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:23.408080 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:24.143031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:24.142635 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:24.720583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:24.720552 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:24.869362 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:24.869325 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx"] Nov 02 18:10:24.941973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:24.934316 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" Nov 02 18:10:25.006683 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:24.976578 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:25.010182 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.010146 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx\" (UID: \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" Nov 02 18:10:25.013797 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.013671 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx"] Nov 02 18:10:25.014368 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.014339 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnq2b\" (UniqueName: \"kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-kube-api-access-mnq2b\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx\" (UID: \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" Nov 02 18:10:25.014652 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.014629 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx\" (UID: \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" Nov 02 18:10:25.014818 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.014800 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx\" (UID: \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" Nov 02 18:10:25.116162 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.116127 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnq2b\" (UniqueName: \"kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-kube-api-access-mnq2b\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx\" (UID: \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" Nov 02 18:10:25.116342 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.116245 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx\" (UID: \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" Nov 02 18:10:25.116342 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.116279 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx\" (UID: \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" Nov 02 18:10:25.116423 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.116382 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx\" (UID: \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" Nov 02 18:10:25.117112 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.117079 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx\" (UID: \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" Nov 02 18:10:25.117251 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.117080 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx\" (UID: \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" Nov 02 18:10:25.119028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.119004 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx\" (UID: \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" Nov 02 18:10:25.124189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.124164 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnq2b\" (UniqueName: \"kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-kube-api-access-mnq2b\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx\" (UID: \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" Nov 02 18:10:25.336649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.336572 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" Nov 02 18:10:25.624329 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:10:25.624299 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e24f2e6_b8a7_4a75_acdb_de49d3796804.slice/crio-4d1ebd98b5708e3af25696ff48e2afe4fa1d9efa1d900013dbb8e10cead956a4 WatchSource:0}: Error finding container 4d1ebd98b5708e3af25696ff48e2afe4fa1d9efa1d900013dbb8e10cead956a4: Status 404 returned error can't find the container with id 4d1ebd98b5708e3af25696ff48e2afe4fa1d9efa1d900013dbb8e10cead956a4 Nov 02 18:10:25.624610 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.624586 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx"] Nov 02 18:10:25.728078 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.727991 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6"] Nov 02 18:10:25.735673 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.735639 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:25.737600 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.737571 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx"] Nov 02 18:10:25.756745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.754099 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" event={"ID":"2e24f2e6-b8a7-4a75-acdb-de49d3796804","Type":"ContainerStarted","Data":"22fe5e4f19f7ce8a9776baa58c135944aedf4597a50572b0f3ac35f7b9860eca"} Nov 02 18:10:25.756745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.754168 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" Nov 02 18:10:25.756745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.754194 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" event={"ID":"2e24f2e6-b8a7-4a75-acdb-de49d3796804","Type":"ContainerStarted","Data":"4d1ebd98b5708e3af25696ff48e2afe4fa1d9efa1d900013dbb8e10cead956a4"} Nov 02 18:10:25.776658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.776600 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" podStartSLOduration=1.776581802 podStartE2EDuration="1.776581802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:25.772012869 +0000 UTC m=+2197.560518398" watchObservedRunningTime="2026-11-02 18:10:25.776581802 +0000 UTC m=+2197.565087331" Nov 02 18:10:25.827913 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:25.827878 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-42d786/agc-cred-c4e11282498a: secret "agc-cred-c4e11282498a" not found Nov 02 18:10:25.827913 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:25.827906 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx: secret "agc-cred-c4e11282498a" not found Nov 02 18:10:25.828186 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:25.827991 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-provider-credentials podName:2e24f2e6-b8a7-4a75-acdb-de49d3796804 nodeName:}" failed. No retries permitted until 2026-11-02 18:10:26.327940649 +0000 UTC m=+2198.116446155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" (UID: "2e24f2e6-b8a7-4a75-acdb-de49d3796804") : secret "agc-cred-c4e11282498a" not found Nov 02 18:10:25.828186 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:25.828172 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-plugins-e2e-extmod-a-42d786: configmap "payload-processing-external-model-plugins-e2e-extmod-a-42d786" not found Nov 02 18:10:25.828311 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:25.828259 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-plugins-config-volume podName:2e24f2e6-b8a7-4a75-acdb-de49d3796804 nodeName:}" failed. No retries permitted until 2026-11-02 18:10:26.328240374 +0000 UTC m=+2198.116745898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" (UID: "2e24f2e6-b8a7-4a75-acdb-de49d3796804") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-42d786" not found Nov 02 18:10:25.871021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:25.870934 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:26.356267 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:26.356235 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:26.387517 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:26.387481 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-plugins-e2e-extmod-a-42d786: configmap "payload-processing-external-model-plugins-e2e-extmod-a-42d786" not found Nov 02 18:10:26.387719 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:26.387553 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-plugins-config-volume podName:2e24f2e6-b8a7-4a75-acdb-de49d3796804 nodeName:}" failed. No retries permitted until 2026-11-02 18:10:27.387534937 +0000 UTC m=+2199.176040443 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" (UID: "2e24f2e6-b8a7-4a75-acdb-de49d3796804") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-42d786" not found Nov 02 18:10:26.387805 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:26.387717 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-42d786/agc-cred-c4e11282498a: secret "agc-cred-c4e11282498a" not found Nov 02 18:10:26.387805 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:26.387742 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx: secret "agc-cred-c4e11282498a" not found Nov 02 18:10:26.387917 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:26.387806 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-provider-credentials podName:2e24f2e6-b8a7-4a75-acdb-de49d3796804 nodeName:}" failed. No retries permitted until 2026-11-02 18:10:27.387788753 +0000 UTC m=+2199.176294276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" (UID: "2e24f2e6-b8a7-4a75-acdb-de49d3796804") : secret "agc-cred-c4e11282498a" not found Nov 02 18:10:27.022012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:27.017982 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:27.421742 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:27.421641 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-plugins-e2e-extmod-a-42d786: configmap "payload-processing-external-model-plugins-e2e-extmod-a-42d786" not found Nov 02 18:10:27.421742 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:27.421730 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-plugins-config-volume podName:2e24f2e6-b8a7-4a75-acdb-de49d3796804 nodeName:}" failed. No retries permitted until 2026-11-02 18:10:29.421709074 +0000 UTC m=+2201.210214597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" (UID: "2e24f2e6-b8a7-4a75-acdb-de49d3796804") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-42d786" not found Nov 02 18:10:27.422052 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:27.421917 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-42d786/agc-cred-c4e11282498a: secret "agc-cred-c4e11282498a" not found Nov 02 18:10:27.422052 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:27.421940 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx: secret "agc-cred-c4e11282498a" not found Nov 02 18:10:27.422170 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:27.422079 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-provider-credentials podName:2e24f2e6-b8a7-4a75-acdb-de49d3796804 nodeName:}" failed. No retries permitted until 2026-11-02 18:10:29.422053087 +0000 UTC m=+2201.210558606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" (UID: "2e24f2e6-b8a7-4a75-acdb-de49d3796804") : secret "agc-cred-c4e11282498a" not found Nov 02 18:10:27.619038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:27.618934 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:28.346023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:28.344193 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:28.982974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:28.976793 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:29.502503 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:29.502466 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-plugins-e2e-extmod-a-42d786: configmap "payload-processing-external-model-plugins-e2e-extmod-a-42d786" not found Nov 02 18:10:29.502934 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:29.502560 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-plugins-config-volume podName:2e24f2e6-b8a7-4a75-acdb-de49d3796804 nodeName:}" failed. No retries permitted until 2026-11-02 18:10:33.502541381 +0000 UTC m=+2205.291046908 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" (UID: "2e24f2e6-b8a7-4a75-acdb-de49d3796804") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-42d786" not found Nov 02 18:10:29.502934 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:29.502614 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-42d786/agc-cred-c4e11282498a: secret "agc-cred-c4e11282498a" not found Nov 02 18:10:29.502934 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:29.502636 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx: secret "agc-cred-c4e11282498a" not found Nov 02 18:10:29.502934 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:29.502692 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-provider-credentials podName:2e24f2e6-b8a7-4a75-acdb-de49d3796804 nodeName:}" failed. No retries permitted until 2026-11-02 18:10:33.502675733 +0000 UTC m=+2205.291181261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" (UID: "2e24f2e6-b8a7-4a75-acdb-de49d3796804") : secret "agc-cred-c4e11282498a" not found Nov 02 18:10:29.688709 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:29.688674 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:29.737492 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:29.737452 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" Nov 02 18:10:29.808206 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:29.808066 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-769b787cbf-h9x98"] Nov 02 18:10:30.426428 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:30.426386 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:30.729213 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:30.729174 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" podUID="2f06f984-c329-4bfc-9be0-5dfb2d44f597" containerName="payload-processing" containerID="cri-o://7d57c574e5cf1115490fabf003a1a2376a759e16a2b72f5f89c3dbcc9ab6722b" gracePeriod=25 Nov 02 18:10:31.045099 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.045066 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" Nov 02 18:10:31.091232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.090973 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:31.256186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.256102 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f06f984-c329-4bfc-9be0-5dfb2d44f597-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f06f984-c329-4bfc-9be0-5dfb2d44f597-provider-credentials\") pod \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\" (UID: \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\") " Nov 02 18:10:31.256186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.256171 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f06f984-c329-4bfc-9be0-5dfb2d44f597-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f06f984-c329-4bfc-9be0-5dfb2d44f597-plugins-config-volume\") pod \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\" (UID: \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\") " Nov 02 18:10:31.256385 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.256343 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f06f984-c329-4bfc-9be0-5dfb2d44f597-kube-api-access-6csfd\" (UniqueName: \"kubernetes.io/projected/2f06f984-c329-4bfc-9be0-5dfb2d44f597-kube-api-access-6csfd\") pod \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\" (UID: \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\") " Nov 02 18:10:31.256385 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.256374 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f06f984-c329-4bfc-9be0-5dfb2d44f597-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f06f984-c329-4bfc-9be0-5dfb2d44f597-routing-overlay-volume\") pod \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\" (UID: \"2f06f984-c329-4bfc-9be0-5dfb2d44f597\") " Nov 02 18:10:31.256770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.256633 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f06f984-c329-4bfc-9be0-5dfb2d44f597-plugins-config-volume" pod "2f06f984-c329-4bfc-9be0-5dfb2d44f597" (UID: "2f06f984-c329-4bfc-9be0-5dfb2d44f597"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:31.256902 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.256796 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f06f984-c329-4bfc-9be0-5dfb2d44f597-routing-overlay-volume" pod "2f06f984-c329-4bfc-9be0-5dfb2d44f597" (UID: "2f06f984-c329-4bfc-9be0-5dfb2d44f597"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:31.257188 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.257166 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f06f984-c329-4bfc-9be0-5dfb2d44f597-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:31.257246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.257193 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f06f984-c329-4bfc-9be0-5dfb2d44f597-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:31.258429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.258402 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f06f984-c329-4bfc-9be0-5dfb2d44f597-provider-credentials" pod "2f06f984-c329-4bfc-9be0-5dfb2d44f597" (UID: "2f06f984-c329-4bfc-9be0-5dfb2d44f597"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:31.258429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.258424 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f06f984-c329-4bfc-9be0-5dfb2d44f597-kube-api-access-6csfd" pod "2f06f984-c329-4bfc-9be0-5dfb2d44f597" (UID: "2f06f984-c329-4bfc-9be0-5dfb2d44f597"). InnerVolumeSpecName "kube-api-access-6csfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:31.358826 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.358788 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f06f984-c329-4bfc-9be0-5dfb2d44f597-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:31.358826 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.358821 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6csfd\" (UniqueName: \"kubernetes.io/projected/2f06f984-c329-4bfc-9be0-5dfb2d44f597-kube-api-access-6csfd\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:31.762946 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.762910 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" podUID="2e24f2e6-b8a7-4a75-acdb-de49d3796804" containerName="payload-processing" containerID="cri-o://22fe5e4f19f7ce8a9776baa58c135944aedf4597a50572b0f3ac35f7b9860eca" gracePeriod=25 Nov 02 18:10:31.840291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.830052 5549 generic.go:356] "Generic (PLEG): container finished" podID="2f06f984-c329-4bfc-9be0-5dfb2d44f597" containerID="7d57c574e5cf1115490fabf003a1a2376a759e16a2b72f5f89c3dbcc9ab6722b" exitCode=0 Nov 02 18:10:31.840291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.830391 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" event={"ID":"2f06f984-c329-4bfc-9be0-5dfb2d44f597","Type":"ContainerDied","Data":"7d57c574e5cf1115490fabf003a1a2376a759e16a2b72f5f89c3dbcc9ab6722b"} Nov 02 18:10:31.840291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.830449 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" event={"ID":"2f06f984-c329-4bfc-9be0-5dfb2d44f597","Type":"ContainerDied","Data":"a4f1e685c6f05c8504c4a38d6fb29c7408f1accf8299ca14ddabec1b1a6b2480"} Nov 02 18:10:31.840291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.830483 5549 scope.go:122] "RemoveContainer" containerID="7d57c574e5cf1115490fabf003a1a2376a759e16a2b72f5f89c3dbcc9ab6722b" Nov 02 18:10:31.840291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.830886 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6" Nov 02 18:10:31.842419 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.842387 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:31.901215 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.900193 5549 scope.go:122] "RemoveContainer" containerID="7d57c574e5cf1115490fabf003a1a2376a759e16a2b72f5f89c3dbcc9ab6722b" Nov 02 18:10:31.901215 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:31.900598 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d57c574e5cf1115490fabf003a1a2376a759e16a2b72f5f89c3dbcc9ab6722b\": container with ID starting with 7d57c574e5cf1115490fabf003a1a2376a759e16a2b72f5f89c3dbcc9ab6722b not found: ID does not exist" containerID="7d57c574e5cf1115490fabf003a1a2376a759e16a2b72f5f89c3dbcc9ab6722b" Nov 02 18:10:31.901215 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.900641 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d57c574e5cf1115490fabf003a1a2376a759e16a2b72f5f89c3dbcc9ab6722b"} err="failed to get container status \"7d57c574e5cf1115490fabf003a1a2376a759e16a2b72f5f89c3dbcc9ab6722b\": rpc error: code = NotFound desc = could not find container \"7d57c574e5cf1115490fabf003a1a2376a759e16a2b72f5f89c3dbcc9ab6722b\": container with ID starting with 7d57c574e5cf1115490fabf003a1a2376a759e16a2b72f5f89c3dbcc9ab6722b not found: ID does not exist" Nov 02 18:10:31.921696 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.921594 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6"] Nov 02 18:10:31.929369 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:31.928946 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-8cbck2wf6"] Nov 02 18:10:32.105057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.105028 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" Nov 02 18:10:32.278382 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.278330 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-routing-overlay-volume\") pod \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\" (UID: \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\") " Nov 02 18:10:32.278562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.278480 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-provider-credentials\") pod \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\" (UID: \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\") " Nov 02 18:10:32.278562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.278552 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-kube-api-access-mnq2b\" (UniqueName: \"kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-kube-api-access-mnq2b\") pod \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\" (UID: \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\") " Nov 02 18:10:32.278657 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.278634 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-routing-overlay-volume" pod "2e24f2e6-b8a7-4a75-acdb-de49d3796804" (UID: "2e24f2e6-b8a7-4a75-acdb-de49d3796804"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:32.278856 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.278835 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-plugins-config-volume\") pod \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\" (UID: \"2e24f2e6-b8a7-4a75-acdb-de49d3796804\") " Nov 02 18:10:32.279287 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.279266 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-plugins-config-volume" pod "2e24f2e6-b8a7-4a75-acdb-de49d3796804" (UID: "2e24f2e6-b8a7-4a75-acdb-de49d3796804"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:32.279632 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.279618 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:32.279677 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.279634 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e24f2e6-b8a7-4a75-acdb-de49d3796804-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:32.280700 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.280674 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-kube-api-access-mnq2b" pod "2e24f2e6-b8a7-4a75-acdb-de49d3796804" (UID: "2e24f2e6-b8a7-4a75-acdb-de49d3796804"). InnerVolumeSpecName "kube-api-access-mnq2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:32.280810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.280703 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-provider-credentials" pod "2e24f2e6-b8a7-4a75-acdb-de49d3796804" (UID: "2e24f2e6-b8a7-4a75-acdb-de49d3796804"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:32.324678 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.324613 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f06f984-c329-4bfc-9be0-5dfb2d44f597" path="/var/lib/kubelet/pods/2f06f984-c329-4bfc-9be0-5dfb2d44f597/volumes" Nov 02 18:10:32.381003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.380975 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:32.381003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.380999 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnq2b\" (UniqueName: \"kubernetes.io/projected/2e24f2e6-b8a7-4a75-acdb-de49d3796804-kube-api-access-mnq2b\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:32.586331 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.586246 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:32.720845 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.720800 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:32.842029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.841906 5549 generic.go:356] "Generic (PLEG): container finished" podID="2e24f2e6-b8a7-4a75-acdb-de49d3796804" containerID="22fe5e4f19f7ce8a9776baa58c135944aedf4597a50572b0f3ac35f7b9860eca" exitCode=0 Nov 02 18:10:32.842029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.841993 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" Nov 02 18:10:32.842029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.841999 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" event={"ID":"2e24f2e6-b8a7-4a75-acdb-de49d3796804","Type":"ContainerDied","Data":"22fe5e4f19f7ce8a9776baa58c135944aedf4597a50572b0f3ac35f7b9860eca"} Nov 02 18:10:32.842579 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.842058 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx" event={"ID":"2e24f2e6-b8a7-4a75-acdb-de49d3796804","Type":"ContainerDied","Data":"4d1ebd98b5708e3af25696ff48e2afe4fa1d9efa1d900013dbb8e10cead956a4"} Nov 02 18:10:32.842579 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.842080 5549 scope.go:122] "RemoveContainer" containerID="22fe5e4f19f7ce8a9776baa58c135944aedf4597a50572b0f3ac35f7b9860eca" Nov 02 18:10:32.853423 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.853399 5549 scope.go:122] "RemoveContainer" containerID="22fe5e4f19f7ce8a9776baa58c135944aedf4597a50572b0f3ac35f7b9860eca" Nov 02 18:10:32.853711 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:32.853684 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22fe5e4f19f7ce8a9776baa58c135944aedf4597a50572b0f3ac35f7b9860eca\": container with ID starting with 22fe5e4f19f7ce8a9776baa58c135944aedf4597a50572b0f3ac35f7b9860eca not found: ID does not exist" containerID="22fe5e4f19f7ce8a9776baa58c135944aedf4597a50572b0f3ac35f7b9860eca" Nov 02 18:10:32.853781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.853720 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22fe5e4f19f7ce8a9776baa58c135944aedf4597a50572b0f3ac35f7b9860eca"} err="failed to get container status \"22fe5e4f19f7ce8a9776baa58c135944aedf4597a50572b0f3ac35f7b9860eca\": rpc error: code = NotFound desc = could not find container \"22fe5e4f19f7ce8a9776baa58c135944aedf4597a50572b0f3ac35f7b9860eca\": container with ID starting with 22fe5e4f19f7ce8a9776baa58c135944aedf4597a50572b0f3ac35f7b9860eca not found: ID does not exist" Nov 02 18:10:32.863141 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.863119 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx"] Nov 02 18:10:32.863271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.863255 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:32.867393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:32.867375 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-7588pl9dx"] Nov 02 18:10:33.267999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:33.267901 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:33.998193 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:33.998159 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:10:34.075100 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:34.071808 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:34.323601 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:34.323525 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e24f2e6-b8a7-4a75-acdb-de49d3796804" path="/var/lib/kubelet/pods/2e24f2e6-b8a7-4a75-acdb-de49d3796804/volumes" Nov 02 18:10:34.808708 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:34.808664 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" podUID="73f075da-0542-4e75-9b04-ae4c5c3879e1" containerName="payload-processing" containerID="cri-o://e999eae5d6f042dde19fef815b0c307800d4b0fb65b1c8b018ad910b5068c3a0" gracePeriod=25 Nov 02 18:10:34.834821 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:34.834759 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:35.350025 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.349995 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" Nov 02 18:10:35.461022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.460993 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f075da-0542-4e75-9b04-ae4c5c3879e1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73f075da-0542-4e75-9b04-ae4c5c3879e1-routing-overlay-volume\") pod \"73f075da-0542-4e75-9b04-ae4c5c3879e1\" (UID: \"73f075da-0542-4e75-9b04-ae4c5c3879e1\") " Nov 02 18:10:35.461216 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.461139 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f075da-0542-4e75-9b04-ae4c5c3879e1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/73f075da-0542-4e75-9b04-ae4c5c3879e1-provider-credentials\") pod \"73f075da-0542-4e75-9b04-ae4c5c3879e1\" (UID: \"73f075da-0542-4e75-9b04-ae4c5c3879e1\") " Nov 02 18:10:35.461216 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.461203 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f075da-0542-4e75-9b04-ae4c5c3879e1-kube-api-access-7m7cc\" (UniqueName: \"kubernetes.io/projected/73f075da-0542-4e75-9b04-ae4c5c3879e1-kube-api-access-7m7cc\") pod \"73f075da-0542-4e75-9b04-ae4c5c3879e1\" (UID: \"73f075da-0542-4e75-9b04-ae4c5c3879e1\") " Nov 02 18:10:35.461346 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.461308 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f075da-0542-4e75-9b04-ae4c5c3879e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f075da-0542-4e75-9b04-ae4c5c3879e1-plugins-config-volume\") pod \"73f075da-0542-4e75-9b04-ae4c5c3879e1\" (UID: \"73f075da-0542-4e75-9b04-ae4c5c3879e1\") " Nov 02 18:10:35.461412 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.461392 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f075da-0542-4e75-9b04-ae4c5c3879e1-routing-overlay-volume" pod "73f075da-0542-4e75-9b04-ae4c5c3879e1" (UID: "73f075da-0542-4e75-9b04-ae4c5c3879e1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:35.461696 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.461670 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f075da-0542-4e75-9b04-ae4c5c3879e1-plugins-config-volume" pod "73f075da-0542-4e75-9b04-ae4c5c3879e1" (UID: "73f075da-0542-4e75-9b04-ae4c5c3879e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:35.462061 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.462041 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73f075da-0542-4e75-9b04-ae4c5c3879e1-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:35.462061 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.462060 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f075da-0542-4e75-9b04-ae4c5c3879e1-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:35.463329 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.463302 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f075da-0542-4e75-9b04-ae4c5c3879e1-kube-api-access-7m7cc" pod "73f075da-0542-4e75-9b04-ae4c5c3879e1" (UID: "73f075da-0542-4e75-9b04-ae4c5c3879e1"). InnerVolumeSpecName "kube-api-access-7m7cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:35.463449 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.463407 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f075da-0542-4e75-9b04-ae4c5c3879e1-provider-credentials" pod "73f075da-0542-4e75-9b04-ae4c5c3879e1" (UID: "73f075da-0542-4e75-9b04-ae4c5c3879e1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:35.575917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.575160 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/73f075da-0542-4e75-9b04-ae4c5c3879e1-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:35.575917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.575197 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m7cc\" (UniqueName: \"kubernetes.io/projected/73f075da-0542-4e75-9b04-ae4c5c3879e1-kube-api-access-7m7cc\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:35.866674 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.866578 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:35.985448 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.976997 5549 generic.go:356] "Generic (PLEG): container finished" podID="73f075da-0542-4e75-9b04-ae4c5c3879e1" containerID="e999eae5d6f042dde19fef815b0c307800d4b0fb65b1c8b018ad910b5068c3a0" exitCode=0 Nov 02 18:10:35.985448 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.977117 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" event={"ID":"73f075da-0542-4e75-9b04-ae4c5c3879e1","Type":"ContainerDied","Data":"e999eae5d6f042dde19fef815b0c307800d4b0fb65b1c8b018ad910b5068c3a0"} Nov 02 18:10:35.985448 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.977209 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" event={"ID":"73f075da-0542-4e75-9b04-ae4c5c3879e1","Type":"ContainerDied","Data":"77fb73e980cf62ae084ef70141b1fb214d113999c8a77fa88b3703bd9c7afb5f"} Nov 02 18:10:35.985448 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.977240 5549 scope.go:122] "RemoveContainer" containerID="e999eae5d6f042dde19fef815b0c307800d4b0fb65b1c8b018ad910b5068c3a0" Nov 02 18:10:35.985448 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:35.977383 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-769b787cbf-h9x98" Nov 02 18:10:36.007885 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:36.007861 5549 scope.go:122] "RemoveContainer" containerID="e999eae5d6f042dde19fef815b0c307800d4b0fb65b1c8b018ad910b5068c3a0" Nov 02 18:10:36.008214 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:36.008186 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e999eae5d6f042dde19fef815b0c307800d4b0fb65b1c8b018ad910b5068c3a0\": container with ID starting with e999eae5d6f042dde19fef815b0c307800d4b0fb65b1c8b018ad910b5068c3a0 not found: ID does not exist" containerID="e999eae5d6f042dde19fef815b0c307800d4b0fb65b1c8b018ad910b5068c3a0" Nov 02 18:10:36.008285 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:36.008229 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e999eae5d6f042dde19fef815b0c307800d4b0fb65b1c8b018ad910b5068c3a0"} err="failed to get container status \"e999eae5d6f042dde19fef815b0c307800d4b0fb65b1c8b018ad910b5068c3a0\": rpc error: code = NotFound desc = could not find container \"e999eae5d6f042dde19fef815b0c307800d4b0fb65b1c8b018ad910b5068c3a0\": container with ID starting with e999eae5d6f042dde19fef815b0c307800d4b0fb65b1c8b018ad910b5068c3a0 not found: ID does not exist" Nov 02 18:10:36.022110 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:36.022068 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-769b787cbf-h9x98"] Nov 02 18:10:36.027738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:36.027701 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-769b787cbf-h9x98"] Nov 02 18:10:36.335406 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:36.335368 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f075da-0542-4e75-9b04-ae4c5c3879e1" path="/var/lib/kubelet/pods/73f075da-0542-4e75-9b04-ae4c5c3879e1/volumes" Nov 02 18:10:36.933975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:36.933511 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:37.826567 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:37.826529 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:38.987977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:38.987788 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:39.546712 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:39.546675 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:39.692638 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:39.692596 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:39.908316 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:39.908224 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:40.342710 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:40.342678 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:40.500937 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:40.500894 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:41.166454 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.166418 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb"] Nov 02 18:10:41.167574 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.167549 5549 cpu_manager.go:414] "containerMap: removing container" podUID="2f06f984-c329-4bfc-9be0-5dfb2d44f597" containerName="payload-processing" Nov 02 18:10:41.167574 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.167570 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f06f984-c329-4bfc-9be0-5dfb2d44f597" containerName="payload-processing" Nov 02 18:10:41.167737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.167608 5549 cpu_manager.go:414] "containerMap: removing container" podUID="2e24f2e6-b8a7-4a75-acdb-de49d3796804" containerName="payload-processing" Nov 02 18:10:41.167737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.167614 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e24f2e6-b8a7-4a75-acdb-de49d3796804" containerName="payload-processing" Nov 02 18:10:41.167737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.167630 5549 cpu_manager.go:414] "containerMap: removing container" podUID="73f075da-0542-4e75-9b04-ae4c5c3879e1" containerName="payload-processing" Nov 02 18:10:41.167737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.167636 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f075da-0542-4e75-9b04-ae4c5c3879e1" containerName="payload-processing" Nov 02 18:10:41.167895 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.167807 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f06f984-c329-4bfc-9be0-5dfb2d44f597" containerName="payload-processing" Nov 02 18:10:41.167895 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.167819 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e24f2e6-b8a7-4a75-acdb-de49d3796804" containerName="payload-processing" Nov 02 18:10:41.167895 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.167828 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f075da-0542-4e75-9b04-ae4c5c3879e1" containerName="payload-processing" Nov 02 18:10:41.173871 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.173849 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" Nov 02 18:10:41.177549 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.177521 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb"] Nov 02 18:10:41.177745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.177728 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1182c71c63db\"" Nov 02 18:10:41.299767 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.299720 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ttfx\" (UniqueName: \"kubernetes.io/projected/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-kube-api-access-5ttfx\") pod \"payload-processing-external-model-d58b5b6b9-n6bqb\" (UID: \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\") " pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" Nov 02 18:10:41.300036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.299794 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-provider-credentials\") pod \"payload-processing-external-model-d58b5b6b9-n6bqb\" (UID: \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\") " pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" Nov 02 18:10:41.300036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.300008 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-routing-overlay-volume\") pod \"payload-processing-external-model-d58b5b6b9-n6bqb\" (UID: \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\") " pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" Nov 02 18:10:41.300178 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.300162 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-plugins-config-volume\") pod \"payload-processing-external-model-d58b5b6b9-n6bqb\" (UID: \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\") " pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" Nov 02 18:10:41.401453 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.401411 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-routing-overlay-volume\") pod \"payload-processing-external-model-d58b5b6b9-n6bqb\" (UID: \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\") " pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" Nov 02 18:10:41.401875 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.401537 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-plugins-config-volume\") pod \"payload-processing-external-model-d58b5b6b9-n6bqb\" (UID: \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\") " pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" Nov 02 18:10:41.401875 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.401747 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ttfx\" (UniqueName: \"kubernetes.io/projected/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-kube-api-access-5ttfx\") pod \"payload-processing-external-model-d58b5b6b9-n6bqb\" (UID: \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\") " pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" Nov 02 18:10:41.401875 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.401816 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-provider-credentials\") pod \"payload-processing-external-model-d58b5b6b9-n6bqb\" (UID: \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\") " pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" Nov 02 18:10:41.402230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.402207 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-routing-overlay-volume\") pod \"payload-processing-external-model-d58b5b6b9-n6bqb\" (UID: \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\") " pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" Nov 02 18:10:41.402382 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.402357 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-plugins-config-volume\") pod \"payload-processing-external-model-d58b5b6b9-n6bqb\" (UID: \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\") " pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" Nov 02 18:10:41.404583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.404558 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-provider-credentials\") pod \"payload-processing-external-model-d58b5b6b9-n6bqb\" (UID: \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\") " pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" Nov 02 18:10:41.410701 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.410672 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ttfx\" (UniqueName: \"kubernetes.io/projected/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-kube-api-access-5ttfx\") pod \"payload-processing-external-model-d58b5b6b9-n6bqb\" (UID: \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\") " pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" Nov 02 18:10:41.494383 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.494350 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" Nov 02 18:10:41.545963 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.545904 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:41.707986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.698530 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:41.845340 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.843051 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb"] Nov 02 18:10:41.853760 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:10:41.853708 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c861c9d_91d1_40a9_8acc_2d1f45f8aecc.slice/crio-c68d5e3ff759d3f7bc6386b51281a79abe909d14aa1861c16a7f753fcf589d8b WatchSource:0}: Error finding container c68d5e3ff759d3f7bc6386b51281a79abe909d14aa1861c16a7f753fcf589d8b: Status 404 returned error can't find the container with id c68d5e3ff759d3f7bc6386b51281a79abe909d14aa1861c16a7f753fcf589d8b Nov 02 18:10:41.924828 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:41.924793 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb"] Nov 02 18:10:42.219974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:42.215408 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" event={"ID":"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc","Type":"ContainerStarted","Data":"3c33f0ddb05813c6d0b02cb2a875792618efb02629f063013a6ed4a45b031b31"} Nov 02 18:10:42.219974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:42.215481 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" event={"ID":"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc","Type":"ContainerStarted","Data":"c68d5e3ff759d3f7bc6386b51281a79abe909d14aa1861c16a7f753fcf589d8b"} Nov 02 18:10:42.219974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:42.215694 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" Nov 02 18:10:42.304228 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:42.304167 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" podStartSLOduration=1.30414531 podStartE2EDuration="1.30414531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:42.248357378 +0000 UTC m=+2214.036862906" watchObservedRunningTime="2026-11-02 18:10:42.30414531 +0000 UTC m=+2214.092650839" Nov 02 18:10:42.568025 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:42.567145 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:42.699945 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:42.699909 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:42.981189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:42.981155 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:43.141200 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:43.134027 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:43.915977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:43.910514 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:44.300609 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:44.300576 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:44.455380 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:44.455349 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:44.656199 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:44.655645 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:44.805152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:44.805116 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:44.942277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:44.942192 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:45.437298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:45.437265 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:45.607997 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:45.596993 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:46.104658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.104622 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz"] Nov 02 18:10:46.110027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.110005 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" Nov 02 18:10:46.117863 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.117836 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"payload-processing-external-model-plugins-e2e-extmod-a-42d786\"" Nov 02 18:10:46.118231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.118148 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"routing-overlay\"" Nov 02 18:10:46.118997 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.118973 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"kube-root-ca.crt\"" Nov 02 18:10:46.119277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.119241 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"payload-processing-external-model-e2e--cd90e2f4-dockercfg-rbcvt\"" Nov 02 18:10:46.119391 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.119311 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"openshift-service-ca.crt\"" Nov 02 18:10:46.119391 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.119351 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"agc-cred-30cc9e28e9a9\"" Nov 02 18:10:46.120281 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.120258 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz"] Nov 02 18:10:46.139506 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.139461 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:46.214919 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.214877 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ccd2ae1-67d1-495c-af49-b164aecb0d04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-586d595wz\" (UID: \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" Nov 02 18:10:46.215152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.215060 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ccd2ae1-67d1-495c-af49-b164aecb0d04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-586d595wz\" (UID: \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" Nov 02 18:10:46.215152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.215135 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qckk7\" (UniqueName: \"kubernetes.io/projected/6ccd2ae1-67d1-495c-af49-b164aecb0d04-kube-api-access-qckk7\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-586d595wz\" (UID: \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" Nov 02 18:10:46.215350 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.215331 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ccd2ae1-67d1-495c-af49-b164aecb0d04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-586d595wz\" (UID: \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" Nov 02 18:10:46.398434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.320327 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ccd2ae1-67d1-495c-af49-b164aecb0d04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-586d595wz\" (UID: \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" Nov 02 18:10:46.398434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.320534 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ccd2ae1-67d1-495c-af49-b164aecb0d04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-586d595wz\" (UID: \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" Nov 02 18:10:46.398434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.320638 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ccd2ae1-67d1-495c-af49-b164aecb0d04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-586d595wz\" (UID: \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" Nov 02 18:10:46.398434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.320698 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qckk7\" (UniqueName: \"kubernetes.io/projected/6ccd2ae1-67d1-495c-af49-b164aecb0d04-kube-api-access-qckk7\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-586d595wz\" (UID: \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" Nov 02 18:10:46.398434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.333584 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ccd2ae1-67d1-495c-af49-b164aecb0d04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-586d595wz\" (UID: \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" Nov 02 18:10:46.398434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.334501 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ccd2ae1-67d1-495c-af49-b164aecb0d04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-586d595wz\" (UID: \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" Nov 02 18:10:46.433207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.399352 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ccd2ae1-67d1-495c-af49-b164aecb0d04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-586d595wz\" (UID: \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" Nov 02 18:10:46.433207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.412916 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qckk7\" (UniqueName: \"kubernetes.io/projected/6ccd2ae1-67d1-495c-af49-b164aecb0d04-kube-api-access-qckk7\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-586d595wz\" (UID: \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" Nov 02 18:10:46.434009 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.433712 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" Nov 02 18:10:46.880389 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:10:46.880357 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ccd2ae1_67d1_495c_af49_b164aecb0d04.slice/crio-20341c9284b0972f02c6178093ea92154db5605c8501675ccf82b79c259b4d0c WatchSource:0}: Error finding container 20341c9284b0972f02c6178093ea92154db5605c8501675ccf82b79c259b4d0c: Status 404 returned error can't find the container with id 20341c9284b0972f02c6178093ea92154db5605c8501675ccf82b79c259b4d0c Nov 02 18:10:46.880703 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.880677 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz"] Nov 02 18:10:46.933029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:46.932237 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:47.216308 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.216269 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" podUID="3c861c9d-91d1-40a9-8acc-2d1f45f8aecc" containerName="payload-processing" containerID="cri-o://3c33f0ddb05813c6d0b02cb2a875792618efb02629f063013a6ed4a45b031b31" gracePeriod=25 Nov 02 18:10:47.478534 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.478431 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" event={"ID":"6ccd2ae1-67d1-495c-af49-b164aecb0d04","Type":"ContainerStarted","Data":"d93cf380bd88d35b5b4ebfac0428f5e98b6f368ea77214898bdf5b3fd352db60"} Nov 02 18:10:47.478534 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.478512 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" event={"ID":"6ccd2ae1-67d1-495c-af49-b164aecb0d04","Type":"ContainerStarted","Data":"20341c9284b0972f02c6178093ea92154db5605c8501675ccf82b79c259b4d0c"} Nov 02 18:10:47.478822 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.478800 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" Nov 02 18:10:47.481618 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.481582 5549 generic.go:356] "Generic (PLEG): container finished" podID="3c861c9d-91d1-40a9-8acc-2d1f45f8aecc" containerID="3c33f0ddb05813c6d0b02cb2a875792618efb02629f063013a6ed4a45b031b31" exitCode=0 Nov 02 18:10:47.481757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.481684 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" event={"ID":"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc","Type":"ContainerDied","Data":"3c33f0ddb05813c6d0b02cb2a875792618efb02629f063013a6ed4a45b031b31"} Nov 02 18:10:47.499801 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.499743 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" podStartSLOduration=1.49972526 podStartE2EDuration="1.49972526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:47.498667818 +0000 UTC m=+2219.287173345" watchObservedRunningTime="2026-11-02 18:10:47.49972526 +0000 UTC m=+2219.288230789" Nov 02 18:10:47.532733 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.532710 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" Nov 02 18:10:47.598944 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:10:47.598904 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:10:47.634651 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.634619 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-plugins-config-volume\") pod \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\" (UID: \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\") " Nov 02 18:10:47.634850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.634676 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-routing-overlay-volume\") pod \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\" (UID: \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\") " Nov 02 18:10:47.634850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.634730 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-kube-api-access-5ttfx\" (UniqueName: \"kubernetes.io/projected/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-kube-api-access-5ttfx\") pod \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\" (UID: \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\") " Nov 02 18:10:47.634850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.634827 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-provider-credentials\") pod \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\" (UID: \"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc\") " Nov 02 18:10:47.635229 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.635197 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-plugins-config-volume" pod "3c861c9d-91d1-40a9-8acc-2d1f45f8aecc" (UID: "3c861c9d-91d1-40a9-8acc-2d1f45f8aecc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:47.635444 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.635419 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-routing-overlay-volume" pod "3c861c9d-91d1-40a9-8acc-2d1f45f8aecc" (UID: "3c861c9d-91d1-40a9-8acc-2d1f45f8aecc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:47.635918 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.635892 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:47.636071 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.635923 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:47.637365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.637328 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-kube-api-access-5ttfx" pod "3c861c9d-91d1-40a9-8acc-2d1f45f8aecc" (UID: "3c861c9d-91d1-40a9-8acc-2d1f45f8aecc"). InnerVolumeSpecName "kube-api-access-5ttfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:47.637652 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.637627 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-provider-credentials" pod "3c861c9d-91d1-40a9-8acc-2d1f45f8aecc" (UID: "3c861c9d-91d1-40a9-8acc-2d1f45f8aecc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:47.719501 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.719365 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:47.755068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.752044 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ttfx\" (UniqueName: \"kubernetes.io/projected/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-kube-api-access-5ttfx\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:47.755068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:47.752084 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:48.518975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:48.515386 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:48.535123 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:48.535087 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" Nov 02 18:10:48.535363 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:48.535178 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb" event={"ID":"3c861c9d-91d1-40a9-8acc-2d1f45f8aecc","Type":"ContainerDied","Data":"c68d5e3ff759d3f7bc6386b51281a79abe909d14aa1861c16a7f753fcf589d8b"} Nov 02 18:10:48.535363 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:48.535246 5549 scope.go:122] "RemoveContainer" containerID="3c33f0ddb05813c6d0b02cb2a875792618efb02629f063013a6ed4a45b031b31" Nov 02 18:10:48.558426 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:48.558400 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb"] Nov 02 18:10:48.562654 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:48.562629 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-d58b5b6b9-n6bqb"] Nov 02 18:10:49.141117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:49.139998 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:49.833985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:49.833102 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:50.323667 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:50.323634 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c861c9d-91d1-40a9-8acc-2d1f45f8aecc" path="/var/lib/kubelet/pods/3c861c9d-91d1-40a9-8acc-2d1f45f8aecc/volumes" Nov 02 18:10:50.531974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:50.531850 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:51.219976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:51.213920 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:52.039813 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.037625 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:52.394749 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.394716 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:52.544420 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.544327 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:52.669174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.669139 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-75456797bd-d542n"] Nov 02 18:10:52.670143 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.670124 5549 cpu_manager.go:414] "containerMap: removing container" podUID="3c861c9d-91d1-40a9-8acc-2d1f45f8aecc" containerName="payload-processing" Nov 02 18:10:52.670143 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.670143 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c861c9d-91d1-40a9-8acc-2d1f45f8aecc" containerName="payload-processing" Nov 02 18:10:52.670334 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.670325 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c861c9d-91d1-40a9-8acc-2d1f45f8aecc" containerName="payload-processing" Nov 02 18:10:52.676425 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.676402 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" Nov 02 18:10:52.682129 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.682101 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75456797bd-d542n"] Nov 02 18:10:52.780960 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.780911 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51433153-d037-4839-b460-d447ac7f301a-routing-overlay-volume\") pod \"payload-processing-external-model-75456797bd-d542n\" (UID: \"51433153-d037-4839-b460-d447ac7f301a\") " pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" Nov 02 18:10:52.781135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.781038 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51433153-d037-4839-b460-d447ac7f301a-plugins-config-volume\") pod \"payload-processing-external-model-75456797bd-d542n\" (UID: \"51433153-d037-4839-b460-d447ac7f301a\") " pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" Nov 02 18:10:52.781239 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.781213 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8gr6\" (UniqueName: \"kubernetes.io/projected/51433153-d037-4839-b460-d447ac7f301a-kube-api-access-f8gr6\") pod \"payload-processing-external-model-75456797bd-d542n\" (UID: \"51433153-d037-4839-b460-d447ac7f301a\") " pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" Nov 02 18:10:52.781392 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.781374 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51433153-d037-4839-b460-d447ac7f301a-provider-credentials\") pod \"payload-processing-external-model-75456797bd-d542n\" (UID: \"51433153-d037-4839-b460-d447ac7f301a\") " pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" Nov 02 18:10:52.887050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.886256 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51433153-d037-4839-b460-d447ac7f301a-provider-credentials\") pod \"payload-processing-external-model-75456797bd-d542n\" (UID: \"51433153-d037-4839-b460-d447ac7f301a\") " pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" Nov 02 18:10:52.887050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.886493 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51433153-d037-4839-b460-d447ac7f301a-routing-overlay-volume\") pod \"payload-processing-external-model-75456797bd-d542n\" (UID: \"51433153-d037-4839-b460-d447ac7f301a\") " pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" Nov 02 18:10:52.887050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.886565 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51433153-d037-4839-b460-d447ac7f301a-plugins-config-volume\") pod \"payload-processing-external-model-75456797bd-d542n\" (UID: \"51433153-d037-4839-b460-d447ac7f301a\") " pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" Nov 02 18:10:52.887050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.886690 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8gr6\" (UniqueName: \"kubernetes.io/projected/51433153-d037-4839-b460-d447ac7f301a-kube-api-access-f8gr6\") pod \"payload-processing-external-model-75456797bd-d542n\" (UID: \"51433153-d037-4839-b460-d447ac7f301a\") " pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" Nov 02 18:10:52.913507 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.898813 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8gr6\" (UniqueName: \"kubernetes.io/projected/51433153-d037-4839-b460-d447ac7f301a-kube-api-access-f8gr6\") pod \"payload-processing-external-model-75456797bd-d542n\" (UID: \"51433153-d037-4839-b460-d447ac7f301a\") " pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" Nov 02 18:10:52.913507 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.902090 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51433153-d037-4839-b460-d447ac7f301a-plugins-config-volume\") pod \"payload-processing-external-model-75456797bd-d542n\" (UID: \"51433153-d037-4839-b460-d447ac7f301a\") " pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" Nov 02 18:10:52.913507 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.902831 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51433153-d037-4839-b460-d447ac7f301a-routing-overlay-volume\") pod \"payload-processing-external-model-75456797bd-d542n\" (UID: \"51433153-d037-4839-b460-d447ac7f301a\") " pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" Nov 02 18:10:52.913507 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:52.912653 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51433153-d037-4839-b460-d447ac7f301a-provider-credentials\") pod \"payload-processing-external-model-75456797bd-d542n\" (UID: \"51433153-d037-4839-b460-d447ac7f301a\") " pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" Nov 02 18:10:53.001990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:53.001012 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" Nov 02 18:10:53.438171 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:53.438138 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75456797bd-d542n"] Nov 02 18:10:53.438641 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:10:53.438325 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51433153_d037_4839_b460_d447ac7f301a.slice/crio-7c2af09426b187a099acba0963317c71b85e0d2193c2e2487ece2ab927560c9c WatchSource:0}: Error finding container 7c2af09426b187a099acba0963317c71b85e0d2193c2e2487ece2ab927560c9c: Status 404 returned error can't find the container with id 7c2af09426b187a099acba0963317c71b85e0d2193c2e2487ece2ab927560c9c Nov 02 18:10:53.478804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:53.478776 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:53.632504 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:53.632464 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" event={"ID":"51433153-d037-4839-b460-d447ac7f301a","Type":"ContainerStarted","Data":"3640a97b9995ac1cf0f16437127d71cabeaf2c7201df6f6ad8ad3f7ac94f182a"} Nov 02 18:10:53.632698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:53.632526 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" event={"ID":"51433153-d037-4839-b460-d447ac7f301a","Type":"ContainerStarted","Data":"7c2af09426b187a099acba0963317c71b85e0d2193c2e2487ece2ab927560c9c"} Nov 02 18:10:53.632698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:53.632552 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" Nov 02 18:10:53.652187 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:53.652137 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" podStartSLOduration=1.652122992 podStartE2EDuration="1.652122992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:53.649377446 +0000 UTC m=+2225.437882981" watchObservedRunningTime="2026-11-02 18:10:53.652122992 +0000 UTC m=+2225.440628559" Nov 02 18:10:53.947875 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:53.947838 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:54.092483 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:54.092431 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:54.250559 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:54.244575 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:54.479462 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:54.479422 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:54.623441 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:54.623324 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:54.782149 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:54.780025 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:54.850991 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:54.847553 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl"] Nov 02 18:10:54.912495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:54.891778 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" Nov 02 18:10:55.017571 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:54.997276 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl"] Nov 02 18:10:55.118243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:55.117720 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75ca4e23-5370-4d23-b63e-58188a74deec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl\" (UID: \"75ca4e23-5370-4d23-b63e-58188a74deec\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" Nov 02 18:10:55.118243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:55.117887 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht5fm\" (UniqueName: \"kubernetes.io/projected/75ca4e23-5370-4d23-b63e-58188a74deec-kube-api-access-ht5fm\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl\" (UID: \"75ca4e23-5370-4d23-b63e-58188a74deec\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" Nov 02 18:10:55.118398 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:55.118307 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ca4e23-5370-4d23-b63e-58188a74deec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl\" (UID: \"75ca4e23-5370-4d23-b63e-58188a74deec\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" Nov 02 18:10:55.118398 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:55.118379 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75ca4e23-5370-4d23-b63e-58188a74deec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl\" (UID: \"75ca4e23-5370-4d23-b63e-58188a74deec\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" Nov 02 18:10:55.223005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:55.222962 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ca4e23-5370-4d23-b63e-58188a74deec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl\" (UID: \"75ca4e23-5370-4d23-b63e-58188a74deec\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" Nov 02 18:10:55.223214 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:55.223027 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75ca4e23-5370-4d23-b63e-58188a74deec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl\" (UID: \"75ca4e23-5370-4d23-b63e-58188a74deec\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" Nov 02 18:10:55.223214 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:55.223178 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75ca4e23-5370-4d23-b63e-58188a74deec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl\" (UID: \"75ca4e23-5370-4d23-b63e-58188a74deec\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" Nov 02 18:10:55.223355 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:55.223253 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht5fm\" (UniqueName: \"kubernetes.io/projected/75ca4e23-5370-4d23-b63e-58188a74deec-kube-api-access-ht5fm\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl\" (UID: \"75ca4e23-5370-4d23-b63e-58188a74deec\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" Nov 02 18:10:55.223970 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:55.223920 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75ca4e23-5370-4d23-b63e-58188a74deec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl\" (UID: \"75ca4e23-5370-4d23-b63e-58188a74deec\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" Nov 02 18:10:55.224104 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:55.223974 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ca4e23-5370-4d23-b63e-58188a74deec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl\" (UID: \"75ca4e23-5370-4d23-b63e-58188a74deec\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" Nov 02 18:10:55.226013 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:55.225991 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75ca4e23-5370-4d23-b63e-58188a74deec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl\" (UID: \"75ca4e23-5370-4d23-b63e-58188a74deec\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" Nov 02 18:10:55.232675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:55.232650 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht5fm\" (UniqueName: \"kubernetes.io/projected/75ca4e23-5370-4d23-b63e-58188a74deec-kube-api-access-ht5fm\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl\" (UID: \"75ca4e23-5370-4d23-b63e-58188a74deec\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" Nov 02 18:10:55.325197 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:55.325159 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:55.429809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:55.429279 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" Nov 02 18:10:55.486027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:55.480583 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:55.964923 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:10:55.964892 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ca4e23_5370_4d23_b63e_58188a74deec.slice/crio-d2f1d10123780521eb1bbf2422824c3e04699c7d1bb3382a4c10c4db9500cfc8 WatchSource:0}: Error finding container d2f1d10123780521eb1bbf2422824c3e04699c7d1bb3382a4c10c4db9500cfc8: Status 404 returned error can't find the container with id d2f1d10123780521eb1bbf2422824c3e04699c7d1bb3382a4c10c4db9500cfc8 Nov 02 18:10:55.969586 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:55.969558 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl"] Nov 02 18:10:56.035015 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.034983 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq"] Nov 02 18:10:56.040438 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.040419 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" Nov 02 18:10:56.047575 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.047543 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"routing-overlay\"" Nov 02 18:10:56.047702 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.047659 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"kube-root-ca.crt\"" Nov 02 18:10:56.047754 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.047720 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"openshift-service-ca.crt\"" Nov 02 18:10:56.047801 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.047781 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"payload-processing-external-model-plugins-e2e-extmod-a-807c08\"" Nov 02 18:10:56.050202 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.050178 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq"] Nov 02 18:10:56.052378 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.052352 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:56.104443 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.104407 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"agc-cred-41558d72238a\"" Nov 02 18:10:56.104638 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.104495 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"payload-processing-external-model-e2e--d2178d1f-dockercfg-8gtsz\"" Nov 02 18:10:56.197525 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.197470 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq\" (UID: \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" Nov 02 18:10:56.197707 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.197642 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq\" (UID: \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" Nov 02 18:10:56.197833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.197814 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq\" (UID: \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" Nov 02 18:10:56.197890 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.197875 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5r95\" (UniqueName: \"kubernetes.io/projected/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-kube-api-access-k5r95\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq\" (UID: \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" Nov 02 18:10:56.212501 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.212465 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:56.344024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.335546 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq\" (UID: \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" Nov 02 18:10:56.344024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.340067 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5r95\" (UniqueName: \"kubernetes.io/projected/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-kube-api-access-k5r95\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq\" (UID: \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" Nov 02 18:10:56.344024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.340219 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq\" (UID: \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" Nov 02 18:10:56.344024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.340329 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq\" (UID: \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" Nov 02 18:10:56.344024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.336601 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq\" (UID: \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" Nov 02 18:10:56.344024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.341525 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq\" (UID: \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" Nov 02 18:10:56.465974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.459601 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq\" (UID: \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" Nov 02 18:10:56.503136 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.488782 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5r95\" (UniqueName: \"kubernetes.io/projected/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-kube-api-access-k5r95\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq\" (UID: \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" Nov 02 18:10:56.512980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.510029 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:56.666375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.666277 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" Nov 02 18:10:56.795109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.790543 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" event={"ID":"75ca4e23-5370-4d23-b63e-58188a74deec","Type":"ContainerStarted","Data":"dcdff0a4a8d040dbb2bba369dedbe15f96200dbd9a9022375091c1f9bdf7f782"} Nov 02 18:10:56.795109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.790619 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" event={"ID":"75ca4e23-5370-4d23-b63e-58188a74deec","Type":"ContainerStarted","Data":"d2f1d10123780521eb1bbf2422824c3e04699c7d1bb3382a4c10c4db9500cfc8"} Nov 02 18:10:56.795109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:56.790992 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" Nov 02 18:10:57.148986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:57.145623 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:57.386811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:57.386763 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" podStartSLOduration=3.386746798 podStartE2EDuration="3.386746798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:56.852361131 +0000 UTC m=+2228.640866659" watchObservedRunningTime="2026-11-02 18:10:57.386746798 +0000 UTC m=+2229.175252323" Nov 02 18:10:57.390566 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:57.390534 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq"] Nov 02 18:10:57.390786 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:10:57.390764 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8444e0_7807_4238_b0a6_aa7d4e0bdde8.slice/crio-1c716ffd5fe0e81a3233ef4def82ca5905ff491a2cdf1321504435e4cd2d9350 WatchSource:0}: Error finding container 1c716ffd5fe0e81a3233ef4def82ca5905ff491a2cdf1321504435e4cd2d9350: Status 404 returned error can't find the container with id 1c716ffd5fe0e81a3233ef4def82ca5905ff491a2cdf1321504435e4cd2d9350 Nov 02 18:10:57.506973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:57.500360 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:57.802628 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:57.802571 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" event={"ID":"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8","Type":"ContainerStarted","Data":"336612faa1c7020d1f2507d09e5c7f5068a0e97d1fdcb9e32fa546382a23dbe4"} Nov 02 18:10:57.802825 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:57.802656 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" event={"ID":"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8","Type":"ContainerStarted","Data":"1c716ffd5fe0e81a3233ef4def82ca5905ff491a2cdf1321504435e4cd2d9350"} Nov 02 18:10:57.803154 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:57.803127 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" Nov 02 18:10:57.820660 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:57.820614 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" podStartSLOduration=1.8206020189999998 podStartE2EDuration="1.820602019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:57.819215231 +0000 UTC m=+2229.607720759" watchObservedRunningTime="2026-11-02 18:10:57.820602019 +0000 UTC m=+2229.609107546" Nov 02 18:10:58.278134 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:58.277332 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:58.538690 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:58.538600 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" Nov 02 18:10:59.033094 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:59.029123 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:59.762974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:10:59.759094 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:00.446005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:00.439136 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:01.125977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:01.118655 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:01.512342 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:01.512306 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:01.758038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:01.757999 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:01.933403 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:01.933318 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:02.084478 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:02.084447 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:03.524224 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:03.524178 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:04.226082 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:04.226038 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:04.645011 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:04.644893 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" Nov 02 18:11:04.700661 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:04.700616 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj"] Nov 02 18:11:04.989675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:04.989637 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:05.930869 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:05.930833 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:06.418693 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:06.418656 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl"] Nov 02 18:11:06.420230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:06.420200 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:06.423178 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:06.423151 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" Nov 02 18:11:06.424304 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:06.424279 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk"] Nov 02 18:11:06.556439 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:06.556403 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:07.076436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:07.076400 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:07.211365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:07.211335 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:07.922558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:07.922519 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:08.643125 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:08.643087 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:08.815046 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:08.815010 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" Nov 02 18:11:09.326575 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:09.326527 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:09.701937 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:09.701898 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" podUID="99704f7e-ab00-4874-a248-ddbbbc88a1b5" containerName="payload-processing" containerID="cri-o://24b906a834783a0015226dc3b8ff693145d801f9e620d6d94bcd4118efb37593" gracePeriod=25 Nov 02 18:11:10.043232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.036424 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" Nov 02 18:11:10.057834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.046439 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:10.088733 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.088692 5549 generic.go:356] "Generic (PLEG): container finished" podID="99704f7e-ab00-4874-a248-ddbbbc88a1b5" containerID="24b906a834783a0015226dc3b8ff693145d801f9e620d6d94bcd4118efb37593" exitCode=0 Nov 02 18:11:10.089036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.088786 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" event={"ID":"99704f7e-ab00-4874-a248-ddbbbc88a1b5","Type":"ContainerDied","Data":"24b906a834783a0015226dc3b8ff693145d801f9e620d6d94bcd4118efb37593"} Nov 02 18:11:10.089036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.088839 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" event={"ID":"99704f7e-ab00-4874-a248-ddbbbc88a1b5","Type":"ContainerDied","Data":"1e7c2f8addb7c7cb51ef1e26628b13ebbfd8ad69ee84604f8dd0230c86a48cec"} Nov 02 18:11:10.089036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.088873 5549 scope.go:122] "RemoveContainer" containerID="24b906a834783a0015226dc3b8ff693145d801f9e620d6d94bcd4118efb37593" Nov 02 18:11:10.089226 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.089067 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj" Nov 02 18:11:10.106653 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.106627 5549 scope.go:122] "RemoveContainer" containerID="24b906a834783a0015226dc3b8ff693145d801f9e620d6d94bcd4118efb37593" Nov 02 18:11:10.107121 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:11:10.107092 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24b906a834783a0015226dc3b8ff693145d801f9e620d6d94bcd4118efb37593\": container with ID starting with 24b906a834783a0015226dc3b8ff693145d801f9e620d6d94bcd4118efb37593 not found: ID does not exist" containerID="24b906a834783a0015226dc3b8ff693145d801f9e620d6d94bcd4118efb37593" Nov 02 18:11:10.107237 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.107130 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24b906a834783a0015226dc3b8ff693145d801f9e620d6d94bcd4118efb37593"} err="failed to get container status \"24b906a834783a0015226dc3b8ff693145d801f9e620d6d94bcd4118efb37593\": rpc error: code = NotFound desc = could not find container \"24b906a834783a0015226dc3b8ff693145d801f9e620d6d94bcd4118efb37593\": container with ID starting with 24b906a834783a0015226dc3b8ff693145d801f9e620d6d94bcd4118efb37593 not found: ID does not exist" Nov 02 18:11:10.161454 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.161423 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99704f7e-ab00-4874-a248-ddbbbc88a1b5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99704f7e-ab00-4874-a248-ddbbbc88a1b5-routing-overlay-volume\") pod \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\" (UID: \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\") " Nov 02 18:11:10.161591 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.161469 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99704f7e-ab00-4874-a248-ddbbbc88a1b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99704f7e-ab00-4874-a248-ddbbbc88a1b5-plugins-config-volume\") pod \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\" (UID: \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\") " Nov 02 18:11:10.161591 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.161497 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99704f7e-ab00-4874-a248-ddbbbc88a1b5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/99704f7e-ab00-4874-a248-ddbbbc88a1b5-provider-credentials\") pod \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\" (UID: \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\") " Nov 02 18:11:10.161591 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.161523 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99704f7e-ab00-4874-a248-ddbbbc88a1b5-kube-api-access-7t2fl\" (UniqueName: \"kubernetes.io/projected/99704f7e-ab00-4874-a248-ddbbbc88a1b5-kube-api-access-7t2fl\") pod \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\" (UID: \"99704f7e-ab00-4874-a248-ddbbbc88a1b5\") " Nov 02 18:11:10.161882 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.161855 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99704f7e-ab00-4874-a248-ddbbbc88a1b5-routing-overlay-volume" pod "99704f7e-ab00-4874-a248-ddbbbc88a1b5" (UID: "99704f7e-ab00-4874-a248-ddbbbc88a1b5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:10.162000 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.161975 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99704f7e-ab00-4874-a248-ddbbbc88a1b5-plugins-config-volume" pod "99704f7e-ab00-4874-a248-ddbbbc88a1b5" (UID: "99704f7e-ab00-4874-a248-ddbbbc88a1b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:10.162465 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.162442 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99704f7e-ab00-4874-a248-ddbbbc88a1b5-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:10.162580 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.162467 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99704f7e-ab00-4874-a248-ddbbbc88a1b5-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:10.163940 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.163913 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99704f7e-ab00-4874-a248-ddbbbc88a1b5-kube-api-access-7t2fl" pod "99704f7e-ab00-4874-a248-ddbbbc88a1b5" (UID: "99704f7e-ab00-4874-a248-ddbbbc88a1b5"). InnerVolumeSpecName "kube-api-access-7t2fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:10.164161 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.164139 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99704f7e-ab00-4874-a248-ddbbbc88a1b5-provider-credentials" pod "99704f7e-ab00-4874-a248-ddbbbc88a1b5" (UID: "99704f7e-ab00-4874-a248-ddbbbc88a1b5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:10.263936 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.263852 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99704f7e-ab00-4874-a248-ddbbbc88a1b5-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:10.263936 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.263885 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t2fl\" (UniqueName: \"kubernetes.io/projected/99704f7e-ab00-4874-a248-ddbbbc88a1b5-kube-api-access-7t2fl\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:10.409826 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.409788 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj"] Nov 02 18:11:10.414246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.414221 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6fc469f668-5rmnj"] Nov 02 18:11:10.831977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:10.831611 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:11.420037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.419999 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" podUID="75ca4e23-5370-4d23-b63e-58188a74deec" containerName="payload-processing" containerID="cri-o://dcdff0a4a8d040dbb2bba369dedbe15f96200dbd9a9022375091c1f9bdf7f782" gracePeriod=25 Nov 02 18:11:11.425544 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.425513 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" podUID="47bd69ca-a6f7-4399-a5b8-b3a27e7a3086" containerName="payload-processing" containerID="cri-o://03c6ff1a5d909e923397ea57adba0339e0ec37a7f8592b731931d40b975dd051" gracePeriod=25 Nov 02 18:11:11.534466 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.534418 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:11.775738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.775709 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" Nov 02 18:11:11.781271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.781244 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" Nov 02 18:11:11.903394 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.903364 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-plugins-config-volume\") pod \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\" (UID: \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\") " Nov 02 18:11:11.903917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.903515 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ca4e23-5370-4d23-b63e-58188a74deec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ca4e23-5370-4d23-b63e-58188a74deec-plugins-config-volume\") pod \"75ca4e23-5370-4d23-b63e-58188a74deec\" (UID: \"75ca4e23-5370-4d23-b63e-58188a74deec\") " Nov 02 18:11:11.903917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.903618 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-routing-overlay-volume\") pod \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\" (UID: \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\") " Nov 02 18:11:11.903917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.903671 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ca4e23-5370-4d23-b63e-58188a74deec-kube-api-access-ht5fm\" (UniqueName: \"kubernetes.io/projected/75ca4e23-5370-4d23-b63e-58188a74deec-kube-api-access-ht5fm\") pod \"75ca4e23-5370-4d23-b63e-58188a74deec\" (UID: \"75ca4e23-5370-4d23-b63e-58188a74deec\") " Nov 02 18:11:11.903917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.903759 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-kube-api-access-6dqpz\" (UniqueName: \"kubernetes.io/projected/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-kube-api-access-6dqpz\") pod \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\" (UID: \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\") " Nov 02 18:11:11.903917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.903788 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ca4e23-5370-4d23-b63e-58188a74deec-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75ca4e23-5370-4d23-b63e-58188a74deec-routing-overlay-volume\") pod \"75ca4e23-5370-4d23-b63e-58188a74deec\" (UID: \"75ca4e23-5370-4d23-b63e-58188a74deec\") " Nov 02 18:11:11.904264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.904000 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-provider-credentials\" (UniqueName: \"kubernetes.io/projected/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-provider-credentials\") pod \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\" (UID: \"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086\") " Nov 02 18:11:11.904264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.904006 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ca4e23-5370-4d23-b63e-58188a74deec-plugins-config-volume" pod "75ca4e23-5370-4d23-b63e-58188a74deec" (UID: "75ca4e23-5370-4d23-b63e-58188a74deec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:11.904264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.904037 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ca4e23-5370-4d23-b63e-58188a74deec-provider-credentials\" (UniqueName: \"kubernetes.io/projected/75ca4e23-5370-4d23-b63e-58188a74deec-provider-credentials\") pod \"75ca4e23-5370-4d23-b63e-58188a74deec\" (UID: \"75ca4e23-5370-4d23-b63e-58188a74deec\") " Nov 02 18:11:11.904264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.904045 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-plugins-config-volume" pod "47bd69ca-a6f7-4399-a5b8-b3a27e7a3086" (UID: "47bd69ca-a6f7-4399-a5b8-b3a27e7a3086"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:11.904264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.904053 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-routing-overlay-volume" pod "47bd69ca-a6f7-4399-a5b8-b3a27e7a3086" (UID: "47bd69ca-a6f7-4399-a5b8-b3a27e7a3086"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:11.904541 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.904362 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ca4e23-5370-4d23-b63e-58188a74deec-routing-overlay-volume" pod "75ca4e23-5370-4d23-b63e-58188a74deec" (UID: "75ca4e23-5370-4d23-b63e-58188a74deec"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:11.904905 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.904887 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:11.905012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.904912 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ca4e23-5370-4d23-b63e-58188a74deec-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:11.905012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.904927 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:11.905012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.904943 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75ca4e23-5370-4d23-b63e-58188a74deec-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:11.906605 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.906573 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ca4e23-5370-4d23-b63e-58188a74deec-kube-api-access-ht5fm" pod "75ca4e23-5370-4d23-b63e-58188a74deec" (UID: "75ca4e23-5370-4d23-b63e-58188a74deec"). InnerVolumeSpecName "kube-api-access-ht5fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:11.906720 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.906619 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ca4e23-5370-4d23-b63e-58188a74deec-provider-credentials" pod "75ca4e23-5370-4d23-b63e-58188a74deec" (UID: "75ca4e23-5370-4d23-b63e-58188a74deec"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:11.906720 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.906676 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-provider-credentials" pod "47bd69ca-a6f7-4399-a5b8-b3a27e7a3086" (UID: "47bd69ca-a6f7-4399-a5b8-b3a27e7a3086"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:11.906720 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:11.906709 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-kube-api-access-6dqpz" pod "47bd69ca-a6f7-4399-a5b8-b3a27e7a3086" (UID: "47bd69ca-a6f7-4399-a5b8-b3a27e7a3086"). InnerVolumeSpecName "kube-api-access-6dqpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:12.006695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.006595 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:12.006695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.006634 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75ca4e23-5370-4d23-b63e-58188a74deec-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:12.006695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.006647 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht5fm\" (UniqueName: \"kubernetes.io/projected/75ca4e23-5370-4d23-b63e-58188a74deec-kube-api-access-ht5fm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:12.006695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.006660 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dqpz\" (UniqueName: \"kubernetes.io/projected/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086-kube-api-access-6dqpz\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:12.128168 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.127876 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:12.218700 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.218656 5549 generic.go:356] "Generic (PLEG): container finished" podID="47bd69ca-a6f7-4399-a5b8-b3a27e7a3086" containerID="03c6ff1a5d909e923397ea57adba0339e0ec37a7f8592b731931d40b975dd051" exitCode=0 Nov 02 18:11:12.218923 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.218712 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" Nov 02 18:11:12.218923 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.218734 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" event={"ID":"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086","Type":"ContainerDied","Data":"03c6ff1a5d909e923397ea57adba0339e0ec37a7f8592b731931d40b975dd051"} Nov 02 18:11:12.218923 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.218870 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk" event={"ID":"47bd69ca-a6f7-4399-a5b8-b3a27e7a3086","Type":"ContainerDied","Data":"e08167d41665fa0f63a4cb1d7dd15808623cce07516f99a0ccdb7e634849c359"} Nov 02 18:11:12.218923 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.218912 5549 scope.go:122] "RemoveContainer" containerID="03c6ff1a5d909e923397ea57adba0339e0ec37a7f8592b731931d40b975dd051" Nov 02 18:11:12.220703 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.220673 5549 generic.go:356] "Generic (PLEG): container finished" podID="75ca4e23-5370-4d23-b63e-58188a74deec" containerID="dcdff0a4a8d040dbb2bba369dedbe15f96200dbd9a9022375091c1f9bdf7f782" exitCode=0 Nov 02 18:11:12.220835 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.220771 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" event={"ID":"75ca4e23-5370-4d23-b63e-58188a74deec","Type":"ContainerDied","Data":"dcdff0a4a8d040dbb2bba369dedbe15f96200dbd9a9022375091c1f9bdf7f782"} Nov 02 18:11:12.220835 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.220814 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" event={"ID":"75ca4e23-5370-4d23-b63e-58188a74deec","Type":"ContainerDied","Data":"d2f1d10123780521eb1bbf2422824c3e04699c7d1bb3382a4c10c4db9500cfc8"} Nov 02 18:11:12.221007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.220896 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl" Nov 02 18:11:12.231721 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.231699 5549 scope.go:122] "RemoveContainer" containerID="03c6ff1a5d909e923397ea57adba0339e0ec37a7f8592b731931d40b975dd051" Nov 02 18:11:12.232080 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:11:12.232058 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c6ff1a5d909e923397ea57adba0339e0ec37a7f8592b731931d40b975dd051\": container with ID starting with 03c6ff1a5d909e923397ea57adba0339e0ec37a7f8592b731931d40b975dd051 not found: ID does not exist" containerID="03c6ff1a5d909e923397ea57adba0339e0ec37a7f8592b731931d40b975dd051" Nov 02 18:11:12.232169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.232092 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c6ff1a5d909e923397ea57adba0339e0ec37a7f8592b731931d40b975dd051"} err="failed to get container status \"03c6ff1a5d909e923397ea57adba0339e0ec37a7f8592b731931d40b975dd051\": rpc error: code = NotFound desc = could not find container \"03c6ff1a5d909e923397ea57adba0339e0ec37a7f8592b731931d40b975dd051\": container with ID starting with 03c6ff1a5d909e923397ea57adba0339e0ec37a7f8592b731931d40b975dd051 not found: ID does not exist" Nov 02 18:11:12.232169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.232116 5549 scope.go:122] "RemoveContainer" containerID="dcdff0a4a8d040dbb2bba369dedbe15f96200dbd9a9022375091c1f9bdf7f782" Nov 02 18:11:12.243643 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.243623 5549 scope.go:122] "RemoveContainer" containerID="dcdff0a4a8d040dbb2bba369dedbe15f96200dbd9a9022375091c1f9bdf7f782" Nov 02 18:11:12.243991 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:11:12.243964 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcdff0a4a8d040dbb2bba369dedbe15f96200dbd9a9022375091c1f9bdf7f782\": container with ID starting with dcdff0a4a8d040dbb2bba369dedbe15f96200dbd9a9022375091c1f9bdf7f782 not found: ID does not exist" containerID="dcdff0a4a8d040dbb2bba369dedbe15f96200dbd9a9022375091c1f9bdf7f782" Nov 02 18:11:12.244098 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.243998 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcdff0a4a8d040dbb2bba369dedbe15f96200dbd9a9022375091c1f9bdf7f782"} err="failed to get container status \"dcdff0a4a8d040dbb2bba369dedbe15f96200dbd9a9022375091c1f9bdf7f782\": rpc error: code = NotFound desc = could not find container \"dcdff0a4a8d040dbb2bba369dedbe15f96200dbd9a9022375091c1f9bdf7f782\": container with ID starting with dcdff0a4a8d040dbb2bba369dedbe15f96200dbd9a9022375091c1f9bdf7f782 not found: ID does not exist" Nov 02 18:11:12.248045 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.248012 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk"] Nov 02 18:11:12.253442 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.253420 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-66f9h2lzk"] Nov 02 18:11:12.261381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.261323 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl"] Nov 02 18:11:12.266822 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.266796 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-5d9fl9fgl"] Nov 02 18:11:12.323233 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.323197 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47bd69ca-a6f7-4399-a5b8-b3a27e7a3086" path="/var/lib/kubelet/pods/47bd69ca-a6f7-4399-a5b8-b3a27e7a3086/volumes" Nov 02 18:11:12.323721 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.323706 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ca4e23-5370-4d23-b63e-58188a74deec" path="/var/lib/kubelet/pods/75ca4e23-5370-4d23-b63e-58188a74deec/volumes" Nov 02 18:11:12.324173 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.324160 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99704f7e-ab00-4874-a248-ddbbbc88a1b5" path="/var/lib/kubelet/pods/99704f7e-ab00-4874-a248-ddbbbc88a1b5/volumes" Nov 02 18:11:12.843988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:12.838067 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:13.446603 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:13.446567 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:13.600837 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:13.600797 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:13.757544 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:13.757453 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:13.892205 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:13.892175 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:14.521708 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:14.521664 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:15.243981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:15.240688 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:15.968487 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:15.966886 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:16.424195 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:16.424105 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq"] Nov 02 18:11:16.624877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:16.624839 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:17.324569 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:17.324533 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:17.590035 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:11:17.589935 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:11:18.069988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:18.068464 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:18.717605 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:18.717515 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:19.313153 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:19.313119 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:19.449061 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:19.449027 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:19.616804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:19.616726 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:19.966701 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:19.966657 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:20.115362 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:20.115324 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:20.262985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:20.262874 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:21.424437 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:21.424401 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:21.424437 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:21.424401 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" podUID="0b8444e0-7807-4238-b0a6-aa7d4e0bdde8" containerName="payload-processing" containerID="cri-o://336612faa1c7020d1f2507d09e5c7f5068a0e97d1fdcb9e32fa546382a23dbe4" gracePeriod=25 Nov 02 18:11:21.558243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:21.558207 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:22.063319 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.063227 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" Nov 02 18:11:22.096925 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.096887 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:22.112657 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.112627 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-routing-overlay-volume\") pod \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\" (UID: \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\") " Nov 02 18:11:22.112834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.112699 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-provider-credentials\") pod \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\" (UID: \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\") " Nov 02 18:11:22.112834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.112807 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-kube-api-access-k5r95\" (UniqueName: \"kubernetes.io/projected/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-kube-api-access-k5r95\") pod \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\" (UID: \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\") " Nov 02 18:11:22.113013 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.112852 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-plugins-config-volume\") pod \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\" (UID: \"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8\") " Nov 02 18:11:22.113228 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.113125 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-routing-overlay-volume" pod "0b8444e0-7807-4238-b0a6-aa7d4e0bdde8" (UID: "0b8444e0-7807-4238-b0a6-aa7d4e0bdde8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:22.113371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.113307 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-plugins-config-volume" pod "0b8444e0-7807-4238-b0a6-aa7d4e0bdde8" (UID: "0b8444e0-7807-4238-b0a6-aa7d4e0bdde8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:22.113632 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.113614 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:22.113725 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.113635 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:22.115074 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.115044 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-provider-credentials" pod "0b8444e0-7807-4238-b0a6-aa7d4e0bdde8" (UID: "0b8444e0-7807-4238-b0a6-aa7d4e0bdde8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:22.115206 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.115185 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-kube-api-access-k5r95" pod "0b8444e0-7807-4238-b0a6-aa7d4e0bdde8" (UID: "0b8444e0-7807-4238-b0a6-aa7d4e0bdde8"). InnerVolumeSpecName "kube-api-access-k5r95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:22.167286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.167245 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb"] Nov 02 18:11:22.168346 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.168323 5549 cpu_manager.go:414] "containerMap: removing container" podUID="75ca4e23-5370-4d23-b63e-58188a74deec" containerName="payload-processing" Nov 02 18:11:22.168404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.168348 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ca4e23-5370-4d23-b63e-58188a74deec" containerName="payload-processing" Nov 02 18:11:22.168404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.168368 5549 cpu_manager.go:414] "containerMap: removing container" podUID="0b8444e0-7807-4238-b0a6-aa7d4e0bdde8" containerName="payload-processing" Nov 02 18:11:22.168404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.168373 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b8444e0-7807-4238-b0a6-aa7d4e0bdde8" containerName="payload-processing" Nov 02 18:11:22.168404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.168391 5549 cpu_manager.go:414] "containerMap: removing container" podUID="99704f7e-ab00-4874-a248-ddbbbc88a1b5" containerName="payload-processing" Nov 02 18:11:22.168404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.168396 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99704f7e-ab00-4874-a248-ddbbbc88a1b5" containerName="payload-processing" Nov 02 18:11:22.168404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.168406 5549 cpu_manager.go:414] "containerMap: removing container" podUID="47bd69ca-a6f7-4399-a5b8-b3a27e7a3086" containerName="payload-processing" Nov 02 18:11:22.168616 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.168411 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47bd69ca-a6f7-4399-a5b8-b3a27e7a3086" containerName="payload-processing" Nov 02 18:11:22.168616 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.168576 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ca4e23-5370-4d23-b63e-58188a74deec" containerName="payload-processing" Nov 02 18:11:22.168616 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.168585 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b8444e0-7807-4238-b0a6-aa7d4e0bdde8" containerName="payload-processing" Nov 02 18:11:22.168616 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.168594 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="99704f7e-ab00-4874-a248-ddbbbc88a1b5" containerName="payload-processing" Nov 02 18:11:22.168616 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.168602 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="47bd69ca-a6f7-4399-a5b8-b3a27e7a3086" containerName="payload-processing" Nov 02 18:11:22.172941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.172921 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" Nov 02 18:11:22.177838 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.177807 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"openshift-service-ca.crt\"" Nov 02 18:11:22.178023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.177937 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2db162\"" Nov 02 18:11:22.178137 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.178070 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"kube-root-ca.crt\"" Nov 02 18:11:22.178196 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.178155 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"payload-processing-external-model-e2e--96b2e561-dockercfg-v9qrb\"" Nov 02 18:11:22.178248 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.178200 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"routing-overlay\"" Nov 02 18:11:22.178248 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.178235 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"agc-cred-ff45ed7b3970\"" Nov 02 18:11:22.182678 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.182642 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb"] Nov 02 18:11:22.215475 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.215432 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0d94c9f-3e07-4fa5-8301-a80629738966-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb\" (UID: \"c0d94c9f-3e07-4fa5-8301-a80629738966\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" Nov 02 18:11:22.215643 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.215556 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d94c9f-3e07-4fa5-8301-a80629738966-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb\" (UID: \"c0d94c9f-3e07-4fa5-8301-a80629738966\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" Nov 02 18:11:22.215714 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.215641 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0d94c9f-3e07-4fa5-8301-a80629738966-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb\" (UID: \"c0d94c9f-3e07-4fa5-8301-a80629738966\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" Nov 02 18:11:22.215827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.215806 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lmz4\" (UniqueName: \"kubernetes.io/projected/c0d94c9f-3e07-4fa5-8301-a80629738966-kube-api-access-6lmz4\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb\" (UID: \"c0d94c9f-3e07-4fa5-8301-a80629738966\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" Nov 02 18:11:22.216158 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.216139 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:22.216247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.216160 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5r95\" (UniqueName: \"kubernetes.io/projected/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8-kube-api-access-k5r95\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:22.345031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.341259 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lmz4\" (UniqueName: \"kubernetes.io/projected/c0d94c9f-3e07-4fa5-8301-a80629738966-kube-api-access-6lmz4\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb\" (UID: \"c0d94c9f-3e07-4fa5-8301-a80629738966\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" Nov 02 18:11:22.345031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.341586 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0d94c9f-3e07-4fa5-8301-a80629738966-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb\" (UID: \"c0d94c9f-3e07-4fa5-8301-a80629738966\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" Nov 02 18:11:22.345031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.341641 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d94c9f-3e07-4fa5-8301-a80629738966-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb\" (UID: \"c0d94c9f-3e07-4fa5-8301-a80629738966\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" Nov 02 18:11:22.345031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.341722 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0d94c9f-3e07-4fa5-8301-a80629738966-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb\" (UID: \"c0d94c9f-3e07-4fa5-8301-a80629738966\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" Nov 02 18:11:22.345031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.344567 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d94c9f-3e07-4fa5-8301-a80629738966-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb\" (UID: \"c0d94c9f-3e07-4fa5-8301-a80629738966\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" Nov 02 18:11:22.360021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.345544 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0d94c9f-3e07-4fa5-8301-a80629738966-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb\" (UID: \"c0d94c9f-3e07-4fa5-8301-a80629738966\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" Nov 02 18:11:22.390843 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.390802 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lmz4\" (UniqueName: \"kubernetes.io/projected/c0d94c9f-3e07-4fa5-8301-a80629738966-kube-api-access-6lmz4\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb\" (UID: \"c0d94c9f-3e07-4fa5-8301-a80629738966\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" Nov 02 18:11:22.395107 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.395070 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0d94c9f-3e07-4fa5-8301-a80629738966-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb\" (UID: \"c0d94c9f-3e07-4fa5-8301-a80629738966\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" Nov 02 18:11:22.492314 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.492271 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:22.513046 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.513011 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" Nov 02 18:11:22.662996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.662237 5549 generic.go:356] "Generic (PLEG): container finished" podID="0b8444e0-7807-4238-b0a6-aa7d4e0bdde8" containerID="336612faa1c7020d1f2507d09e5c7f5068a0e97d1fdcb9e32fa546382a23dbe4" exitCode=0 Nov 02 18:11:22.662996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.662436 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" event={"ID":"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8","Type":"ContainerDied","Data":"336612faa1c7020d1f2507d09e5c7f5068a0e97d1fdcb9e32fa546382a23dbe4"} Nov 02 18:11:22.662996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.662511 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" event={"ID":"0b8444e0-7807-4238-b0a6-aa7d4e0bdde8","Type":"ContainerDied","Data":"1c716ffd5fe0e81a3233ef4def82ca5905ff491a2cdf1321504435e4cd2d9350"} Nov 02 18:11:22.662996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.662562 5549 scope.go:122] "RemoveContainer" containerID="336612faa1c7020d1f2507d09e5c7f5068a0e97d1fdcb9e32fa546382a23dbe4" Nov 02 18:11:22.662996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.662810 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq" Nov 02 18:11:22.691884 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.691858 5549 scope.go:122] "RemoveContainer" containerID="336612faa1c7020d1f2507d09e5c7f5068a0e97d1fdcb9e32fa546382a23dbe4" Nov 02 18:11:22.699640 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:11:22.699600 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"336612faa1c7020d1f2507d09e5c7f5068a0e97d1fdcb9e32fa546382a23dbe4\": container with ID starting with 336612faa1c7020d1f2507d09e5c7f5068a0e97d1fdcb9e32fa546382a23dbe4 not found: ID does not exist" containerID="336612faa1c7020d1f2507d09e5c7f5068a0e97d1fdcb9e32fa546382a23dbe4" Nov 02 18:11:22.699895 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.699863 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"336612faa1c7020d1f2507d09e5c7f5068a0e97d1fdcb9e32fa546382a23dbe4"} err="failed to get container status \"336612faa1c7020d1f2507d09e5c7f5068a0e97d1fdcb9e32fa546382a23dbe4\": rpc error: code = NotFound desc = could not find container \"336612faa1c7020d1f2507d09e5c7f5068a0e97d1fdcb9e32fa546382a23dbe4\": container with ID starting with 336612faa1c7020d1f2507d09e5c7f5068a0e97d1fdcb9e32fa546382a23dbe4 not found: ID does not exist" Nov 02 18:11:22.738568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.729942 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh"] Nov 02 18:11:22.818989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.807320 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh"] Nov 02 18:11:22.818989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.807448 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" Nov 02 18:11:22.818989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.813132 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"payload-processing-external-model-e2e--59b59218-dockercfg-lp4xp\"" Nov 02 18:11:22.830004 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.819445 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"agc-cred-0ebc5b098a14\"" Nov 02 18:11:22.884482 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.884434 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"openshift-service-ca.crt\"" Nov 02 18:11:22.924027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.920820 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"payload-processing-external-model-plugins-e2e-extmod-a-36a991\"" Nov 02 18:11:22.924027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.920982 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"routing-overlay\"" Nov 02 18:11:22.924027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.921177 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"kube-root-ca.crt\"" Nov 02 18:11:22.947583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.925870 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjglk\" (UniqueName: \"kubernetes.io/projected/1020655f-3c31-4bf1-ba61-aaca9e61a489-kube-api-access-zjglk\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh\" (UID: \"1020655f-3c31-4bf1-ba61-aaca9e61a489\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" Nov 02 18:11:22.947583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.925987 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1020655f-3c31-4bf1-ba61-aaca9e61a489-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh\" (UID: \"1020655f-3c31-4bf1-ba61-aaca9e61a489\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" Nov 02 18:11:22.947583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.926046 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1020655f-3c31-4bf1-ba61-aaca9e61a489-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh\" (UID: \"1020655f-3c31-4bf1-ba61-aaca9e61a489\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" Nov 02 18:11:22.947583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.926363 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1020655f-3c31-4bf1-ba61-aaca9e61a489-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh\" (UID: \"1020655f-3c31-4bf1-ba61-aaca9e61a489\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" Nov 02 18:11:23.006566 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:22.954393 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq"] Nov 02 18:11:23.140974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:23.140179 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjglk\" (UniqueName: \"kubernetes.io/projected/1020655f-3c31-4bf1-ba61-aaca9e61a489-kube-api-access-zjglk\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh\" (UID: \"1020655f-3c31-4bf1-ba61-aaca9e61a489\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" Nov 02 18:11:23.140974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:23.140382 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1020655f-3c31-4bf1-ba61-aaca9e61a489-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh\" (UID: \"1020655f-3c31-4bf1-ba61-aaca9e61a489\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" Nov 02 18:11:23.140974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:23.140445 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1020655f-3c31-4bf1-ba61-aaca9e61a489-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh\" (UID: \"1020655f-3c31-4bf1-ba61-aaca9e61a489\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" Nov 02 18:11:23.140974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:23.140800 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1020655f-3c31-4bf1-ba61-aaca9e61a489-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh\" (UID: \"1020655f-3c31-4bf1-ba61-aaca9e61a489\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" Nov 02 18:11:23.154002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:23.142834 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1020655f-3c31-4bf1-ba61-aaca9e61a489-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh\" (UID: \"1020655f-3c31-4bf1-ba61-aaca9e61a489\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" Nov 02 18:11:23.154002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:23.144026 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1020655f-3c31-4bf1-ba61-aaca9e61a489-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh\" (UID: \"1020655f-3c31-4bf1-ba61-aaca9e61a489\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" Nov 02 18:11:23.184888 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:23.169759 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1020655f-3c31-4bf1-ba61-aaca9e61a489-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh\" (UID: \"1020655f-3c31-4bf1-ba61-aaca9e61a489\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" Nov 02 18:11:23.184888 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:23.174500 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579ghxvq"] Nov 02 18:11:23.184888 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:23.175405 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjglk\" (UniqueName: \"kubernetes.io/projected/1020655f-3c31-4bf1-ba61-aaca9e61a489-kube-api-access-zjglk\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh\" (UID: \"1020655f-3c31-4bf1-ba61-aaca9e61a489\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" Nov 02 18:11:23.292027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:23.291991 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:23.444803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:23.444347 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:23.445054 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:23.444942 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" Nov 02 18:11:23.456682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:23.456648 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb"] Nov 02 18:11:23.466218 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:11:23.466186 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d94c9f_3e07_4fa5_8301_a80629738966.slice/crio-c076cbd2b07c0a326dd46e633a156352635d5fa50213ac6fc922962b80cbf0b9 WatchSource:0}: Error finding container c076cbd2b07c0a326dd46e633a156352635d5fa50213ac6fc922962b80cbf0b9: Status 404 returned error can't find the container with id c076cbd2b07c0a326dd46e633a156352635d5fa50213ac6fc922962b80cbf0b9 Nov 02 18:11:23.803623 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:23.790494 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:23.909805 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:23.909730 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" event={"ID":"c0d94c9f-3e07-4fa5-8301-a80629738966","Type":"ContainerStarted","Data":"c076cbd2b07c0a326dd46e633a156352635d5fa50213ac6fc922962b80cbf0b9"} Nov 02 18:11:23.957737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:23.957622 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh"] Nov 02 18:11:23.959251 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:11:23.959222 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1020655f_3c31_4bf1_ba61_aaca9e61a489.slice/crio-265c46ac7110bdebb88767c018ae909a4b220dbb19ff90b01f81ccd1f551bbb4 WatchSource:0}: Error finding container 265c46ac7110bdebb88767c018ae909a4b220dbb19ff90b01f81ccd1f551bbb4: Status 404 returned error can't find the container with id 265c46ac7110bdebb88767c018ae909a4b220dbb19ff90b01f81ccd1f551bbb4 Nov 02 18:11:24.131700 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.131664 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn"] Nov 02 18:11:24.136996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.136970 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" Nov 02 18:11:24.145577 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.145292 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"agc-cred-811d350f868d\"" Nov 02 18:11:24.145577 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.145552 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn"] Nov 02 18:11:24.220195 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.220153 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68203c3a-014d-4f66-9fe3-c54d019f244a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn\" (UID: \"68203c3a-014d-4f66-9fe3-c54d019f244a\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" Nov 02 18:11:24.220195 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.220199 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p6gr\" (UniqueName: \"kubernetes.io/projected/68203c3a-014d-4f66-9fe3-c54d019f244a-kube-api-access-9p6gr\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn\" (UID: \"68203c3a-014d-4f66-9fe3-c54d019f244a\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" Nov 02 18:11:24.220426 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.220324 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68203c3a-014d-4f66-9fe3-c54d019f244a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn\" (UID: \"68203c3a-014d-4f66-9fe3-c54d019f244a\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" Nov 02 18:11:24.220470 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.220425 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68203c3a-014d-4f66-9fe3-c54d019f244a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn\" (UID: \"68203c3a-014d-4f66-9fe3-c54d019f244a\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" Nov 02 18:11:24.322010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.321918 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68203c3a-014d-4f66-9fe3-c54d019f244a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn\" (UID: \"68203c3a-014d-4f66-9fe3-c54d019f244a\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" Nov 02 18:11:24.322152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.322141 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68203c3a-014d-4f66-9fe3-c54d019f244a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn\" (UID: \"68203c3a-014d-4f66-9fe3-c54d019f244a\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" Nov 02 18:11:24.322207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.322167 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p6gr\" (UniqueName: \"kubernetes.io/projected/68203c3a-014d-4f66-9fe3-c54d019f244a-kube-api-access-9p6gr\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn\" (UID: \"68203c3a-014d-4f66-9fe3-c54d019f244a\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" Nov 02 18:11:24.322330 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.322305 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68203c3a-014d-4f66-9fe3-c54d019f244a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn\" (UID: \"68203c3a-014d-4f66-9fe3-c54d019f244a\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" Nov 02 18:11:24.322768 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.322737 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68203c3a-014d-4f66-9fe3-c54d019f244a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn\" (UID: \"68203c3a-014d-4f66-9fe3-c54d019f244a\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" Nov 02 18:11:24.322909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.322887 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68203c3a-014d-4f66-9fe3-c54d019f244a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn\" (UID: \"68203c3a-014d-4f66-9fe3-c54d019f244a\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" Nov 02 18:11:24.324476 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.324451 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b8444e0-7807-4238-b0a6-aa7d4e0bdde8" path="/var/lib/kubelet/pods/0b8444e0-7807-4238-b0a6-aa7d4e0bdde8/volumes" Nov 02 18:11:24.324863 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.324841 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68203c3a-014d-4f66-9fe3-c54d019f244a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn\" (UID: \"68203c3a-014d-4f66-9fe3-c54d019f244a\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" Nov 02 18:11:24.331207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.331187 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p6gr\" (UniqueName: \"kubernetes.io/projected/68203c3a-014d-4f66-9fe3-c54d019f244a-kube-api-access-9p6gr\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn\" (UID: \"68203c3a-014d-4f66-9fe3-c54d019f244a\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" Nov 02 18:11:24.456526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.456485 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" Nov 02 18:11:24.708906 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.708866 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:24.715679 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.715642 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr"] Nov 02 18:11:24.724909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.724417 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr"] Nov 02 18:11:24.724909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.724545 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" Nov 02 18:11:24.728596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.728564 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"agc-cred-12ef34d6c6c7\"" Nov 02 18:11:24.764088 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.764061 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn"] Nov 02 18:11:24.767858 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:11:24.767825 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68203c3a_014d_4f66_9fe3_c54d019f244a.slice/crio-9da756b4e531fa3ad9d4ff220913165db7e8bf866885bd5ead36e8bf941401f3 WatchSource:0}: Error finding container 9da756b4e531fa3ad9d4ff220913165db7e8bf866885bd5ead36e8bf941401f3: Status 404 returned error can't find the container with id 9da756b4e531fa3ad9d4ff220913165db7e8bf866885bd5ead36e8bf941401f3 Nov 02 18:11:24.861108 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.854061 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:24.876761 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.864718 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82kkl\" (UniqueName: \"kubernetes.io/projected/bc62e9fc-8ebe-4009-af16-309c48fd7691-kube-api-access-82kkl\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr\" (UID: \"bc62e9fc-8ebe-4009-af16-309c48fd7691\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" Nov 02 18:11:24.876761 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.864932 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc62e9fc-8ebe-4009-af16-309c48fd7691-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr\" (UID: \"bc62e9fc-8ebe-4009-af16-309c48fd7691\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" Nov 02 18:11:24.876761 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.864994 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc62e9fc-8ebe-4009-af16-309c48fd7691-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr\" (UID: \"bc62e9fc-8ebe-4009-af16-309c48fd7691\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" Nov 02 18:11:24.876761 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.865072 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc62e9fc-8ebe-4009-af16-309c48fd7691-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr\" (UID: \"bc62e9fc-8ebe-4009-af16-309c48fd7691\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" Nov 02 18:11:24.960687 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.960616 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" event={"ID":"68203c3a-014d-4f66-9fe3-c54d019f244a","Type":"ContainerStarted","Data":"9da756b4e531fa3ad9d4ff220913165db7e8bf866885bd5ead36e8bf941401f3"} Nov 02 18:11:24.963707 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.963671 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" event={"ID":"1020655f-3c31-4bf1-ba61-aaca9e61a489","Type":"ContainerStarted","Data":"468a1086033f04fd166667c153dd9ff257b08c6aa2cd3a10d58ce57259e056cc"} Nov 02 18:11:24.963846 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.963729 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" event={"ID":"1020655f-3c31-4bf1-ba61-aaca9e61a489","Type":"ContainerStarted","Data":"265c46ac7110bdebb88767c018ae909a4b220dbb19ff90b01f81ccd1f551bbb4"} Nov 02 18:11:24.963846 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.963754 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" Nov 02 18:11:24.965763 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.965711 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" event={"ID":"c0d94c9f-3e07-4fa5-8301-a80629738966","Type":"ContainerStarted","Data":"b9bbe03982a15726f66965f1291b4e4a33764e8d0bca621e61e648f00139043f"} Nov 02 18:11:24.966298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.966273 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc62e9fc-8ebe-4009-af16-309c48fd7691-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr\" (UID: \"bc62e9fc-8ebe-4009-af16-309c48fd7691\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" Nov 02 18:11:24.966449 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.966437 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82kkl\" (UniqueName: \"kubernetes.io/projected/bc62e9fc-8ebe-4009-af16-309c48fd7691-kube-api-access-82kkl\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr\" (UID: \"bc62e9fc-8ebe-4009-af16-309c48fd7691\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" Nov 02 18:11:24.966556 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.966544 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc62e9fc-8ebe-4009-af16-309c48fd7691-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr\" (UID: \"bc62e9fc-8ebe-4009-af16-309c48fd7691\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" Nov 02 18:11:24.966603 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.966583 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc62e9fc-8ebe-4009-af16-309c48fd7691-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr\" (UID: \"bc62e9fc-8ebe-4009-af16-309c48fd7691\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" Nov 02 18:11:24.966741 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.966724 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" Nov 02 18:11:24.967196 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.967176 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc62e9fc-8ebe-4009-af16-309c48fd7691-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr\" (UID: \"bc62e9fc-8ebe-4009-af16-309c48fd7691\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" Nov 02 18:11:24.967718 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.967698 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc62e9fc-8ebe-4009-af16-309c48fd7691-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr\" (UID: \"bc62e9fc-8ebe-4009-af16-309c48fd7691\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" Nov 02 18:11:24.970213 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.970193 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc62e9fc-8ebe-4009-af16-309c48fd7691-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr\" (UID: \"bc62e9fc-8ebe-4009-af16-309c48fd7691\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" Nov 02 18:11:24.986731 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.986700 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82kkl\" (UniqueName: \"kubernetes.io/projected/bc62e9fc-8ebe-4009-af16-309c48fd7691-kube-api-access-82kkl\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr\" (UID: \"bc62e9fc-8ebe-4009-af16-309c48fd7691\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" Nov 02 18:11:24.995734 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:24.995683 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" podStartSLOduration=2.995668535 podStartE2EDuration="2.995668535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:24.994666982 +0000 UTC m=+2256.783172512" watchObservedRunningTime="2026-11-02 18:11:24.995668535 +0000 UTC m=+2256.784174062" Nov 02 18:11:25.025728 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:25.025677 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" podStartSLOduration=3.025660501 podStartE2EDuration="3.025660501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:25.021152548 +0000 UTC m=+2256.809658075" watchObservedRunningTime="2026-11-02 18:11:25.025660501 +0000 UTC m=+2256.814166028" Nov 02 18:11:25.048420 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:25.048389 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" Nov 02 18:11:25.343538 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:25.343505 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr"] Nov 02 18:11:25.344281 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:11:25.344250 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc62e9fc_8ebe_4009_af16_309c48fd7691.slice/crio-11c052681747334599342f5167a993595ae783782f0fead021a31fdb502781d1 WatchSource:0}: Error finding container 11c052681747334599342f5167a993595ae783782f0fead021a31fdb502781d1: Status 404 returned error can't find the container with id 11c052681747334599342f5167a993595ae783782f0fead021a31fdb502781d1 Nov 02 18:11:25.977554 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:25.977514 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" event={"ID":"68203c3a-014d-4f66-9fe3-c54d019f244a","Type":"ContainerStarted","Data":"58d8d41d7eec05787d20ca31c6a03f81a6e121cb1fb97b708652fb48ef6af2c3"} Nov 02 18:11:25.978065 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:25.977599 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" Nov 02 18:11:25.979612 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:25.979577 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" event={"ID":"bc62e9fc-8ebe-4009-af16-309c48fd7691","Type":"ContainerStarted","Data":"1a147752416bbef1d9e9e1b614035a506fe3e7050a3ef10eb6d99806f6eae7dc"} Nov 02 18:11:25.979750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:25.979637 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" event={"ID":"bc62e9fc-8ebe-4009-af16-309c48fd7691","Type":"ContainerStarted","Data":"11c052681747334599342f5167a993595ae783782f0fead021a31fdb502781d1"} Nov 02 18:11:25.979860 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:25.979840 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" Nov 02 18:11:25.996581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:25.996526 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" podStartSLOduration=1.9965088739999999 podStartE2EDuration="1.996508874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:25.99533422 +0000 UTC m=+2257.783839750" watchObservedRunningTime="2026-11-02 18:11:25.996508874 +0000 UTC m=+2257.785014404" Nov 02 18:11:26.019415 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:26.019360 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" podStartSLOduration=2.019344634 podStartE2EDuration="2.019344634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:26.014346334 +0000 UTC m=+2257.802851862" watchObservedRunningTime="2026-11-02 18:11:26.019344634 +0000 UTC m=+2257.807850212" Nov 02 18:11:26.172978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:26.172540 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:26.320056 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:26.319947 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:26.834936 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:26.833499 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:27.542460 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:27.541243 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:28.170514 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:28.170473 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:28.917988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:28.913873 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:29.612972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:29.611614 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:30.305041 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:30.298865 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:30.320985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:30.318508 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5"] Nov 02 18:11:30.350826 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:30.344798 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5"] Nov 02 18:11:30.350826 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:30.349346 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5" Nov 02 18:11:30.356281 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:30.356242 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-a208ef\"/\"kube-root-ca.crt\"" Nov 02 18:11:30.356473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:30.356328 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-a208ef\"/\"e2e-usage-sim-7c2607-cert\"" Nov 02 18:11:30.356473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:30.356348 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-a208ef\"/\"openshift-service-ca.crt\"" Nov 02 18:11:30.356473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:30.356463 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-a208ef\"/\"default-dockercfg-xx89k\"" Nov 02 18:11:30.440934 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:30.440893 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clqms\" (UniqueName: \"kubernetes.io/projected/7b3ed03b-5899-4984-b221-12059d4ea928-kube-api-access-clqms\") pod \"e2e-usage-sim-7c2607-5f95848c6b-79wt5\" (UID: \"7b3ed03b-5899-4984-b221-12059d4ea928\") " pod="e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5" Nov 02 18:11:30.441121 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:30.441081 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7b3ed03b-5899-4984-b221-12059d4ea928-tls\") pod \"e2e-usage-sim-7c2607-5f95848c6b-79wt5\" (UID: \"7b3ed03b-5899-4984-b221-12059d4ea928\") " pod="e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5" Nov 02 18:11:30.543013 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:30.542962 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clqms\" (UniqueName: \"kubernetes.io/projected/7b3ed03b-5899-4984-b221-12059d4ea928-kube-api-access-clqms\") pod \"e2e-usage-sim-7c2607-5f95848c6b-79wt5\" (UID: \"7b3ed03b-5899-4984-b221-12059d4ea928\") " pod="e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5" Nov 02 18:11:30.543240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:30.543060 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7b3ed03b-5899-4984-b221-12059d4ea928-tls\") pod \"e2e-usage-sim-7c2607-5f95848c6b-79wt5\" (UID: \"7b3ed03b-5899-4984-b221-12059d4ea928\") " pod="e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5" Nov 02 18:11:30.545584 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:30.545561 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7b3ed03b-5899-4984-b221-12059d4ea928-tls\") pod \"e2e-usage-sim-7c2607-5f95848c6b-79wt5\" (UID: \"7b3ed03b-5899-4984-b221-12059d4ea928\") " pod="e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5" Nov 02 18:11:30.551805 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:30.551781 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clqms\" (UniqueName: \"kubernetes.io/projected/7b3ed03b-5899-4984-b221-12059d4ea928-kube-api-access-clqms\") pod \"e2e-usage-sim-7c2607-5f95848c6b-79wt5\" (UID: \"7b3ed03b-5899-4984-b221-12059d4ea928\") " pod="e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5" Nov 02 18:11:30.682607 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:30.682519 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5" Nov 02 18:11:30.972460 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:30.972427 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5"] Nov 02 18:11:30.972849 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:11:30.972827 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b3ed03b_5899_4984_b221_12059d4ea928.slice/crio-680b83365af99be53554bf6c6b21776c5c49eec42b03e3ba4147df56b5134f11 WatchSource:0}: Error finding container 680b83365af99be53554bf6c6b21776c5c49eec42b03e3ba4147df56b5134f11: Status 404 returned error can't find the container with id 680b83365af99be53554bf6c6b21776c5c49eec42b03e3ba4147df56b5134f11 Nov 02 18:11:31.069975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:31.068996 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:31.095043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:31.091703 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5" event={"ID":"7b3ed03b-5899-4984-b221-12059d4ea928","Type":"ContainerStarted","Data":"680b83365af99be53554bf6c6b21776c5c49eec42b03e3ba4147df56b5134f11"} Nov 02 18:11:31.846402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:31.832878 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:32.101596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:32.101448 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5" event={"ID":"7b3ed03b-5899-4984-b221-12059d4ea928","Type":"ContainerStarted","Data":"500ebcc6450a5aba18847ea5901cbb35487a2e1a89b49d3e73f7b432bb4cd242"} Nov 02 18:11:32.101921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:32.101891 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5" Nov 02 18:11:32.106907 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:32.106873 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5" Nov 02 18:11:32.121797 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:32.121747 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5" podStartSLOduration=2.121733702 podStartE2EDuration="2.121733702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:32.117747479 +0000 UTC m=+2263.906253009" watchObservedRunningTime="2026-11-02 18:11:32.121733702 +0000 UTC m=+2263.910239230" Nov 02 18:11:32.551040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:32.541141 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:33.262631 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:33.262593 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:34.026977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:34.023683 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:34.658239 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:34.655743 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:35.284270 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:35.284227 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:35.983264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:35.983229 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" Nov 02 18:11:36.025067 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:36.025022 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:36.242013 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:36.241900 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:36.430466 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:36.430432 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:36.991757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:36.991563 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" Nov 02 18:11:36.992367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:36.992344 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" Nov 02 18:11:36.993224 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:36.993200 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" Nov 02 18:11:37.069197 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:37.069162 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh"] Nov 02 18:11:37.081356 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:37.081312 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb"] Nov 02 18:11:37.617980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:37.617869 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:38.387440 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:38.383904 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:39.002420 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:39.002384 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:11:39.103462 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:39.103349 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:39.922497 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:39.922450 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:41.103027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:41.102985 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:42.045975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:42.042679 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:42.112512 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:42.112310 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" podUID="1020655f-3c31-4bf1-ba61-aaca9e61a489" containerName="payload-processing" containerID="cri-o://468a1086033f04fd166667c153dd9ff257b08c6aa2cd3a10d58ce57259e056cc" gracePeriod=25 Nov 02 18:11:42.148636 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:42.148579 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" podUID="c0d94c9f-3e07-4fa5-8301-a80629738966" containerName="payload-processing" containerID="cri-o://b9bbe03982a15726f66965f1291b4e4a33764e8d0bca621e61e648f00139043f" gracePeriod=25 Nov 02 18:11:42.279619 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:42.279472 5549 generic.go:356] "Generic (PLEG): container finished" podID="1020655f-3c31-4bf1-ba61-aaca9e61a489" containerID="468a1086033f04fd166667c153dd9ff257b08c6aa2cd3a10d58ce57259e056cc" exitCode=0 Nov 02 18:11:42.279997 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:42.279883 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" event={"ID":"1020655f-3c31-4bf1-ba61-aaca9e61a489","Type":"ContainerDied","Data":"468a1086033f04fd166667c153dd9ff257b08c6aa2cd3a10d58ce57259e056cc"} Nov 02 18:11:42.282661 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:42.282375 5549 generic.go:356] "Generic (PLEG): container finished" podID="c0d94c9f-3e07-4fa5-8301-a80629738966" containerID="b9bbe03982a15726f66965f1291b4e4a33764e8d0bca621e61e648f00139043f" exitCode=0 Nov 02 18:11:42.282661 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:42.282592 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" event={"ID":"c0d94c9f-3e07-4fa5-8301-a80629738966","Type":"ContainerDied","Data":"b9bbe03982a15726f66965f1291b4e4a33764e8d0bca621e61e648f00139043f"} Nov 02 18:11:42.910541 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:42.910124 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" Nov 02 18:11:43.065990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:42.987619 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1020655f-3c31-4bf1-ba61-aaca9e61a489-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1020655f-3c31-4bf1-ba61-aaca9e61a489-plugins-config-volume\") pod \"1020655f-3c31-4bf1-ba61-aaca9e61a489\" (UID: \"1020655f-3c31-4bf1-ba61-aaca9e61a489\") " Nov 02 18:11:43.065990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:42.987774 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1020655f-3c31-4bf1-ba61-aaca9e61a489-kube-api-access-zjglk\" (UniqueName: \"kubernetes.io/projected/1020655f-3c31-4bf1-ba61-aaca9e61a489-kube-api-access-zjglk\") pod \"1020655f-3c31-4bf1-ba61-aaca9e61a489\" (UID: \"1020655f-3c31-4bf1-ba61-aaca9e61a489\") " Nov 02 18:11:43.065990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:42.990362 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1020655f-3c31-4bf1-ba61-aaca9e61a489-plugins-config-volume" pod "1020655f-3c31-4bf1-ba61-aaca9e61a489" (UID: "1020655f-3c31-4bf1-ba61-aaca9e61a489"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:43.065990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.057706 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" Nov 02 18:11:43.098977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.098775 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1020655f-3c31-4bf1-ba61-aaca9e61a489-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1020655f-3c31-4bf1-ba61-aaca9e61a489-routing-overlay-volume\") pod \"1020655f-3c31-4bf1-ba61-aaca9e61a489\" (UID: \"1020655f-3c31-4bf1-ba61-aaca9e61a489\") " Nov 02 18:11:43.098977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.098930 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1020655f-3c31-4bf1-ba61-aaca9e61a489-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1020655f-3c31-4bf1-ba61-aaca9e61a489-provider-credentials\") pod \"1020655f-3c31-4bf1-ba61-aaca9e61a489\" (UID: \"1020655f-3c31-4bf1-ba61-aaca9e61a489\") " Nov 02 18:11:43.128118 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.125184 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1020655f-3c31-4bf1-ba61-aaca9e61a489-routing-overlay-volume" pod "1020655f-3c31-4bf1-ba61-aaca9e61a489" (UID: "1020655f-3c31-4bf1-ba61-aaca9e61a489"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:43.144990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.140460 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1020655f-3c31-4bf1-ba61-aaca9e61a489-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:43.144990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.140500 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1020655f-3c31-4bf1-ba61-aaca9e61a489-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:43.164030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.162492 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:43.212535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.212207 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1020655f-3c31-4bf1-ba61-aaca9e61a489-kube-api-access-zjglk" pod "1020655f-3c31-4bf1-ba61-aaca9e61a489" (UID: "1020655f-3c31-4bf1-ba61-aaca9e61a489"). InnerVolumeSpecName "kube-api-access-zjglk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:43.296022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.244597 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d94c9f-3e07-4fa5-8301-a80629738966-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d94c9f-3e07-4fa5-8301-a80629738966-plugins-config-volume\") pod \"c0d94c9f-3e07-4fa5-8301-a80629738966\" (UID: \"c0d94c9f-3e07-4fa5-8301-a80629738966\") " Nov 02 18:11:43.296022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.244868 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d94c9f-3e07-4fa5-8301-a80629738966-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0d94c9f-3e07-4fa5-8301-a80629738966-provider-credentials\") pod \"c0d94c9f-3e07-4fa5-8301-a80629738966\" (UID: \"c0d94c9f-3e07-4fa5-8301-a80629738966\") " Nov 02 18:11:43.296022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.245015 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d94c9f-3e07-4fa5-8301-a80629738966-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0d94c9f-3e07-4fa5-8301-a80629738966-routing-overlay-volume\") pod \"c0d94c9f-3e07-4fa5-8301-a80629738966\" (UID: \"c0d94c9f-3e07-4fa5-8301-a80629738966\") " Nov 02 18:11:43.296022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.245166 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d94c9f-3e07-4fa5-8301-a80629738966-kube-api-access-6lmz4\" (UniqueName: \"kubernetes.io/projected/c0d94c9f-3e07-4fa5-8301-a80629738966-kube-api-access-6lmz4\") pod \"c0d94c9f-3e07-4fa5-8301-a80629738966\" (UID: \"c0d94c9f-3e07-4fa5-8301-a80629738966\") " Nov 02 18:11:43.296022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.246007 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjglk\" (UniqueName: \"kubernetes.io/projected/1020655f-3c31-4bf1-ba61-aaca9e61a489-kube-api-access-zjglk\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:43.425307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.425212 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d94c9f-3e07-4fa5-8301-a80629738966-routing-overlay-volume" pod "c0d94c9f-3e07-4fa5-8301-a80629738966" (UID: "c0d94c9f-3e07-4fa5-8301-a80629738966"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:43.560364 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.516709 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0d94c9f-3e07-4fa5-8301-a80629738966-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:43.576983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.576929 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d94c9f-3e07-4fa5-8301-a80629738966-plugins-config-volume" pod "c0d94c9f-3e07-4fa5-8301-a80629738966" (UID: "c0d94c9f-3e07-4fa5-8301-a80629738966"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:43.628029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.593505 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d94c9f-3e07-4fa5-8301-a80629738966-kube-api-access-6lmz4" pod "c0d94c9f-3e07-4fa5-8301-a80629738966" (UID: "c0d94c9f-3e07-4fa5-8301-a80629738966"). InnerVolumeSpecName "kube-api-access-6lmz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:43.628029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.593939 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d94c9f-3e07-4fa5-8301-a80629738966-provider-credentials" pod "c0d94c9f-3e07-4fa5-8301-a80629738966" (UID: "c0d94c9f-3e07-4fa5-8301-a80629738966"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:43.628029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.594044 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1020655f-3c31-4bf1-ba61-aaca9e61a489-provider-credentials" pod "1020655f-3c31-4bf1-ba61-aaca9e61a489" (UID: "1020655f-3c31-4bf1-ba61-aaca9e61a489"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:43.632144 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.632109 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0d94c9f-3e07-4fa5-8301-a80629738966-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:43.672743 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.672705 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lmz4\" (UniqueName: \"kubernetes.io/projected/c0d94c9f-3e07-4fa5-8301-a80629738966-kube-api-access-6lmz4\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:43.673017 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.672996 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d94c9f-3e07-4fa5-8301-a80629738966-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:43.673150 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.673135 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1020655f-3c31-4bf1-ba61-aaca9e61a489-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:43.737412 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.737370 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" event={"ID":"1020655f-3c31-4bf1-ba61-aaca9e61a489","Type":"ContainerDied","Data":"265c46ac7110bdebb88767c018ae909a4b220dbb19ff90b01f81ccd1f551bbb4"} Nov 02 18:11:43.737755 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.737734 5549 scope.go:122] "RemoveContainer" containerID="468a1086033f04fd166667c153dd9ff257b08c6aa2cd3a10d58ce57259e056cc" Nov 02 18:11:43.738043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.738022 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh" Nov 02 18:11:43.841497 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.836382 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" Nov 02 18:11:43.841497 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.836482 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb" event={"ID":"c0d94c9f-3e07-4fa5-8301-a80629738966","Type":"ContainerDied","Data":"c076cbd2b07c0a326dd46e633a156352635d5fa50213ac6fc922962b80cbf0b9"} Nov 02 18:11:43.867977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:43.862085 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:11:44.043737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:44.042082 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb"] Nov 02 18:11:44.109492 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:44.109447 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:44.111894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:44.111869 5549 scope.go:122] "RemoveContainer" containerID="b9bbe03982a15726f66965f1291b4e4a33764e8d0bca621e61e648f00139043f" Nov 02 18:11:44.113407 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:44.112943 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95gc4zb"] Nov 02 18:11:44.238116 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:44.217007 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh"] Nov 02 18:11:44.342161 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:44.340441 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d94c9f-3e07-4fa5-8301-a80629738966" path="/var/lib/kubelet/pods/c0d94c9f-3e07-4fa5-8301-a80629738966/volumes" Nov 02 18:11:44.342161 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:44.341000 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d457cd4qh"] Nov 02 18:11:44.908984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:44.904423 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:45.629307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:45.626697 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:46.324033 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:46.323996 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1020655f-3c31-4bf1-ba61-aaca9e61a489" path="/var/lib/kubelet/pods/1020655f-3c31-4bf1-ba61-aaca9e61a489/volumes" Nov 02 18:11:46.518541 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:46.518504 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:47.352976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:47.351069 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:47.590827 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:11:47.590787 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:11:48.137474 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:48.137424 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:48.938690 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:48.936718 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:49.687552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:49.687512 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:49.950806 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:49.950778 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr"] Nov 02 18:11:50.348093 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:50.348007 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:50.979827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:50.979790 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn"] Nov 02 18:11:51.118526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:51.118494 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:51.823976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:51.817386 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:52.528976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:52.526905 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:53.226916 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:53.226863 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:54.020775 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:54.020732 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:54.825987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:54.810100 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:54.951561 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:54.951517 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" podUID="bc62e9fc-8ebe-4009-af16-309c48fd7691" containerName="payload-processing" containerID="cri-o://1a147752416bbef1d9e9e1b614035a506fe3e7050a3ef10eb6d99806f6eae7dc" gracePeriod=25 Nov 02 18:11:55.271758 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.271729 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" Nov 02 18:11:55.356045 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.356011 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc62e9fc-8ebe-4009-af16-309c48fd7691-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc62e9fc-8ebe-4009-af16-309c48fd7691-plugins-config-volume\") pod \"bc62e9fc-8ebe-4009-af16-309c48fd7691\" (UID: \"bc62e9fc-8ebe-4009-af16-309c48fd7691\") " Nov 02 18:11:55.356343 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.356324 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc62e9fc-8ebe-4009-af16-309c48fd7691-kube-api-access-82kkl\" (UniqueName: \"kubernetes.io/projected/bc62e9fc-8ebe-4009-af16-309c48fd7691-kube-api-access-82kkl\") pod \"bc62e9fc-8ebe-4009-af16-309c48fd7691\" (UID: \"bc62e9fc-8ebe-4009-af16-309c48fd7691\") " Nov 02 18:11:55.356434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.356401 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc62e9fc-8ebe-4009-af16-309c48fd7691-plugins-config-volume" pod "bc62e9fc-8ebe-4009-af16-309c48fd7691" (UID: "bc62e9fc-8ebe-4009-af16-309c48fd7691"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:55.356434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.356422 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc62e9fc-8ebe-4009-af16-309c48fd7691-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc62e9fc-8ebe-4009-af16-309c48fd7691-routing-overlay-volume\") pod \"bc62e9fc-8ebe-4009-af16-309c48fd7691\" (UID: \"bc62e9fc-8ebe-4009-af16-309c48fd7691\") " Nov 02 18:11:55.356552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.356457 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc62e9fc-8ebe-4009-af16-309c48fd7691-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc62e9fc-8ebe-4009-af16-309c48fd7691-provider-credentials\") pod \"bc62e9fc-8ebe-4009-af16-309c48fd7691\" (UID: \"bc62e9fc-8ebe-4009-af16-309c48fd7691\") " Nov 02 18:11:55.356684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.356667 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc62e9fc-8ebe-4009-af16-309c48fd7691-routing-overlay-volume" pod "bc62e9fc-8ebe-4009-af16-309c48fd7691" (UID: "bc62e9fc-8ebe-4009-af16-309c48fd7691"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:55.357128 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.357103 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc62e9fc-8ebe-4009-af16-309c48fd7691-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:55.357128 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.357125 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc62e9fc-8ebe-4009-af16-309c48fd7691-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:55.358580 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.358550 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc62e9fc-8ebe-4009-af16-309c48fd7691-kube-api-access-82kkl" pod "bc62e9fc-8ebe-4009-af16-309c48fd7691" (UID: "bc62e9fc-8ebe-4009-af16-309c48fd7691"). InnerVolumeSpecName "kube-api-access-82kkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:55.358693 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.358647 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc62e9fc-8ebe-4009-af16-309c48fd7691-provider-credentials" pod "bc62e9fc-8ebe-4009-af16-309c48fd7691" (UID: "bc62e9fc-8ebe-4009-af16-309c48fd7691"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:55.474977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.472992 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82kkl\" (UniqueName: \"kubernetes.io/projected/bc62e9fc-8ebe-4009-af16-309c48fd7691-kube-api-access-82kkl\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:55.474977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.473345 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc62e9fc-8ebe-4009-af16-309c48fd7691-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:55.487255 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.487189 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:55.955660 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.955623 5549 generic.go:356] "Generic (PLEG): container finished" podID="bc62e9fc-8ebe-4009-af16-309c48fd7691" containerID="1a147752416bbef1d9e9e1b614035a506fe3e7050a3ef10eb6d99806f6eae7dc" exitCode=0 Nov 02 18:11:55.955877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.955671 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" Nov 02 18:11:55.955877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.955705 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" event={"ID":"bc62e9fc-8ebe-4009-af16-309c48fd7691","Type":"ContainerDied","Data":"1a147752416bbef1d9e9e1b614035a506fe3e7050a3ef10eb6d99806f6eae7dc"} Nov 02 18:11:55.955877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.955755 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr" event={"ID":"bc62e9fc-8ebe-4009-af16-309c48fd7691","Type":"ContainerDied","Data":"11c052681747334599342f5167a993595ae783782f0fead021a31fdb502781d1"} Nov 02 18:11:55.955877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.955780 5549 scope.go:122] "RemoveContainer" containerID="1a147752416bbef1d9e9e1b614035a506fe3e7050a3ef10eb6d99806f6eae7dc" Nov 02 18:11:55.967760 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.967739 5549 scope.go:122] "RemoveContainer" containerID="1a147752416bbef1d9e9e1b614035a506fe3e7050a3ef10eb6d99806f6eae7dc" Nov 02 18:11:55.968100 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:11:55.968078 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a147752416bbef1d9e9e1b614035a506fe3e7050a3ef10eb6d99806f6eae7dc\": container with ID starting with 1a147752416bbef1d9e9e1b614035a506fe3e7050a3ef10eb6d99806f6eae7dc not found: ID does not exist" containerID="1a147752416bbef1d9e9e1b614035a506fe3e7050a3ef10eb6d99806f6eae7dc" Nov 02 18:11:55.968170 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.968109 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a147752416bbef1d9e9e1b614035a506fe3e7050a3ef10eb6d99806f6eae7dc"} err="failed to get container status \"1a147752416bbef1d9e9e1b614035a506fe3e7050a3ef10eb6d99806f6eae7dc\": rpc error: code = NotFound desc = could not find container \"1a147752416bbef1d9e9e1b614035a506fe3e7050a3ef10eb6d99806f6eae7dc\": container with ID starting with 1a147752416bbef1d9e9e1b614035a506fe3e7050a3ef10eb6d99806f6eae7dc not found: ID does not exist" Nov 02 18:11:55.980309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.980265 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" podUID="68203c3a-014d-4f66-9fe3-c54d019f244a" containerName="payload-processing" containerID="cri-o://58d8d41d7eec05787d20ca31c6a03f81a6e121cb1fb97b708652fb48ef6af2c3" gracePeriod=25 Nov 02 18:11:55.982222 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.982199 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr"] Nov 02 18:11:55.989014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:55.988986 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5npbnr"] Nov 02 18:11:56.179455 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.168100 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:56.326909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.326872 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc62e9fc-8ebe-4009-af16-309c48fd7691" path="/var/lib/kubelet/pods/bc62e9fc-8ebe-4009-af16-309c48fd7691/volumes" Nov 02 18:11:56.343002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.342946 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" Nov 02 18:11:56.505818 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.505783 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68203c3a-014d-4f66-9fe3-c54d019f244a-kube-api-access-9p6gr\" (UniqueName: \"kubernetes.io/projected/68203c3a-014d-4f66-9fe3-c54d019f244a-kube-api-access-9p6gr\") pod \"68203c3a-014d-4f66-9fe3-c54d019f244a\" (UID: \"68203c3a-014d-4f66-9fe3-c54d019f244a\") " Nov 02 18:11:56.505818 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.505816 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68203c3a-014d-4f66-9fe3-c54d019f244a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/68203c3a-014d-4f66-9fe3-c54d019f244a-provider-credentials\") pod \"68203c3a-014d-4f66-9fe3-c54d019f244a\" (UID: \"68203c3a-014d-4f66-9fe3-c54d019f244a\") " Nov 02 18:11:56.506159 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.505895 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68203c3a-014d-4f66-9fe3-c54d019f244a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68203c3a-014d-4f66-9fe3-c54d019f244a-plugins-config-volume\") pod \"68203c3a-014d-4f66-9fe3-c54d019f244a\" (UID: \"68203c3a-014d-4f66-9fe3-c54d019f244a\") " Nov 02 18:11:56.506159 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.505930 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68203c3a-014d-4f66-9fe3-c54d019f244a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68203c3a-014d-4f66-9fe3-c54d019f244a-routing-overlay-volume\") pod \"68203c3a-014d-4f66-9fe3-c54d019f244a\" (UID: \"68203c3a-014d-4f66-9fe3-c54d019f244a\") " Nov 02 18:11:56.506368 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.506340 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68203c3a-014d-4f66-9fe3-c54d019f244a-routing-overlay-volume" pod "68203c3a-014d-4f66-9fe3-c54d019f244a" (UID: "68203c3a-014d-4f66-9fe3-c54d019f244a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:56.506442 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.506349 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68203c3a-014d-4f66-9fe3-c54d019f244a-plugins-config-volume" pod "68203c3a-014d-4f66-9fe3-c54d019f244a" (UID: "68203c3a-014d-4f66-9fe3-c54d019f244a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:56.506819 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.506800 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68203c3a-014d-4f66-9fe3-c54d019f244a-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:56.506862 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.506823 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68203c3a-014d-4f66-9fe3-c54d019f244a-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:56.508267 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.508232 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68203c3a-014d-4f66-9fe3-c54d019f244a-provider-credentials" pod "68203c3a-014d-4f66-9fe3-c54d019f244a" (UID: "68203c3a-014d-4f66-9fe3-c54d019f244a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:56.508400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.508277 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68203c3a-014d-4f66-9fe3-c54d019f244a-kube-api-access-9p6gr" pod "68203c3a-014d-4f66-9fe3-c54d019f244a" (UID: "68203c3a-014d-4f66-9fe3-c54d019f244a"). InnerVolumeSpecName "kube-api-access-9p6gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:56.607913 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.607825 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p6gr\" (UniqueName: \"kubernetes.io/projected/68203c3a-014d-4f66-9fe3-c54d019f244a-kube-api-access-9p6gr\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:56.607913 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.607858 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68203c3a-014d-4f66-9fe3-c54d019f244a-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:11:56.866024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.860381 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:56.967706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.967674 5549 generic.go:356] "Generic (PLEG): container finished" podID="68203c3a-014d-4f66-9fe3-c54d019f244a" containerID="58d8d41d7eec05787d20ca31c6a03f81a6e121cb1fb97b708652fb48ef6af2c3" exitCode=0 Nov 02 18:11:56.967909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.967734 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" Nov 02 18:11:56.967909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.967756 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" event={"ID":"68203c3a-014d-4f66-9fe3-c54d019f244a","Type":"ContainerDied","Data":"58d8d41d7eec05787d20ca31c6a03f81a6e121cb1fb97b708652fb48ef6af2c3"} Nov 02 18:11:56.967909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.967896 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn" event={"ID":"68203c3a-014d-4f66-9fe3-c54d019f244a","Type":"ContainerDied","Data":"9da756b4e531fa3ad9d4ff220913165db7e8bf866885bd5ead36e8bf941401f3"} Nov 02 18:11:56.968123 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.967929 5549 scope.go:122] "RemoveContainer" containerID="58d8d41d7eec05787d20ca31c6a03f81a6e121cb1fb97b708652fb48ef6af2c3" Nov 02 18:11:56.980855 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.980830 5549 scope.go:122] "RemoveContainer" containerID="58d8d41d7eec05787d20ca31c6a03f81a6e121cb1fb97b708652fb48ef6af2c3" Nov 02 18:11:56.981158 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:11:56.981136 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d8d41d7eec05787d20ca31c6a03f81a6e121cb1fb97b708652fb48ef6af2c3\": container with ID starting with 58d8d41d7eec05787d20ca31c6a03f81a6e121cb1fb97b708652fb48ef6af2c3 not found: ID does not exist" containerID="58d8d41d7eec05787d20ca31c6a03f81a6e121cb1fb97b708652fb48ef6af2c3" Nov 02 18:11:56.981257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.981166 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d8d41d7eec05787d20ca31c6a03f81a6e121cb1fb97b708652fb48ef6af2c3"} err="failed to get container status \"58d8d41d7eec05787d20ca31c6a03f81a6e121cb1fb97b708652fb48ef6af2c3\": rpc error: code = NotFound desc = could not find container \"58d8d41d7eec05787d20ca31c6a03f81a6e121cb1fb97b708652fb48ef6af2c3\": container with ID starting with 58d8d41d7eec05787d20ca31c6a03f81a6e121cb1fb97b708652fb48ef6af2c3 not found: ID does not exist" Nov 02 18:11:56.995694 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.995672 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn"] Nov 02 18:11:56.999823 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:56.999804 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bd8rkqn"] Nov 02 18:11:57.545975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:57.545252 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:58.176977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:58.176412 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:58.324422 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:58.324385 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68203c3a-014d-4f66-9fe3-c54d019f244a" path="/var/lib/kubelet/pods/68203c3a-014d-4f66-9fe3-c54d019f244a/volumes" Nov 02 18:11:58.838977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:58.837352 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:59.513382 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:11:59.513341 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:00.150058 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:00.147909 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:00.866103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:00.866067 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:01.523809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:01.523769 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:01.672439 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:01.672392 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:02.170838 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:02.170745 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:02.905978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:02.898908 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:03.688979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:03.687408 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:04.302998 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:04.302927 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:05.036019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:05.030674 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:05.663972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:05.663636 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:06.516446 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:06.516395 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:07.410382 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:07.410330 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:08.339180 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:08.339121 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:09.240541 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:09.240500 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:09.895663 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:09.895604 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:10.597973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:10.593457 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:10.751284 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:10.751247 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:10.892592 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:10.892497 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:11.413508 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.413474 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg"] Nov 02 18:12:11.414429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.414412 5549 cpu_manager.go:414] "containerMap: removing container" podUID="1020655f-3c31-4bf1-ba61-aaca9e61a489" containerName="payload-processing" Nov 02 18:12:11.414488 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.414431 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1020655f-3c31-4bf1-ba61-aaca9e61a489" containerName="payload-processing" Nov 02 18:12:11.414488 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.414459 5549 cpu_manager.go:414] "containerMap: removing container" podUID="c0d94c9f-3e07-4fa5-8301-a80629738966" containerName="payload-processing" Nov 02 18:12:11.414488 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.414464 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d94c9f-3e07-4fa5-8301-a80629738966" containerName="payload-processing" Nov 02 18:12:11.414589 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.414497 5549 cpu_manager.go:414] "containerMap: removing container" podUID="68203c3a-014d-4f66-9fe3-c54d019f244a" containerName="payload-processing" Nov 02 18:12:11.414589 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.414501 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68203c3a-014d-4f66-9fe3-c54d019f244a" containerName="payload-processing" Nov 02 18:12:11.414589 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.414515 5549 cpu_manager.go:414] "containerMap: removing container" podUID="bc62e9fc-8ebe-4009-af16-309c48fd7691" containerName="payload-processing" Nov 02 18:12:11.414589 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.414520 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc62e9fc-8ebe-4009-af16-309c48fd7691" containerName="payload-processing" Nov 02 18:12:11.414721 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.414646 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1020655f-3c31-4bf1-ba61-aaca9e61a489" containerName="payload-processing" Nov 02 18:12:11.414721 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.414657 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d94c9f-3e07-4fa5-8301-a80629738966" containerName="payload-processing" Nov 02 18:12:11.414721 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.414670 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="68203c3a-014d-4f66-9fe3-c54d019f244a" containerName="payload-processing" Nov 02 18:12:11.414721 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.414679 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc62e9fc-8ebe-4009-af16-309c48fd7691" containerName="payload-processing" Nov 02 18:12:11.420245 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.420226 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" Nov 02 18:12:11.425254 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.425228 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"agc-cred-85b1df01d456\"" Nov 02 18:12:11.426537 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.426512 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:11.426757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.426548 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"kube-root-ca.crt\"" Nov 02 18:12:11.426877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.426561 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"payload-processing-external-model-e2e--69602eb2-dockercfg-72qtk\"" Nov 02 18:12:11.426877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.426580 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"openshift-service-ca.crt\"" Nov 02 18:12:11.426877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.426608 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"routing-overlay\"" Nov 02 18:12:11.426877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.426662 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"payload-processing-external-model-plugins-e2e-extmod-b-327608\"" Nov 02 18:12:11.428898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.428876 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg"] Nov 02 18:12:11.571058 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.571021 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9fd1072-34d9-4168-9389-d522ae521491-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg\" (UID: \"e9fd1072-34d9-4168-9389-d522ae521491\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" Nov 02 18:12:11.571256 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.571232 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fd1072-34d9-4168-9389-d522ae521491-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg\" (UID: \"e9fd1072-34d9-4168-9389-d522ae521491\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" Nov 02 18:12:11.571387 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.571372 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkknr\" (UniqueName: \"kubernetes.io/projected/e9fd1072-34d9-4168-9389-d522ae521491-kube-api-access-lkknr\") pod \"payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg\" (UID: \"e9fd1072-34d9-4168-9389-d522ae521491\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" Nov 02 18:12:11.571523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.571509 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9fd1072-34d9-4168-9389-d522ae521491-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg\" (UID: \"e9fd1072-34d9-4168-9389-d522ae521491\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" Nov 02 18:12:11.673342 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.673248 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fd1072-34d9-4168-9389-d522ae521491-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg\" (UID: \"e9fd1072-34d9-4168-9389-d522ae521491\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" Nov 02 18:12:11.674097 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.674070 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkknr\" (UniqueName: \"kubernetes.io/projected/e9fd1072-34d9-4168-9389-d522ae521491-kube-api-access-lkknr\") pod \"payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg\" (UID: \"e9fd1072-34d9-4168-9389-d522ae521491\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" Nov 02 18:12:11.674406 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.674385 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9fd1072-34d9-4168-9389-d522ae521491-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg\" (UID: \"e9fd1072-34d9-4168-9389-d522ae521491\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" Nov 02 18:12:11.674734 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.674715 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9fd1072-34d9-4168-9389-d522ae521491-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg\" (UID: \"e9fd1072-34d9-4168-9389-d522ae521491\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" Nov 02 18:12:11.675105 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.674814 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fd1072-34d9-4168-9389-d522ae521491-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg\" (UID: \"e9fd1072-34d9-4168-9389-d522ae521491\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" Nov 02 18:12:11.675708 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.675677 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9fd1072-34d9-4168-9389-d522ae521491-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg\" (UID: \"e9fd1072-34d9-4168-9389-d522ae521491\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" Nov 02 18:12:11.717988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.716212 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9fd1072-34d9-4168-9389-d522ae521491-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg\" (UID: \"e9fd1072-34d9-4168-9389-d522ae521491\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" Nov 02 18:12:11.729575 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.729539 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkknr\" (UniqueName: \"kubernetes.io/projected/e9fd1072-34d9-4168-9389-d522ae521491-kube-api-access-lkknr\") pod \"payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg\" (UID: \"e9fd1072-34d9-4168-9389-d522ae521491\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" Nov 02 18:12:11.740308 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:11.740272 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" Nov 02 18:12:12.041994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:12.040828 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:12.231140 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:12:12.231103 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9fd1072_34d9_4168_9389_d522ae521491.slice/crio-3672f80a0d29893796f3cf4cd205ba06e030dc8fcd92c44fe6d017e82cc82ad7 WatchSource:0}: Error finding container 3672f80a0d29893796f3cf4cd205ba06e030dc8fcd92c44fe6d017e82cc82ad7: Status 404 returned error can't find the container with id 3672f80a0d29893796f3cf4cd205ba06e030dc8fcd92c44fe6d017e82cc82ad7 Nov 02 18:12:12.231322 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:12.231299 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg"] Nov 02 18:12:12.731052 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:12.724650 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:13.189379 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:13.189293 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" event={"ID":"e9fd1072-34d9-4168-9389-d522ae521491","Type":"ContainerStarted","Data":"db32b9f1661fcb7278674379e61de1e1b422630943df77c01267e5fd3dff8840"} Nov 02 18:12:13.189379 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:13.189350 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" event={"ID":"e9fd1072-34d9-4168-9389-d522ae521491","Type":"ContainerStarted","Data":"3672f80a0d29893796f3cf4cd205ba06e030dc8fcd92c44fe6d017e82cc82ad7"} Nov 02 18:12:13.189379 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:13.189376 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" Nov 02 18:12:13.208799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:13.208746 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" podStartSLOduration=2.20873427 podStartE2EDuration="2.20873427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:12:13.206801179 +0000 UTC m=+2304.995306709" watchObservedRunningTime="2026-11-02 18:12:13.20873427 +0000 UTC m=+2304.997239798" Nov 02 18:12:13.440517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:13.440427 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:14.263556 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:14.263519 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:14.906007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:14.905607 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:15.836999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:15.825937 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:16.499352 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:16.499235 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:17.190009 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:17.180815 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:17.589758 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:12:17.589724 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:12:17.814921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:17.814873 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:18.516119 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:18.516082 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:19.228434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:19.228379 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:19.998116 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:19.998074 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:20.634426 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:20.633631 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:21.354169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:21.354116 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:22.056974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:22.053278 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:22.716623 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:22.716575 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:23.420370 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:23.420335 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:24.132583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:24.132543 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:24.201305 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:24.201275 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" Nov 02 18:12:24.559488 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:24.559455 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs"] Nov 02 18:12:24.559680 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:24.559646 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" podUID="026e7c96-2366-483a-ae9c-55a92607394c" containerName="istio-proxy" containerID="cri-o://2529251fe28dadded5ae109c0c749b2e90f4fcd3606f44676f5dd1cbfe16422d" gracePeriod=30 Nov 02 18:12:24.572239 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:24.572207 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn"] Nov 02 18:12:24.572469 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:24.572443 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" podUID="9a22d1c3-76ca-445d-ab56-052577f4fcff" containerName="istio-proxy" containerID="cri-o://d8b83019bdd4a6b996b690c6c81a33e2ce0f869611bba896ccfc8ca5544c4143" gracePeriod=30 Nov 02 18:12:25.034170 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:25.024021 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:25.720750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:25.720659 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:26.439011 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:26.438871 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:27.105942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:27.105903 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:27.826982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:27.821565 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:28.575533 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:28.553607 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:29.236087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:29.236038 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:29.930091 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:29.910181 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 18:12:29.996832 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:29.996792 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:30.041467 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.041432 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 18:12:30.077496 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.077465 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-workload-certs\") pod \"026e7c96-2366-483a-ae9c-55a92607394c\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " Nov 02 18:12:30.077698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.077515 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-istio-data\") pod \"026e7c96-2366-483a-ae9c-55a92607394c\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " Nov 02 18:12:30.077698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.077560 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-credential-socket\") pod \"026e7c96-2366-483a-ae9c-55a92607394c\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " Nov 02 18:12:30.077698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.077677 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026e7c96-2366-483a-ae9c-55a92607394c-istio-token\" (UniqueName: \"kubernetes.io/projected/026e7c96-2366-483a-ae9c-55a92607394c-istio-token\") pod \"026e7c96-2366-483a-ae9c-55a92607394c\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " Nov 02 18:12:30.077887 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.077760 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-workload-socket\") pod \"026e7c96-2366-483a-ae9c-55a92607394c\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " Nov 02 18:12:30.077887 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.077855 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-istio-envoy\") pod \"026e7c96-2366-483a-ae9c-55a92607394c\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " Nov 02 18:12:30.078014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.077910 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026e7c96-2366-483a-ae9c-55a92607394c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026e7c96-2366-483a-ae9c-55a92607394c-istiod-ca-cert\") pod \"026e7c96-2366-483a-ae9c-55a92607394c\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " Nov 02 18:12:30.078075 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.078053 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026e7c96-2366-483a-ae9c-55a92607394c-kube-api-access-tx4qs\" (UniqueName: \"kubernetes.io/projected/026e7c96-2366-483a-ae9c-55a92607394c-kube-api-access-tx4qs\") pod \"026e7c96-2366-483a-ae9c-55a92607394c\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " Nov 02 18:12:30.078165 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.078149 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/026e7c96-2366-483a-ae9c-55a92607394c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026e7c96-2366-483a-ae9c-55a92607394c-istio-podinfo\") pod \"026e7c96-2366-483a-ae9c-55a92607394c\" (UID: \"026e7c96-2366-483a-ae9c-55a92607394c\") " Nov 02 18:12:30.079833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.078702 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-istio-data" pod "026e7c96-2366-483a-ae9c-55a92607394c" (UID: "026e7c96-2366-483a-ae9c-55a92607394c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:30.079833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.078915 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-workload-certs" pod "026e7c96-2366-483a-ae9c-55a92607394c" (UID: "026e7c96-2366-483a-ae9c-55a92607394c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:30.079833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.079237 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026e7c96-2366-483a-ae9c-55a92607394c-istiod-ca-cert" pod "026e7c96-2366-483a-ae9c-55a92607394c" (UID: "026e7c96-2366-483a-ae9c-55a92607394c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:30.079833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.079350 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-credential-socket" pod "026e7c96-2366-483a-ae9c-55a92607394c" (UID: "026e7c96-2366-483a-ae9c-55a92607394c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:30.079833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.079791 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-workload-socket" pod "026e7c96-2366-483a-ae9c-55a92607394c" (UID: "026e7c96-2366-483a-ae9c-55a92607394c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:30.081171 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.081149 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026e7c96-2366-483a-ae9c-55a92607394c-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.081294 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.081277 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.081365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.081298 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.081365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.081312 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.081365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.081326 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.082214 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.082193 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-istio-envoy" pod "026e7c96-2366-483a-ae9c-55a92607394c" (UID: "026e7c96-2366-483a-ae9c-55a92607394c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:30.082425 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.082398 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026e7c96-2366-483a-ae9c-55a92607394c-istio-token" pod "026e7c96-2366-483a-ae9c-55a92607394c" (UID: "026e7c96-2366-483a-ae9c-55a92607394c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:30.084070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.084048 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026e7c96-2366-483a-ae9c-55a92607394c-kube-api-access-tx4qs" pod "026e7c96-2366-483a-ae9c-55a92607394c" (UID: "026e7c96-2366-483a-ae9c-55a92607394c"). InnerVolumeSpecName "kube-api-access-tx4qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:30.085624 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.085603 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/026e7c96-2366-483a-ae9c-55a92607394c-istio-podinfo" pod "026e7c96-2366-483a-ae9c-55a92607394c" (UID: "026e7c96-2366-483a-ae9c-55a92607394c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:12:30.183211 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.183107 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-workload-certs\") pod \"9a22d1c3-76ca-445d-ab56-052577f4fcff\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " Nov 02 18:12:30.183211 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.183163 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-data\") pod \"9a22d1c3-76ca-445d-ab56-052577f4fcff\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " Nov 02 18:12:30.183558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.183247 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-token\" (UniqueName: \"kubernetes.io/projected/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-token\") pod \"9a22d1c3-76ca-445d-ab56-052577f4fcff\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " Nov 02 18:12:30.183558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.183288 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a22d1c3-76ca-445d-ab56-052577f4fcff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a22d1c3-76ca-445d-ab56-052577f4fcff-istiod-ca-cert\") pod \"9a22d1c3-76ca-445d-ab56-052577f4fcff\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " Nov 02 18:12:30.183558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.183373 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-envoy\") pod \"9a22d1c3-76ca-445d-ab56-052577f4fcff\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " Nov 02 18:12:30.183558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.183449 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-podinfo\") pod \"9a22d1c3-76ca-445d-ab56-052577f4fcff\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " Nov 02 18:12:30.183558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.183501 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a22d1c3-76ca-445d-ab56-052577f4fcff-kube-api-access-fk88c\" (UniqueName: \"kubernetes.io/projected/9a22d1c3-76ca-445d-ab56-052577f4fcff-kube-api-access-fk88c\") pod \"9a22d1c3-76ca-445d-ab56-052577f4fcff\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " Nov 02 18:12:30.183785 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.183704 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-workload-socket\") pod \"9a22d1c3-76ca-445d-ab56-052577f4fcff\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " Nov 02 18:12:30.184007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.183942 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-credential-socket\") pod \"9a22d1c3-76ca-445d-ab56-052577f4fcff\" (UID: \"9a22d1c3-76ca-445d-ab56-052577f4fcff\") " Nov 02 18:12:30.184716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.184697 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026e7c96-2366-483a-ae9c-55a92607394c-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.184789 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.184718 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026e7c96-2366-483a-ae9c-55a92607394c-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.184789 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.184728 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026e7c96-2366-483a-ae9c-55a92607394c-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.184789 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.184737 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx4qs\" (UniqueName: \"kubernetes.io/projected/026e7c96-2366-483a-ae9c-55a92607394c-kube-api-access-tx4qs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.185001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.184987 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-credential-socket" pod "9a22d1c3-76ca-445d-ab56-052577f4fcff" (UID: "9a22d1c3-76ca-445d-ab56-052577f4fcff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:30.185180 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.185165 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-workload-certs" pod "9a22d1c3-76ca-445d-ab56-052577f4fcff" (UID: "9a22d1c3-76ca-445d-ab56-052577f4fcff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:30.185323 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.185310 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-data" pod "9a22d1c3-76ca-445d-ab56-052577f4fcff" (UID: "9a22d1c3-76ca-445d-ab56-052577f4fcff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:30.185885 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.185858 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-workload-socket" pod "9a22d1c3-76ca-445d-ab56-052577f4fcff" (UID: "9a22d1c3-76ca-445d-ab56-052577f4fcff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:30.186098 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.185994 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a22d1c3-76ca-445d-ab56-052577f4fcff-istiod-ca-cert" pod "9a22d1c3-76ca-445d-ab56-052577f4fcff" (UID: "9a22d1c3-76ca-445d-ab56-052577f4fcff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:30.187479 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.187452 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-token" pod "9a22d1c3-76ca-445d-ab56-052577f4fcff" (UID: "9a22d1c3-76ca-445d-ab56-052577f4fcff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:30.188409 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.188386 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-envoy" pod "9a22d1c3-76ca-445d-ab56-052577f4fcff" (UID: "9a22d1c3-76ca-445d-ab56-052577f4fcff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:30.189319 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.189294 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a22d1c3-76ca-445d-ab56-052577f4fcff-kube-api-access-fk88c" pod "9a22d1c3-76ca-445d-ab56-052577f4fcff" (UID: "9a22d1c3-76ca-445d-ab56-052577f4fcff"). InnerVolumeSpecName "kube-api-access-fk88c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:30.189424 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.189408 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-podinfo" pod "9a22d1c3-76ca-445d-ab56-052577f4fcff" (UID: "9a22d1c3-76ca-445d-ab56-052577f4fcff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:12:30.286553 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.286517 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk88c\" (UniqueName: \"kubernetes.io/projected/9a22d1c3-76ca-445d-ab56-052577f4fcff-kube-api-access-fk88c\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.286553 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.286550 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.286747 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.286560 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.286747 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.286570 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.286747 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.286579 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.286747 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.286587 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.286747 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.286596 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a22d1c3-76ca-445d-ab56-052577f4fcff-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.286747 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.286604 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.286747 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.286611 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a22d1c3-76ca-445d-ab56-052577f4fcff-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.382436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.382402 5549 generic.go:356] "Generic (PLEG): container finished" podID="9a22d1c3-76ca-445d-ab56-052577f4fcff" containerID="d8b83019bdd4a6b996b690c6c81a33e2ce0f869611bba896ccfc8ca5544c4143" exitCode=0 Nov 02 18:12:30.382590 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.382447 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" Nov 02 18:12:30.382590 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.382479 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" event={"ID":"9a22d1c3-76ca-445d-ab56-052577f4fcff","Type":"ContainerDied","Data":"d8b83019bdd4a6b996b690c6c81a33e2ce0f869611bba896ccfc8ca5544c4143"} Nov 02 18:12:30.382590 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.382537 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn" event={"ID":"9a22d1c3-76ca-445d-ab56-052577f4fcff","Type":"ContainerDied","Data":"8423912090dc2eeabc4b8214c32b676087baf3f749c2a0c2ae3ddd61c9149117"} Nov 02 18:12:30.382590 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.382568 5549 scope.go:122] "RemoveContainer" containerID="d8b83019bdd4a6b996b690c6c81a33e2ce0f869611bba896ccfc8ca5544c4143" Nov 02 18:12:30.384151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.384127 5549 generic.go:356] "Generic (PLEG): container finished" podID="026e7c96-2366-483a-ae9c-55a92607394c" containerID="2529251fe28dadded5ae109c0c749b2e90f4fcd3606f44676f5dd1cbfe16422d" exitCode=0 Nov 02 18:12:30.384249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.384178 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" Nov 02 18:12:30.384249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.384209 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" event={"ID":"026e7c96-2366-483a-ae9c-55a92607394c","Type":"ContainerDied","Data":"2529251fe28dadded5ae109c0c749b2e90f4fcd3606f44676f5dd1cbfe16422d"} Nov 02 18:12:30.384341 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.384263 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs" event={"ID":"026e7c96-2366-483a-ae9c-55a92607394c","Type":"ContainerDied","Data":"a392ad03dcc764a5a2bb5951088e5edf20ceedf3414655fb90b20e210c0aaeac"} Nov 02 18:12:30.394669 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.394653 5549 scope.go:122] "RemoveContainer" containerID="a52b88620fb74528f2f11b3f3b294a868725e09db95d8ec2a5e9c3a0e17de335" Nov 02 18:12:30.405541 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.405518 5549 scope.go:122] "RemoveContainer" containerID="d8b83019bdd4a6b996b690c6c81a33e2ce0f869611bba896ccfc8ca5544c4143" Nov 02 18:12:30.405826 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:12:30.405806 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b83019bdd4a6b996b690c6c81a33e2ce0f869611bba896ccfc8ca5544c4143\": container with ID starting with d8b83019bdd4a6b996b690c6c81a33e2ce0f869611bba896ccfc8ca5544c4143 not found: ID does not exist" containerID="d8b83019bdd4a6b996b690c6c81a33e2ce0f869611bba896ccfc8ca5544c4143" Nov 02 18:12:30.405879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.405835 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b83019bdd4a6b996b690c6c81a33e2ce0f869611bba896ccfc8ca5544c4143"} err="failed to get container status \"d8b83019bdd4a6b996b690c6c81a33e2ce0f869611bba896ccfc8ca5544c4143\": rpc error: code = NotFound desc = could not find container \"d8b83019bdd4a6b996b690c6c81a33e2ce0f869611bba896ccfc8ca5544c4143\": container with ID starting with d8b83019bdd4a6b996b690c6c81a33e2ce0f869611bba896ccfc8ca5544c4143 not found: ID does not exist" Nov 02 18:12:30.405879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.405854 5549 scope.go:122] "RemoveContainer" containerID="a52b88620fb74528f2f11b3f3b294a868725e09db95d8ec2a5e9c3a0e17de335" Nov 02 18:12:30.406183 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.406163 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn"] Nov 02 18:12:30.406257 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:12:30.406173 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52b88620fb74528f2f11b3f3b294a868725e09db95d8ec2a5e9c3a0e17de335\": container with ID starting with a52b88620fb74528f2f11b3f3b294a868725e09db95d8ec2a5e9c3a0e17de335 not found: ID does not exist" containerID="a52b88620fb74528f2f11b3f3b294a868725e09db95d8ec2a5e9c3a0e17de335" Nov 02 18:12:30.406257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.406202 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52b88620fb74528f2f11b3f3b294a868725e09db95d8ec2a5e9c3a0e17de335"} err="failed to get container status \"a52b88620fb74528f2f11b3f3b294a868725e09db95d8ec2a5e9c3a0e17de335\": rpc error: code = NotFound desc = could not find container \"a52b88620fb74528f2f11b3f3b294a868725e09db95d8ec2a5e9c3a0e17de335\": container with ID starting with a52b88620fb74528f2f11b3f3b294a868725e09db95d8ec2a5e9c3a0e17de335 not found: ID does not exist" Nov 02 18:12:30.406257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.406225 5549 scope.go:122] "RemoveContainer" containerID="2529251fe28dadded5ae109c0c749b2e90f4fcd3606f44676f5dd1cbfe16422d" Nov 02 18:12:30.413269 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.413246 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-bzvrn"] Nov 02 18:12:30.416895 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.416874 5549 scope.go:122] "RemoveContainer" containerID="2529251fe28dadded5ae109c0c749b2e90f4fcd3606f44676f5dd1cbfe16422d" Nov 02 18:12:30.417174 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:12:30.417157 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2529251fe28dadded5ae109c0c749b2e90f4fcd3606f44676f5dd1cbfe16422d\": container with ID starting with 2529251fe28dadded5ae109c0c749b2e90f4fcd3606f44676f5dd1cbfe16422d not found: ID does not exist" containerID="2529251fe28dadded5ae109c0c749b2e90f4fcd3606f44676f5dd1cbfe16422d" Nov 02 18:12:30.417234 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.417180 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2529251fe28dadded5ae109c0c749b2e90f4fcd3606f44676f5dd1cbfe16422d"} err="failed to get container status \"2529251fe28dadded5ae109c0c749b2e90f4fcd3606f44676f5dd1cbfe16422d\": rpc error: code = NotFound desc = could not find container \"2529251fe28dadded5ae109c0c749b2e90f4fcd3606f44676f5dd1cbfe16422d\": container with ID starting with 2529251fe28dadded5ae109c0c749b2e90f4fcd3606f44676f5dd1cbfe16422d not found: ID does not exist" Nov 02 18:12:30.424408 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.424381 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs"] Nov 02 18:12:30.428798 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.428778 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6g8bs"] Nov 02 18:12:30.724823 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:30.724789 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:31.447623 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:31.435765 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:32.200019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:32.198437 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:32.323109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:32.323076 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="026e7c96-2366-483a-ae9c-55a92607394c" path="/var/lib/kubelet/pods/026e7c96-2366-483a-ae9c-55a92607394c/volumes" Nov 02 18:12:32.323695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:32.323679 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a22d1c3-76ca-445d-ab56-052577f4fcff" path="/var/lib/kubelet/pods/9a22d1c3-76ca-445d-ab56-052577f4fcff/volumes" Nov 02 18:12:32.482922 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:32.482839 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg"] Nov 02 18:12:32.483317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:32.483064 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" podUID="435cec96-4072-46ce-852a-d1faec26752f" containerName="istio-proxy" containerID="cri-o://4494a059d70a7d4df5e9cc4563e9a6c11f64db3331306cc396a4a56c0650dada" gracePeriod=30 Nov 02 18:12:32.953042 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:32.952993 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:33.725591 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:33.724560 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:34.457973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:34.456451 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:35.114269 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:35.114232 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:35.848003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:35.833852 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:36.628508 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:36.628467 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:37.363505 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.363452 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:37.782411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.782383 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 18:12:37.863925 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.863886 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-credential-socket\") pod \"435cec96-4072-46ce-852a-d1faec26752f\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " Nov 02 18:12:37.864119 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.863991 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/435cec96-4072-46ce-852a-d1faec26752f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435cec96-4072-46ce-852a-d1faec26752f-istiod-ca-cert\") pod \"435cec96-4072-46ce-852a-d1faec26752f\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " Nov 02 18:12:37.864119 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.864046 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/435cec96-4072-46ce-852a-d1faec26752f-istio-token\" (UniqueName: \"kubernetes.io/projected/435cec96-4072-46ce-852a-d1faec26752f-istio-token\") pod \"435cec96-4072-46ce-852a-d1faec26752f\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " Nov 02 18:12:37.864119 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.864115 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-workload-socket\") pod \"435cec96-4072-46ce-852a-d1faec26752f\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " Nov 02 18:12:37.864258 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.864187 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-istio-data\") pod \"435cec96-4072-46ce-852a-d1faec26752f\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " Nov 02 18:12:37.864304 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.864242 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-credential-socket" pod "435cec96-4072-46ce-852a-d1faec26752f" (UID: "435cec96-4072-46ce-852a-d1faec26752f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:37.864352 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.864310 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/435cec96-4072-46ce-852a-d1faec26752f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435cec96-4072-46ce-852a-d1faec26752f-istio-podinfo\") pod \"435cec96-4072-46ce-852a-d1faec26752f\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " Nov 02 18:12:37.864509 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.864483 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/435cec96-4072-46ce-852a-d1faec26752f-kube-api-access-ptrq4\" (UniqueName: \"kubernetes.io/projected/435cec96-4072-46ce-852a-d1faec26752f-kube-api-access-ptrq4\") pod \"435cec96-4072-46ce-852a-d1faec26752f\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " Nov 02 18:12:37.864830 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.864538 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-workload-certs\") pod \"435cec96-4072-46ce-852a-d1faec26752f\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " Nov 02 18:12:37.864830 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.864699 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-istio-envoy\") pod \"435cec96-4072-46ce-852a-d1faec26752f\" (UID: \"435cec96-4072-46ce-852a-d1faec26752f\") " Nov 02 18:12:37.864830 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.864702 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-workload-socket" pod "435cec96-4072-46ce-852a-d1faec26752f" (UID: "435cec96-4072-46ce-852a-d1faec26752f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:37.865598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.865554 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:37.865598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.865583 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:37.865818 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.865614 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-istio-data" pod "435cec96-4072-46ce-852a-d1faec26752f" (UID: "435cec96-4072-46ce-852a-d1faec26752f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:37.865908 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.865881 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/435cec96-4072-46ce-852a-d1faec26752f-istiod-ca-cert" pod "435cec96-4072-46ce-852a-d1faec26752f" (UID: "435cec96-4072-46ce-852a-d1faec26752f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:37.866166 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.866139 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-workload-certs" pod "435cec96-4072-46ce-852a-d1faec26752f" (UID: "435cec96-4072-46ce-852a-d1faec26752f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:37.868368 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.868338 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-istio-envoy" pod "435cec96-4072-46ce-852a-d1faec26752f" (UID: "435cec96-4072-46ce-852a-d1faec26752f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:37.868488 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.868423 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435cec96-4072-46ce-852a-d1faec26752f-istio-token" pod "435cec96-4072-46ce-852a-d1faec26752f" (UID: "435cec96-4072-46ce-852a-d1faec26752f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:37.869484 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.869315 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/435cec96-4072-46ce-852a-d1faec26752f-istio-podinfo" pod "435cec96-4072-46ce-852a-d1faec26752f" (UID: "435cec96-4072-46ce-852a-d1faec26752f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:12:37.869484 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.869451 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435cec96-4072-46ce-852a-d1faec26752f-kube-api-access-ptrq4" pod "435cec96-4072-46ce-852a-d1faec26752f" (UID: "435cec96-4072-46ce-852a-d1faec26752f"). InnerVolumeSpecName "kube-api-access-ptrq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:37.967595 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.967562 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptrq4\" (UniqueName: \"kubernetes.io/projected/435cec96-4072-46ce-852a-d1faec26752f-kube-api-access-ptrq4\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:37.967595 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.967595 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:37.967802 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.967612 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:37.967802 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.967626 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435cec96-4072-46ce-852a-d1faec26752f-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:37.967802 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.967637 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435cec96-4072-46ce-852a-d1faec26752f-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:37.967802 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.967646 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435cec96-4072-46ce-852a-d1faec26752f-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:37.967802 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:37.967654 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435cec96-4072-46ce-852a-d1faec26752f-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:38.051135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:38.048389 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:38.456344 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:38.456304 5549 generic.go:356] "Generic (PLEG): container finished" podID="435cec96-4072-46ce-852a-d1faec26752f" containerID="4494a059d70a7d4df5e9cc4563e9a6c11f64db3331306cc396a4a56c0650dada" exitCode=0 Nov 02 18:12:38.456531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:38.456350 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" Nov 02 18:12:38.456531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:38.456381 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" event={"ID":"435cec96-4072-46ce-852a-d1faec26752f","Type":"ContainerDied","Data":"4494a059d70a7d4df5e9cc4563e9a6c11f64db3331306cc396a4a56c0650dada"} Nov 02 18:12:38.456531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:38.456438 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg" event={"ID":"435cec96-4072-46ce-852a-d1faec26752f","Type":"ContainerDied","Data":"fcc55ad598878c680d557bf17c2347ad2ffa9cc9d2984f7550ad209f911a323c"} Nov 02 18:12:38.456531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:38.456460 5549 scope.go:122] "RemoveContainer" containerID="4494a059d70a7d4df5e9cc4563e9a6c11f64db3331306cc396a4a56c0650dada" Nov 02 18:12:38.468566 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:38.468548 5549 scope.go:122] "RemoveContainer" containerID="4494a059d70a7d4df5e9cc4563e9a6c11f64db3331306cc396a4a56c0650dada" Nov 02 18:12:38.468893 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:12:38.468865 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4494a059d70a7d4df5e9cc4563e9a6c11f64db3331306cc396a4a56c0650dada\": container with ID starting with 4494a059d70a7d4df5e9cc4563e9a6c11f64db3331306cc396a4a56c0650dada not found: ID does not exist" containerID="4494a059d70a7d4df5e9cc4563e9a6c11f64db3331306cc396a4a56c0650dada" Nov 02 18:12:38.468973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:38.468900 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4494a059d70a7d4df5e9cc4563e9a6c11f64db3331306cc396a4a56c0650dada"} err="failed to get container status \"4494a059d70a7d4df5e9cc4563e9a6c11f64db3331306cc396a4a56c0650dada\": rpc error: code = NotFound desc = could not find container \"4494a059d70a7d4df5e9cc4563e9a6c11f64db3331306cc396a4a56c0650dada\": container with ID starting with 4494a059d70a7d4df5e9cc4563e9a6c11f64db3331306cc396a4a56c0650dada not found: ID does not exist" Nov 02 18:12:38.495917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:38.495881 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg"] Nov 02 18:12:38.500525 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:38.500491 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-8txbg"] Nov 02 18:12:38.634088 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:38.634003 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:12:38.851975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:38.845588 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:39.927119 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:39.927085 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:40.369996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:40.358671 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="435cec96-4072-46ce-852a-d1faec26752f" path="/var/lib/kubelet/pods/435cec96-4072-46ce-852a-d1faec26752f/volumes" Nov 02 18:12:41.246098 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:41.238281 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:42.132980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:42.132435 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:43.375984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:43.374352 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:43.549132 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:43.549081 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:12:43.579146 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:43.579102 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:12:43.998115 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:43.998078 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:44.754354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:44.754316 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:45.429900 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:45.429847 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:46.299977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:46.294780 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:47.196518 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:47.196485 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:47.592486 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:12:47.592406 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:12:47.733676 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:47.733629 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:48.046162 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:48.046119 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:48.416941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:48.416852 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:49.098992 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:49.098938 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:49.837988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:49.837329 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:50.512701 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:50.512653 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:51.173257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:51.173202 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:51.967616 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:51.967576 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:52.598089 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:52.598052 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:53.330118 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:53.330077 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:54.036987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:54.036861 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:54.655151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:54.655103 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:55.411331 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:55.411296 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:56.121986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:56.121711 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:56.897093 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:56.878627 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:57.615974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:57.615494 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:58.307580 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:12:58.307543 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:00.055912 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:00.055851 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:00.441549 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:00.441277 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:01.199530 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:01.199489 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:01.629166 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:01.629088 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:02.054188 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.047495 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:02.116979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.116019 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p"] Nov 02 18:13:02.123976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.117631 5549 cpu_manager.go:414] "containerMap: removing container" podUID="026e7c96-2366-483a-ae9c-55a92607394c" containerName="istio-proxy" Nov 02 18:13:02.123976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.117660 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026e7c96-2366-483a-ae9c-55a92607394c" containerName="istio-proxy" Nov 02 18:13:02.123976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.117680 5549 cpu_manager.go:414] "containerMap: removing container" podUID="9a22d1c3-76ca-445d-ab56-052577f4fcff" containerName="istio-proxy" Nov 02 18:13:02.123976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.117687 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a22d1c3-76ca-445d-ab56-052577f4fcff" containerName="istio-proxy" Nov 02 18:13:02.123976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.117717 5549 cpu_manager.go:414] "containerMap: removing container" podUID="9a22d1c3-76ca-445d-ab56-052577f4fcff" containerName="istio-proxy" Nov 02 18:13:02.123976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.117725 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a22d1c3-76ca-445d-ab56-052577f4fcff" containerName="istio-proxy" Nov 02 18:13:02.123976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.117759 5549 cpu_manager.go:414] "containerMap: removing container" podUID="435cec96-4072-46ce-852a-d1faec26752f" containerName="istio-proxy" Nov 02 18:13:02.123976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.117766 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="435cec96-4072-46ce-852a-d1faec26752f" containerName="istio-proxy" Nov 02 18:13:02.123976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.118006 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a22d1c3-76ca-445d-ab56-052577f4fcff" containerName="istio-proxy" Nov 02 18:13:02.123976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.118026 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="435cec96-4072-46ce-852a-d1faec26752f" containerName="istio-proxy" Nov 02 18:13:02.123976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.118049 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="026e7c96-2366-483a-ae9c-55a92607394c" containerName="istio-proxy" Nov 02 18:13:02.139337 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.139306 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" Nov 02 18:13:02.211626 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.211577 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p"] Nov 02 18:13:02.240149 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.240115 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/792e50d4-e838-43e5-a5d6-499a523f9cbf-routing-overlay-volume\") pod \"payload-processing-external-model-56fb475f76-2gk4p\" (UID: \"792e50d4-e838-43e5-a5d6-499a523f9cbf\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" Nov 02 18:13:02.248113 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.248071 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/792e50d4-e838-43e5-a5d6-499a523f9cbf-plugins-config-volume\") pod \"payload-processing-external-model-56fb475f76-2gk4p\" (UID: \"792e50d4-e838-43e5-a5d6-499a523f9cbf\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" Nov 02 18:13:02.274519 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.269918 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg5p9\" (UniqueName: \"kubernetes.io/projected/792e50d4-e838-43e5-a5d6-499a523f9cbf-kube-api-access-jg5p9\") pod \"payload-processing-external-model-56fb475f76-2gk4p\" (UID: \"792e50d4-e838-43e5-a5d6-499a523f9cbf\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" Nov 02 18:13:02.274878 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.274838 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/792e50d4-e838-43e5-a5d6-499a523f9cbf-provider-credentials\") pod \"payload-processing-external-model-56fb475f76-2gk4p\" (UID: \"792e50d4-e838-43e5-a5d6-499a523f9cbf\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" Nov 02 18:13:02.275927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.275893 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:02.383069 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.382978 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg5p9\" (UniqueName: \"kubernetes.io/projected/792e50d4-e838-43e5-a5d6-499a523f9cbf-kube-api-access-jg5p9\") pod \"payload-processing-external-model-56fb475f76-2gk4p\" (UID: \"792e50d4-e838-43e5-a5d6-499a523f9cbf\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" Nov 02 18:13:02.383390 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.383362 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/792e50d4-e838-43e5-a5d6-499a523f9cbf-provider-credentials\") pod \"payload-processing-external-model-56fb475f76-2gk4p\" (UID: \"792e50d4-e838-43e5-a5d6-499a523f9cbf\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" Nov 02 18:13:02.383845 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.383821 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/792e50d4-e838-43e5-a5d6-499a523f9cbf-routing-overlay-volume\") pod \"payload-processing-external-model-56fb475f76-2gk4p\" (UID: \"792e50d4-e838-43e5-a5d6-499a523f9cbf\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" Nov 02 18:13:02.384030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.384013 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/792e50d4-e838-43e5-a5d6-499a523f9cbf-plugins-config-volume\") pod \"payload-processing-external-model-56fb475f76-2gk4p\" (UID: \"792e50d4-e838-43e5-a5d6-499a523f9cbf\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" Nov 02 18:13:02.392837 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.388794 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/792e50d4-e838-43e5-a5d6-499a523f9cbf-routing-overlay-volume\") pod \"payload-processing-external-model-56fb475f76-2gk4p\" (UID: \"792e50d4-e838-43e5-a5d6-499a523f9cbf\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" Nov 02 18:13:02.392837 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.389710 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/792e50d4-e838-43e5-a5d6-499a523f9cbf-plugins-config-volume\") pod \"payload-processing-external-model-56fb475f76-2gk4p\" (UID: \"792e50d4-e838-43e5-a5d6-499a523f9cbf\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" Nov 02 18:13:02.422303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.422225 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/792e50d4-e838-43e5-a5d6-499a523f9cbf-provider-credentials\") pod \"payload-processing-external-model-56fb475f76-2gk4p\" (UID: \"792e50d4-e838-43e5-a5d6-499a523f9cbf\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" Nov 02 18:13:02.424932 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.424031 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg5p9\" (UniqueName: \"kubernetes.io/projected/792e50d4-e838-43e5-a5d6-499a523f9cbf-kube-api-access-jg5p9\") pod \"payload-processing-external-model-56fb475f76-2gk4p\" (UID: \"792e50d4-e838-43e5-a5d6-499a523f9cbf\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" Nov 02 18:13:02.433289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.433254 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:02.585410 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:02.585325 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" Nov 02 18:13:03.066460 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:13:03.066424 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792e50d4_e838_43e5_a5d6_499a523f9cbf.slice/crio-334e04f3d8808ece6ca0cb99530544316bd0c61a1b4d5931e997c88706b739e6 WatchSource:0}: Error finding container 334e04f3d8808ece6ca0cb99530544316bd0c61a1b4d5931e997c88706b739e6: Status 404 returned error can't find the container with id 334e04f3d8808ece6ca0cb99530544316bd0c61a1b4d5931e997c88706b739e6 Nov 02 18:13:03.068875 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:03.068851 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p"] Nov 02 18:13:03.726582 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:03.726467 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:04.000916 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.000832 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" event={"ID":"792e50d4-e838-43e5-a5d6-499a523f9cbf","Type":"ContainerStarted","Data":"9d83bff412717c0b670028d4e57f2ee33de92c0b465714bb334580c12c297b75"} Nov 02 18:13:04.001134 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.000919 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" event={"ID":"792e50d4-e838-43e5-a5d6-499a523f9cbf","Type":"ContainerStarted","Data":"334e04f3d8808ece6ca0cb99530544316bd0c61a1b4d5931e997c88706b739e6"} Nov 02 18:13:04.001134 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.000941 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" Nov 02 18:13:04.023849 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.023800 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" podStartSLOduration=2.023787404 podStartE2EDuration="2.023787404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:13:04.019714512 +0000 UTC m=+2355.808220041" watchObservedRunningTime="2026-11-02 18:13:04.023787404 +0000 UTC m=+2355.812292961" Nov 02 18:13:04.055987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.055910 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:04.219823 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.219767 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:04.715304 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.715262 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p"] Nov 02 18:13:04.751849 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.751813 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7875b97c67-mgttk"] Nov 02 18:13:04.752945 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.752928 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a22d1c3-76ca-445d-ab56-052577f4fcff" containerName="istio-proxy" Nov 02 18:13:04.757429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.757409 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" Nov 02 18:13:04.763300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.763275 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ecf6b1cb54e8\"" Nov 02 18:13:04.765053 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.765031 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7875b97c67-mgttk"] Nov 02 18:13:04.887512 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.887449 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh74n\" (UniqueName: \"kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-kube-api-access-zh74n\") pod \"payload-processing-external-model-7875b97c67-mgttk\" (UID: \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" Nov 02 18:13:04.887725 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.887514 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c9849c6-280c-4fbc-87d7-b50a5b90a355-routing-overlay-volume\") pod \"payload-processing-external-model-7875b97c67-mgttk\" (UID: \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" Nov 02 18:13:04.887792 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.887767 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-provider-credentials\") pod \"payload-processing-external-model-7875b97c67-mgttk\" (UID: \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" Nov 02 18:13:04.887836 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.887818 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9849c6-280c-4fbc-87d7-b50a5b90a355-plugins-config-volume\") pod \"payload-processing-external-model-7875b97c67-mgttk\" (UID: \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" Nov 02 18:13:04.989118 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.989040 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9849c6-280c-4fbc-87d7-b50a5b90a355-plugins-config-volume\") pod \"payload-processing-external-model-7875b97c67-mgttk\" (UID: \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" Nov 02 18:13:04.989273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.989189 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh74n\" (UniqueName: \"kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-kube-api-access-zh74n\") pod \"payload-processing-external-model-7875b97c67-mgttk\" (UID: \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" Nov 02 18:13:04.989273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.989219 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c9849c6-280c-4fbc-87d7-b50a5b90a355-routing-overlay-volume\") pod \"payload-processing-external-model-7875b97c67-mgttk\" (UID: \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" Nov 02 18:13:04.989413 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.989396 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-provider-credentials\") pod \"payload-processing-external-model-7875b97c67-mgttk\" (UID: \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" Nov 02 18:13:04.989573 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:13:04.989552 5549 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ecf6b1cb54e8: secret "agc-cred-ecf6b1cb54e8" not found Nov 02 18:13:04.989657 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:13:04.989575 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7875b97c67-mgttk: secret "agc-cred-ecf6b1cb54e8" not found Nov 02 18:13:04.989657 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:13:04.989637 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-provider-credentials podName:7c9849c6-280c-4fbc-87d7-b50a5b90a355 nodeName:}" failed. No retries permitted until 2026-11-02 18:13:05.48961838 +0000 UTC m=+2357.278123903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-provider-credentials") pod "payload-processing-external-model-7875b97c67-mgttk" (UID: "7c9849c6-280c-4fbc-87d7-b50a5b90a355") : secret "agc-cred-ecf6b1cb54e8" not found Nov 02 18:13:04.989836 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.989812 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9849c6-280c-4fbc-87d7-b50a5b90a355-plugins-config-volume\") pod \"payload-processing-external-model-7875b97c67-mgttk\" (UID: \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" Nov 02 18:13:04.989889 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.989875 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c9849c6-280c-4fbc-87d7-b50a5b90a355-routing-overlay-volume\") pod \"payload-processing-external-model-7875b97c67-mgttk\" (UID: \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" Nov 02 18:13:04.999914 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:04.999883 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh74n\" (UniqueName: \"kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-kube-api-access-zh74n\") pod \"payload-processing-external-model-7875b97c67-mgttk\" (UID: \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" Nov 02 18:13:05.060175 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:05.060137 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:05.217976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:05.216922 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:05.512889 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:05.512725 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-provider-credentials\") pod \"payload-processing-external-model-7875b97c67-mgttk\" (UID: \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" Nov 02 18:13:05.513122 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:13:05.513080 5549 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ecf6b1cb54e8: secret "agc-cred-ecf6b1cb54e8" not found Nov 02 18:13:05.513122 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:13:05.513100 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7875b97c67-mgttk: secret "agc-cred-ecf6b1cb54e8" not found Nov 02 18:13:05.513266 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:13:05.513157 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-provider-credentials podName:7c9849c6-280c-4fbc-87d7-b50a5b90a355 nodeName:}" failed. No retries permitted until 2026-11-02 18:13:06.513139021 +0000 UTC m=+2358.301644546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-provider-credentials") pod "payload-processing-external-model-7875b97c67-mgttk" (UID: "7c9849c6-280c-4fbc-87d7-b50a5b90a355") : secret "agc-cred-ecf6b1cb54e8" not found Nov 02 18:13:05.703598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:05.703558 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7875b97c67-mgttk"] Nov 02 18:13:05.704091 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:13:05.704070 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" podUID="7c9849c6-280c-4fbc-87d7-b50a5b90a355" Nov 02 18:13:05.760029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:05.759999 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx"] Nov 02 18:13:05.766598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:05.766542 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" Nov 02 18:13:05.781781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:05.781755 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx"] Nov 02 18:13:05.924590 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:05.924477 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e5d6fb-2285-40d4-9751-b544fe09e818-plugins-config-volume\") pod \"payload-processing-external-model-56fb475f76-fvjmx\" (UID: \"c9e5d6fb-2285-40d4-9751-b544fe09e818\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" Nov 02 18:13:05.924590 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:05.924568 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9e5d6fb-2285-40d4-9751-b544fe09e818-routing-overlay-volume\") pod \"payload-processing-external-model-56fb475f76-fvjmx\" (UID: \"c9e5d6fb-2285-40d4-9751-b544fe09e818\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" Nov 02 18:13:05.924944 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:05.924716 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9e5d6fb-2285-40d4-9751-b544fe09e818-provider-credentials\") pod \"payload-processing-external-model-56fb475f76-fvjmx\" (UID: \"c9e5d6fb-2285-40d4-9751-b544fe09e818\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" Nov 02 18:13:05.925045 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:05.924996 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv5bk\" (UniqueName: \"kubernetes.io/projected/c9e5d6fb-2285-40d4-9751-b544fe09e818-kube-api-access-zv5bk\") pod \"payload-processing-external-model-56fb475f76-fvjmx\" (UID: \"c9e5d6fb-2285-40d4-9751-b544fe09e818\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" Nov 02 18:13:05.930829 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:05.930580 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:06.019210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.019126 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" Nov 02 18:13:06.027312 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.027277 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e5d6fb-2285-40d4-9751-b544fe09e818-plugins-config-volume\") pod \"payload-processing-external-model-56fb475f76-fvjmx\" (UID: \"c9e5d6fb-2285-40d4-9751-b544fe09e818\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" Nov 02 18:13:06.027486 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.027360 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9e5d6fb-2285-40d4-9751-b544fe09e818-routing-overlay-volume\") pod \"payload-processing-external-model-56fb475f76-fvjmx\" (UID: \"c9e5d6fb-2285-40d4-9751-b544fe09e818\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" Nov 02 18:13:06.027546 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.027530 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9e5d6fb-2285-40d4-9751-b544fe09e818-provider-credentials\") pod \"payload-processing-external-model-56fb475f76-fvjmx\" (UID: \"c9e5d6fb-2285-40d4-9751-b544fe09e818\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" Nov 02 18:13:06.027829 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.027808 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv5bk\" (UniqueName: \"kubernetes.io/projected/c9e5d6fb-2285-40d4-9751-b544fe09e818-kube-api-access-zv5bk\") pod \"payload-processing-external-model-56fb475f76-fvjmx\" (UID: \"c9e5d6fb-2285-40d4-9751-b544fe09e818\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" Nov 02 18:13:06.028336 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.028312 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9e5d6fb-2285-40d4-9751-b544fe09e818-routing-overlay-volume\") pod \"payload-processing-external-model-56fb475f76-fvjmx\" (UID: \"c9e5d6fb-2285-40d4-9751-b544fe09e818\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" Nov 02 18:13:06.028473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.028451 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e5d6fb-2285-40d4-9751-b544fe09e818-plugins-config-volume\") pod \"payload-processing-external-model-56fb475f76-fvjmx\" (UID: \"c9e5d6fb-2285-40d4-9751-b544fe09e818\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" Nov 02 18:13:06.029886 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.029864 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" Nov 02 18:13:06.030310 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.030286 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9e5d6fb-2285-40d4-9751-b544fe09e818-provider-credentials\") pod \"payload-processing-external-model-56fb475f76-fvjmx\" (UID: \"c9e5d6fb-2285-40d4-9751-b544fe09e818\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" Nov 02 18:13:06.039300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.039271 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv5bk\" (UniqueName: \"kubernetes.io/projected/c9e5d6fb-2285-40d4-9751-b544fe09e818-kube-api-access-zv5bk\") pod \"payload-processing-external-model-56fb475f76-fvjmx\" (UID: \"c9e5d6fb-2285-40d4-9751-b544fe09e818\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" Nov 02 18:13:06.080597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.080563 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:06.086664 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.086630 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" Nov 02 18:13:06.128853 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.128816 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c9849c6-280c-4fbc-87d7-b50a5b90a355-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9849c6-280c-4fbc-87d7-b50a5b90a355-plugins-config-volume\") pod \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\" (UID: \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\") " Nov 02 18:13:06.129201 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.129174 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c9849c6-280c-4fbc-87d7-b50a5b90a355-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c9849c6-280c-4fbc-87d7-b50a5b90a355-routing-overlay-volume\") pod \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\" (UID: \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\") " Nov 02 18:13:06.129304 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.129233 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-kube-api-access-zh74n\" (UniqueName: \"kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-kube-api-access-zh74n\") pod \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\" (UID: \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\") " Nov 02 18:13:06.129741 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.129715 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9849c6-280c-4fbc-87d7-b50a5b90a355-plugins-config-volume" pod "7c9849c6-280c-4fbc-87d7-b50a5b90a355" (UID: "7c9849c6-280c-4fbc-87d7-b50a5b90a355"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:06.129854 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.129819 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9849c6-280c-4fbc-87d7-b50a5b90a355-routing-overlay-volume" pod "7c9849c6-280c-4fbc-87d7-b50a5b90a355" (UID: "7c9849c6-280c-4fbc-87d7-b50a5b90a355"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:06.130171 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.130151 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c9849c6-280c-4fbc-87d7-b50a5b90a355-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:06.130243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.130177 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9849c6-280c-4fbc-87d7-b50a5b90a355-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:06.132362 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.132339 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-kube-api-access-zh74n" pod "7c9849c6-280c-4fbc-87d7-b50a5b90a355" (UID: "7c9849c6-280c-4fbc-87d7-b50a5b90a355"). InnerVolumeSpecName "kube-api-access-zh74n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:06.259091 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.244787 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:06.302086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.301999 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh74n\" (UniqueName: \"kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-kube-api-access-zh74n\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:06.605292 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.595866 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-provider-credentials\") pod \"payload-processing-external-model-7875b97c67-mgttk\" (UID: \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" Nov 02 18:13:06.632632 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.631663 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-provider-credentials\") pod \"payload-processing-external-model-7875b97c67-mgttk\" (UID: \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" Nov 02 18:13:06.632632 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.632214 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx"] Nov 02 18:13:06.635902 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:13:06.635873 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e5d6fb_2285_40d4_9751_b544fe09e818.slice/crio-4356ec6c86f1141a39bb66704e804d548de4dbfd1ce56d116513a9e7231a1b62 WatchSource:0}: Error finding container 4356ec6c86f1141a39bb66704e804d548de4dbfd1ce56d116513a9e7231a1b62: Status 404 returned error can't find the container with id 4356ec6c86f1141a39bb66704e804d548de4dbfd1ce56d116513a9e7231a1b62 Nov 02 18:13:06.707216 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.707184 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-provider-credentials\") pod \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\" (UID: \"7c9849c6-280c-4fbc-87d7-b50a5b90a355\") " Nov 02 18:13:06.709223 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.709196 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-provider-credentials" pod "7c9849c6-280c-4fbc-87d7-b50a5b90a355" (UID: "7c9849c6-280c-4fbc-87d7-b50a5b90a355"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:06.809833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.809801 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c9849c6-280c-4fbc-87d7-b50a5b90a355-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:06.817811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.817777 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx"] Nov 02 18:13:06.852080 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.852045 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4"] Nov 02 18:13:06.857486 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.857417 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" Nov 02 18:13:06.867345 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.867318 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4"] Nov 02 18:13:06.911595 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.911560 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80d29792-af25-4e27-8f51-07547397be40-provider-credentials\") pod \"payload-processing-external-model-7875b97c67-r8hd4\" (UID: \"80d29792-af25-4e27-8f51-07547397be40\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" Nov 02 18:13:06.911764 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.911644 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80d29792-af25-4e27-8f51-07547397be40-plugins-config-volume\") pod \"payload-processing-external-model-7875b97c67-r8hd4\" (UID: \"80d29792-af25-4e27-8f51-07547397be40\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" Nov 02 18:13:06.911928 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.911903 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80d29792-af25-4e27-8f51-07547397be40-routing-overlay-volume\") pod \"payload-processing-external-model-7875b97c67-r8hd4\" (UID: \"80d29792-af25-4e27-8f51-07547397be40\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" Nov 02 18:13:06.912021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:06.912005 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89rg8\" (UniqueName: \"kubernetes.io/projected/80d29792-af25-4e27-8f51-07547397be40-kube-api-access-89rg8\") pod \"payload-processing-external-model-7875b97c67-r8hd4\" (UID: \"80d29792-af25-4e27-8f51-07547397be40\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" Nov 02 18:13:07.013257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:07.013220 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80d29792-af25-4e27-8f51-07547397be40-provider-credentials\") pod \"payload-processing-external-model-7875b97c67-r8hd4\" (UID: \"80d29792-af25-4e27-8f51-07547397be40\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" Nov 02 18:13:07.013443 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:07.013312 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80d29792-af25-4e27-8f51-07547397be40-plugins-config-volume\") pod \"payload-processing-external-model-7875b97c67-r8hd4\" (UID: \"80d29792-af25-4e27-8f51-07547397be40\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" Nov 02 18:13:07.013443 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:07.013419 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80d29792-af25-4e27-8f51-07547397be40-routing-overlay-volume\") pod \"payload-processing-external-model-7875b97c67-r8hd4\" (UID: \"80d29792-af25-4e27-8f51-07547397be40\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" Nov 02 18:13:07.013556 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:07.013450 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89rg8\" (UniqueName: \"kubernetes.io/projected/80d29792-af25-4e27-8f51-07547397be40-kube-api-access-89rg8\") pod \"payload-processing-external-model-7875b97c67-r8hd4\" (UID: \"80d29792-af25-4e27-8f51-07547397be40\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" Nov 02 18:13:07.014105 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:07.014076 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80d29792-af25-4e27-8f51-07547397be40-routing-overlay-volume\") pod \"payload-processing-external-model-7875b97c67-r8hd4\" (UID: \"80d29792-af25-4e27-8f51-07547397be40\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" Nov 02 18:13:07.014271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:07.014114 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80d29792-af25-4e27-8f51-07547397be40-plugins-config-volume\") pod \"payload-processing-external-model-7875b97c67-r8hd4\" (UID: \"80d29792-af25-4e27-8f51-07547397be40\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" Nov 02 18:13:07.015856 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:07.015836 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80d29792-af25-4e27-8f51-07547397be40-provider-credentials\") pod \"payload-processing-external-model-7875b97c67-r8hd4\" (UID: \"80d29792-af25-4e27-8f51-07547397be40\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" Nov 02 18:13:07.022746 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:07.022726 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89rg8\" (UniqueName: \"kubernetes.io/projected/80d29792-af25-4e27-8f51-07547397be40-kube-api-access-89rg8\") pod \"payload-processing-external-model-7875b97c67-r8hd4\" (UID: \"80d29792-af25-4e27-8f51-07547397be40\") " pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" Nov 02 18:13:07.028966 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:07.028929 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" event={"ID":"c9e5d6fb-2285-40d4-9751-b544fe09e818","Type":"ContainerStarted","Data":"7a971d673fee34b1da87b00e2e651e45d8bea11f84424da7ca29d73af628bcfc"} Nov 02 18:13:07.029077 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:07.029033 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" Nov 02 18:13:07.029077 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:07.029057 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" event={"ID":"c9e5d6fb-2285-40d4-9751-b544fe09e818","Type":"ContainerStarted","Data":"4356ec6c86f1141a39bb66704e804d548de4dbfd1ce56d116513a9e7231a1b62"} Nov 02 18:13:07.029077 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:07.029065 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7875b97c67-mgttk" Nov 02 18:13:07.052418 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:07.052376 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" podStartSLOduration=2.052362603 podStartE2EDuration="2.052362603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:13:07.047438374 +0000 UTC m=+2358.835943902" watchObservedRunningTime="2026-11-02 18:13:07.052362603 +0000 UTC m=+2358.840868131" Nov 02 18:13:07.086494 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:07.086460 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7875b97c67-mgttk"] Nov 02 18:13:07.093621 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:07.093589 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7875b97c67-mgttk"] Nov 02 18:13:07.195080 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:07.195054 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" Nov 02 18:13:07.388991 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:07.386343 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:07.487698 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:13:07.487661 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d29792_af25_4e27_8f51_07547397be40.slice/crio-8b35ad81dd6d02b87ba58ab53386ad09e2af6d28dba090807346ab8ce8b45b51 WatchSource:0}: Error finding container 8b35ad81dd6d02b87ba58ab53386ad09e2af6d28dba090807346ab8ce8b45b51: Status 404 returned error can't find the container with id 8b35ad81dd6d02b87ba58ab53386ad09e2af6d28dba090807346ab8ce8b45b51 Nov 02 18:13:07.487875 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:07.487853 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4"] Nov 02 18:13:08.038631 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:08.038593 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" event={"ID":"80d29792-af25-4e27-8f51-07547397be40","Type":"ContainerStarted","Data":"9db22a2b60ba4e03c49a26451a095ffa1346eeff0e42047dc4a1fec243d41f0f"} Nov 02 18:13:08.039166 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:08.038652 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" event={"ID":"80d29792-af25-4e27-8f51-07547397be40","Type":"ContainerStarted","Data":"8b35ad81dd6d02b87ba58ab53386ad09e2af6d28dba090807346ab8ce8b45b51"} Nov 02 18:13:08.039166 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:08.038678 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" Nov 02 18:13:08.081417 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:08.081358 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" podStartSLOduration=2.081343144 podStartE2EDuration="2.081343144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:13:08.080564086 +0000 UTC m=+2359.869069615" watchObservedRunningTime="2026-11-02 18:13:08.081343144 +0000 UTC m=+2359.869848706" Nov 02 18:13:08.131986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:08.128664 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:08.323048 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:08.322936 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c9849c6-280c-4fbc-87d7-b50a5b90a355" path="/var/lib/kubelet/pods/7c9849c6-280c-4fbc-87d7-b50a5b90a355/volumes" Nov 02 18:13:09.043987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:09.042877 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:09.719978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:09.713032 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:10.432764 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:10.432677 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:11.019350 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:11.019314 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" podUID="792e50d4-e838-43e5-a5d6-499a523f9cbf" containerName="payload-processing" containerID="cri-o://9d83bff412717c0b670028d4e57f2ee33de92c0b465714bb334580c12c297b75" gracePeriod=25 Nov 02 18:13:11.249532 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:11.243870 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:11.381886 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:11.381855 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" Nov 02 18:13:11.526277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:11.526247 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/792e50d4-e838-43e5-a5d6-499a523f9cbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/792e50d4-e838-43e5-a5d6-499a523f9cbf-plugins-config-volume\") pod \"792e50d4-e838-43e5-a5d6-499a523f9cbf\" (UID: \"792e50d4-e838-43e5-a5d6-499a523f9cbf\") " Nov 02 18:13:11.526825 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:11.526349 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/792e50d4-e838-43e5-a5d6-499a523f9cbf-kube-api-access-jg5p9\" (UniqueName: \"kubernetes.io/projected/792e50d4-e838-43e5-a5d6-499a523f9cbf-kube-api-access-jg5p9\") pod \"792e50d4-e838-43e5-a5d6-499a523f9cbf\" (UID: \"792e50d4-e838-43e5-a5d6-499a523f9cbf\") " Nov 02 18:13:11.526825 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:11.526369 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/792e50d4-e838-43e5-a5d6-499a523f9cbf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/792e50d4-e838-43e5-a5d6-499a523f9cbf-provider-credentials\") pod \"792e50d4-e838-43e5-a5d6-499a523f9cbf\" (UID: \"792e50d4-e838-43e5-a5d6-499a523f9cbf\") " Nov 02 18:13:11.526825 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:11.526398 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/792e50d4-e838-43e5-a5d6-499a523f9cbf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/792e50d4-e838-43e5-a5d6-499a523f9cbf-routing-overlay-volume\") pod \"792e50d4-e838-43e5-a5d6-499a523f9cbf\" (UID: \"792e50d4-e838-43e5-a5d6-499a523f9cbf\") " Nov 02 18:13:11.526825 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:11.526740 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792e50d4-e838-43e5-a5d6-499a523f9cbf-plugins-config-volume" pod "792e50d4-e838-43e5-a5d6-499a523f9cbf" (UID: "792e50d4-e838-43e5-a5d6-499a523f9cbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:11.526825 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:11.526814 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792e50d4-e838-43e5-a5d6-499a523f9cbf-routing-overlay-volume" pod "792e50d4-e838-43e5-a5d6-499a523f9cbf" (UID: "792e50d4-e838-43e5-a5d6-499a523f9cbf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:11.527102 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:11.527061 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/792e50d4-e838-43e5-a5d6-499a523f9cbf-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:11.527102 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:11.527077 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/792e50d4-e838-43e5-a5d6-499a523f9cbf-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:11.528926 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:11.528895 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792e50d4-e838-43e5-a5d6-499a523f9cbf-kube-api-access-jg5p9" pod "792e50d4-e838-43e5-a5d6-499a523f9cbf" (UID: "792e50d4-e838-43e5-a5d6-499a523f9cbf"). InnerVolumeSpecName "kube-api-access-jg5p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:11.529074 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:11.528983 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792e50d4-e838-43e5-a5d6-499a523f9cbf-provider-credentials" pod "792e50d4-e838-43e5-a5d6-499a523f9cbf" (UID: "792e50d4-e838-43e5-a5d6-499a523f9cbf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:11.629147 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:11.629067 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg5p9\" (UniqueName: \"kubernetes.io/projected/792e50d4-e838-43e5-a5d6-499a523f9cbf-kube-api-access-jg5p9\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:11.629147 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:11.629098 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/792e50d4-e838-43e5-a5d6-499a523f9cbf-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:12.006454 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.006380 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:12.091867 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.091835 5549 generic.go:356] "Generic (PLEG): container finished" podID="792e50d4-e838-43e5-a5d6-499a523f9cbf" containerID="9d83bff412717c0b670028d4e57f2ee33de92c0b465714bb334580c12c297b75" exitCode=0 Nov 02 18:13:12.092092 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.091882 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" Nov 02 18:13:12.092092 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.091915 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" event={"ID":"792e50d4-e838-43e5-a5d6-499a523f9cbf","Type":"ContainerDied","Data":"9d83bff412717c0b670028d4e57f2ee33de92c0b465714bb334580c12c297b75"} Nov 02 18:13:12.092092 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.091992 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p" event={"ID":"792e50d4-e838-43e5-a5d6-499a523f9cbf","Type":"ContainerDied","Data":"334e04f3d8808ece6ca0cb99530544316bd0c61a1b4d5931e997c88706b739e6"} Nov 02 18:13:12.092092 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.092016 5549 scope.go:122] "RemoveContainer" containerID="9d83bff412717c0b670028d4e57f2ee33de92c0b465714bb334580c12c297b75" Nov 02 18:13:12.104901 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.104881 5549 scope.go:122] "RemoveContainer" containerID="9d83bff412717c0b670028d4e57f2ee33de92c0b465714bb334580c12c297b75" Nov 02 18:13:12.105230 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:13:12.105209 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d83bff412717c0b670028d4e57f2ee33de92c0b465714bb334580c12c297b75\": container with ID starting with 9d83bff412717c0b670028d4e57f2ee33de92c0b465714bb334580c12c297b75 not found: ID does not exist" containerID="9d83bff412717c0b670028d4e57f2ee33de92c0b465714bb334580c12c297b75" Nov 02 18:13:12.105298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.105237 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d83bff412717c0b670028d4e57f2ee33de92c0b465714bb334580c12c297b75"} err="failed to get container status \"9d83bff412717c0b670028d4e57f2ee33de92c0b465714bb334580c12c297b75\": rpc error: code = NotFound desc = could not find container \"9d83bff412717c0b670028d4e57f2ee33de92c0b465714bb334580c12c297b75\": container with ID starting with 9d83bff412717c0b670028d4e57f2ee33de92c0b465714bb334580c12c297b75 not found: ID does not exist" Nov 02 18:13:12.118113 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.118086 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p"] Nov 02 18:13:12.122322 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.122302 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fb475f76-2gk4p"] Nov 02 18:13:12.331741 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.331661 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="792e50d4-e838-43e5-a5d6-499a523f9cbf" path="/var/lib/kubelet/pods/792e50d4-e838-43e5-a5d6-499a523f9cbf/volumes" Nov 02 18:13:12.405926 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.405891 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:12.561108 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.561072 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:12.671571 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.671472 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4"] Nov 02 18:13:12.681376 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.681331 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-56fb475f76-m7d62"] Nov 02 18:13:12.683480 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.683312 5549 cpu_manager.go:414] "containerMap: removing container" podUID="792e50d4-e838-43e5-a5d6-499a523f9cbf" containerName="payload-processing" Nov 02 18:13:12.683682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.683663 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="792e50d4-e838-43e5-a5d6-499a523f9cbf" containerName="payload-processing" Nov 02 18:13:12.684096 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.684078 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="792e50d4-e838-43e5-a5d6-499a523f9cbf" containerName="payload-processing" Nov 02 18:13:12.715856 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.715823 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" Nov 02 18:13:12.716148 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.716128 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fb475f76-m7d62"] Nov 02 18:13:12.716308 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.715901 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" Nov 02 18:13:12.848743 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.848701 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a34c22-53fc-4da2-a5ae-60d86f7160e7-plugins-config-volume\") pod \"payload-processing-external-model-56fb475f76-m7d62\" (UID: \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" Nov 02 18:13:12.848966 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.848909 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46a34c22-53fc-4da2-a5ae-60d86f7160e7-routing-overlay-volume\") pod \"payload-processing-external-model-56fb475f76-m7d62\" (UID: \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" Nov 02 18:13:12.849055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.848941 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46a34c22-53fc-4da2-a5ae-60d86f7160e7-provider-credentials\") pod \"payload-processing-external-model-56fb475f76-m7d62\" (UID: \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" Nov 02 18:13:12.849205 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.849174 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9m6m\" (UniqueName: \"kubernetes.io/projected/46a34c22-53fc-4da2-a5ae-60d86f7160e7-kube-api-access-l9m6m\") pod \"payload-processing-external-model-56fb475f76-m7d62\" (UID: \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" Nov 02 18:13:12.956975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.951034 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a34c22-53fc-4da2-a5ae-60d86f7160e7-plugins-config-volume\") pod \"payload-processing-external-model-56fb475f76-m7d62\" (UID: \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" Nov 02 18:13:12.956975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.951188 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46a34c22-53fc-4da2-a5ae-60d86f7160e7-routing-overlay-volume\") pod \"payload-processing-external-model-56fb475f76-m7d62\" (UID: \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" Nov 02 18:13:12.956975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.951221 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46a34c22-53fc-4da2-a5ae-60d86f7160e7-provider-credentials\") pod \"payload-processing-external-model-56fb475f76-m7d62\" (UID: \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" Nov 02 18:13:12.956975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.951356 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9m6m\" (UniqueName: \"kubernetes.io/projected/46a34c22-53fc-4da2-a5ae-60d86f7160e7-kube-api-access-l9m6m\") pod \"payload-processing-external-model-56fb475f76-m7d62\" (UID: \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" Nov 02 18:13:12.956975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.952816 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a34c22-53fc-4da2-a5ae-60d86f7160e7-plugins-config-volume\") pod \"payload-processing-external-model-56fb475f76-m7d62\" (UID: \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" Nov 02 18:13:12.983979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.975585 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46a34c22-53fc-4da2-a5ae-60d86f7160e7-routing-overlay-volume\") pod \"payload-processing-external-model-56fb475f76-m7d62\" (UID: \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" Nov 02 18:13:12.986373 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.984660 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46a34c22-53fc-4da2-a5ae-60d86f7160e7-provider-credentials\") pod \"payload-processing-external-model-56fb475f76-m7d62\" (UID: \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" Nov 02 18:13:12.993969 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:12.990914 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9m6m\" (UniqueName: \"kubernetes.io/projected/46a34c22-53fc-4da2-a5ae-60d86f7160e7-kube-api-access-l9m6m\") pod \"payload-processing-external-model-56fb475f76-m7d62\" (UID: \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\") " pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" Nov 02 18:13:13.037973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:13.037249 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" Nov 02 18:13:13.046558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:13.046264 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" podUID="c9e5d6fb-2285-40d4-9751-b544fe09e818" containerName="payload-processing" containerID="cri-o://7a971d673fee34b1da87b00e2e651e45d8bea11f84424da7ca29d73af628bcfc" gracePeriod=25 Nov 02 18:13:13.422287 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:13.422260 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" Nov 02 18:13:13.479242 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:13.479207 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e5d6fb-2285-40d4-9751-b544fe09e818-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9e5d6fb-2285-40d4-9751-b544fe09e818-routing-overlay-volume\") pod \"c9e5d6fb-2285-40d4-9751-b544fe09e818\" (UID: \"c9e5d6fb-2285-40d4-9751-b544fe09e818\") " Nov 02 18:13:13.479457 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:13.479270 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e5d6fb-2285-40d4-9751-b544fe09e818-kube-api-access-zv5bk\" (UniqueName: \"kubernetes.io/projected/c9e5d6fb-2285-40d4-9751-b544fe09e818-kube-api-access-zv5bk\") pod \"c9e5d6fb-2285-40d4-9751-b544fe09e818\" (UID: \"c9e5d6fb-2285-40d4-9751-b544fe09e818\") " Nov 02 18:13:13.479457 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:13.479382 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e5d6fb-2285-40d4-9751-b544fe09e818-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9e5d6fb-2285-40d4-9751-b544fe09e818-provider-credentials\") pod \"c9e5d6fb-2285-40d4-9751-b544fe09e818\" (UID: \"c9e5d6fb-2285-40d4-9751-b544fe09e818\") " Nov 02 18:13:13.479588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:13.479563 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e5d6fb-2285-40d4-9751-b544fe09e818-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e5d6fb-2285-40d4-9751-b544fe09e818-plugins-config-volume\") pod \"c9e5d6fb-2285-40d4-9751-b544fe09e818\" (UID: \"c9e5d6fb-2285-40d4-9751-b544fe09e818\") " Nov 02 18:13:13.479708 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:13.479670 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e5d6fb-2285-40d4-9751-b544fe09e818-routing-overlay-volume" pod "c9e5d6fb-2285-40d4-9751-b544fe09e818" (UID: "c9e5d6fb-2285-40d4-9751-b544fe09e818"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:13.480019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:13.479985 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e5d6fb-2285-40d4-9751-b544fe09e818-plugins-config-volume" pod "c9e5d6fb-2285-40d4-9751-b544fe09e818" (UID: "c9e5d6fb-2285-40d4-9751-b544fe09e818"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:13.480547 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:13.480506 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9e5d6fb-2285-40d4-9751-b544fe09e818-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:13.480630 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:13.480553 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e5d6fb-2285-40d4-9751-b544fe09e818-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:13.481718 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:13.481692 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e5d6fb-2285-40d4-9751-b544fe09e818-kube-api-access-zv5bk" pod "c9e5d6fb-2285-40d4-9751-b544fe09e818" (UID: "c9e5d6fb-2285-40d4-9751-b544fe09e818"). InnerVolumeSpecName "kube-api-access-zv5bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:13.482581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:13.482556 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e5d6fb-2285-40d4-9751-b544fe09e818-provider-credentials" pod "c9e5d6fb-2285-40d4-9751-b544fe09e818" (UID: "c9e5d6fb-2285-40d4-9751-b544fe09e818"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:13.492900 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:13:13.489918 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a34c22_53fc_4da2_a5ae_60d86f7160e7.slice/crio-e1991df61459ea6cddc901d914c0dc00e25dee75f9650bffefcb8f9218cc7b8a WatchSource:0}: Error finding container e1991df61459ea6cddc901d914c0dc00e25dee75f9650bffefcb8f9218cc7b8a: Status 404 returned error can't find the container with id e1991df61459ea6cddc901d914c0dc00e25dee75f9650bffefcb8f9218cc7b8a Nov 02 18:13:13.492900 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:13.492384 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fb475f76-m7d62"] Nov 02 18:13:13.510176 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:13.510120 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:13.598175 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:13.594894 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9e5d6fb-2285-40d4-9751-b544fe09e818-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:13.598175 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:13.594941 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv5bk\" (UniqueName: \"kubernetes.io/projected/c9e5d6fb-2285-40d4-9751-b544fe09e818-kube-api-access-zv5bk\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:13.986218 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:13.986174 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:14.208720 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.208672 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" event={"ID":"46a34c22-53fc-4da2-a5ae-60d86f7160e7","Type":"ContainerStarted","Data":"2bb2d9f3690bc8d871e60e3645ec88deea0a131e7230139832568a9717722c9d"} Nov 02 18:13:14.208898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.208740 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" event={"ID":"46a34c22-53fc-4da2-a5ae-60d86f7160e7","Type":"ContainerStarted","Data":"e1991df61459ea6cddc901d914c0dc00e25dee75f9650bffefcb8f9218cc7b8a"} Nov 02 18:13:14.208898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.208772 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" Nov 02 18:13:14.210209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.210183 5549 generic.go:356] "Generic (PLEG): container finished" podID="c9e5d6fb-2285-40d4-9751-b544fe09e818" containerID="7a971d673fee34b1da87b00e2e651e45d8bea11f84424da7ca29d73af628bcfc" exitCode=0 Nov 02 18:13:14.210368 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.210226 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" Nov 02 18:13:14.210368 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.210266 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" event={"ID":"c9e5d6fb-2285-40d4-9751-b544fe09e818","Type":"ContainerDied","Data":"7a971d673fee34b1da87b00e2e651e45d8bea11f84424da7ca29d73af628bcfc"} Nov 02 18:13:14.210368 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.210292 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx" event={"ID":"c9e5d6fb-2285-40d4-9751-b544fe09e818","Type":"ContainerDied","Data":"4356ec6c86f1141a39bb66704e804d548de4dbfd1ce56d116513a9e7231a1b62"} Nov 02 18:13:14.210368 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.210314 5549 scope.go:122] "RemoveContainer" containerID="7a971d673fee34b1da87b00e2e651e45d8bea11f84424da7ca29d73af628bcfc" Nov 02 18:13:14.224086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.224061 5549 scope.go:122] "RemoveContainer" containerID="7a971d673fee34b1da87b00e2e651e45d8bea11f84424da7ca29d73af628bcfc" Nov 02 18:13:14.224402 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:13:14.224376 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a971d673fee34b1da87b00e2e651e45d8bea11f84424da7ca29d73af628bcfc\": container with ID starting with 7a971d673fee34b1da87b00e2e651e45d8bea11f84424da7ca29d73af628bcfc not found: ID does not exist" containerID="7a971d673fee34b1da87b00e2e651e45d8bea11f84424da7ca29d73af628bcfc" Nov 02 18:13:14.224520 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.224407 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a971d673fee34b1da87b00e2e651e45d8bea11f84424da7ca29d73af628bcfc"} err="failed to get container status \"7a971d673fee34b1da87b00e2e651e45d8bea11f84424da7ca29d73af628bcfc\": rpc error: code = NotFound desc = could not find container \"7a971d673fee34b1da87b00e2e651e45d8bea11f84424da7ca29d73af628bcfc\": container with ID starting with 7a971d673fee34b1da87b00e2e651e45d8bea11f84424da7ca29d73af628bcfc not found: ID does not exist" Nov 02 18:13:14.234909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.234859 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" podStartSLOduration=2.234846984 podStartE2EDuration="2.234846984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:13:14.228589396 +0000 UTC m=+2366.017094926" watchObservedRunningTime="2026-11-02 18:13:14.234846984 +0000 UTC m=+2366.023352511" Nov 02 18:13:14.245711 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.245660 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx"] Nov 02 18:13:14.251314 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.251292 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fb475f76-fvjmx"] Nov 02 18:13:14.289768 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.289732 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8"] Nov 02 18:13:14.291207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.291183 5549 cpu_manager.go:414] "containerMap: removing container" podUID="c9e5d6fb-2285-40d4-9751-b544fe09e818" containerName="payload-processing" Nov 02 18:13:14.291207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.291205 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e5d6fb-2285-40d4-9751-b544fe09e818" containerName="payload-processing" Nov 02 18:13:14.291403 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.291363 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e5d6fb-2285-40d4-9751-b544fe09e818" containerName="payload-processing" Nov 02 18:13:14.296920 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.296900 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.302122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.301790 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-78383f-openshift-default-dockercfg-7556m\"" Nov 02 18:13:14.306702 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.306676 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8"] Nov 02 18:13:14.363236 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.363197 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e5d6fb-2285-40d4-9751-b544fe09e818" path="/var/lib/kubelet/pods/c9e5d6fb-2285-40d4-9751-b544fe09e818/volumes" Nov 02 18:13:14.375783 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.375753 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:14.408327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.408285 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-token\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.408487 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.408395 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-workload-certs\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.408487 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.408446 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-workload-socket\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.408605 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.408583 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-envoy\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.408689 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.408672 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istiod-ca-cert\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.408869 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.408849 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfzxb\" (UniqueName: \"kubernetes.io/projected/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-kube-api-access-lfzxb\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.408970 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.408907 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-credential-socket\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.409260 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.409241 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-podinfo\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.409336 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.409289 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-data\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.536023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.526946 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-podinfo\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.536023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.527057 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-data\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.536023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.527107 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-token\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.536023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.527175 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-workload-certs\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.536023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.527239 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-workload-socket\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.536023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.527322 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-envoy\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.536023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.527367 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istiod-ca-cert\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.536023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.527470 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfzxb\" (UniqueName: \"kubernetes.io/projected/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-kube-api-access-lfzxb\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.536023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.527538 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-credential-socket\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.536023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.528474 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-credential-socket\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.549983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.537701 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-workload-socket\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.575974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.550572 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-data\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.576226 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.552480 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istiod-ca-cert\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.576371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.553843 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-workload-certs\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.596902 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.596863 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:14.614975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.610568 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-token\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.629972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.626479 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-envoy\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.629972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.627079 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-podinfo\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.659980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.659897 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfzxb\" (UniqueName: \"kubernetes.io/projected/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-kube-api-access-lfzxb\") pod \"e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:14.774034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.772598 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:14.939667 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:14.939582 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:15.231884 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:15.231852 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8"] Nov 02 18:13:15.232056 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:13:15.232024 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8e221e_3c26_48d8_9a17_3fd88e6c99ef.slice/crio-704ce3c10bbee656b84d71314d0f5edbd12d79308042ebd03bb3cf0a7757397f WatchSource:0}: Error finding container 704ce3c10bbee656b84d71314d0f5edbd12d79308042ebd03bb3cf0a7757397f: Status 404 returned error can't find the container with id 704ce3c10bbee656b84d71314d0f5edbd12d79308042ebd03bb3cf0a7757397f Nov 02 18:13:15.235413 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:15.235374 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:13:15.235569 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:15.235457 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:13:15.235569 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:15.235502 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:13:15.325211 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:15.317422 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:16.000973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:15.995541 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:16.230666 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:16.230630 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" event={"ID":"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef","Type":"ContainerStarted","Data":"aac067ee553514a799a2413e2799512f06b8fd446137d39748b46c57248a089b"} Nov 02 18:13:16.230846 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:16.230691 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" event={"ID":"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef","Type":"ContainerStarted","Data":"704ce3c10bbee656b84d71314d0f5edbd12d79308042ebd03bb3cf0a7757397f"} Nov 02 18:13:16.259488 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:16.259387 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" podStartSLOduration=2.259375202 podStartE2EDuration="2.259375202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:13:16.255812423 +0000 UTC m=+2368.044317953" watchObservedRunningTime="2026-11-02 18:13:16.259375202 +0000 UTC m=+2368.047880730" Nov 02 18:13:16.797981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:16.788657 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:16.939922 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:16.939889 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:16.947288 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:16.947260 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:17.239599 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:17.239568 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:17.240618 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:17.240594 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:13:17.516054 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:17.515808 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:17.599994 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:13:17.599927 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:13:17.672545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:17.672487 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" podUID="80d29792-af25-4e27-8f51-07547397be40" containerName="payload-processing" containerID="cri-o://9db22a2b60ba4e03c49a26451a095ffa1346eeff0e42047dc4a1fec243d41f0f" gracePeriod=25 Nov 02 18:13:17.992807 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:17.992779 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" Nov 02 18:13:18.130172 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.129901 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80d29792-af25-4e27-8f51-07547397be40-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80d29792-af25-4e27-8f51-07547397be40-routing-overlay-volume\") pod \"80d29792-af25-4e27-8f51-07547397be40\" (UID: \"80d29792-af25-4e27-8f51-07547397be40\") " Nov 02 18:13:18.133704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.133659 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80d29792-af25-4e27-8f51-07547397be40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80d29792-af25-4e27-8f51-07547397be40-plugins-config-volume\") pod \"80d29792-af25-4e27-8f51-07547397be40\" (UID: \"80d29792-af25-4e27-8f51-07547397be40\") " Nov 02 18:13:18.133704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.133733 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d29792-af25-4e27-8f51-07547397be40-kube-api-access-89rg8\" (UniqueName: \"kubernetes.io/projected/80d29792-af25-4e27-8f51-07547397be40-kube-api-access-89rg8\") pod \"80d29792-af25-4e27-8f51-07547397be40\" (UID: \"80d29792-af25-4e27-8f51-07547397be40\") " Nov 02 18:13:18.134122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.133815 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d29792-af25-4e27-8f51-07547397be40-routing-overlay-volume" pod "80d29792-af25-4e27-8f51-07547397be40" (UID: "80d29792-af25-4e27-8f51-07547397be40"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:18.134122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.133842 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d29792-af25-4e27-8f51-07547397be40-provider-credentials\" (UniqueName: \"kubernetes.io/projected/80d29792-af25-4e27-8f51-07547397be40-provider-credentials\") pod \"80d29792-af25-4e27-8f51-07547397be40\" (UID: \"80d29792-af25-4e27-8f51-07547397be40\") " Nov 02 18:13:18.134336 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.134308 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d29792-af25-4e27-8f51-07547397be40-plugins-config-volume" pod "80d29792-af25-4e27-8f51-07547397be40" (UID: "80d29792-af25-4e27-8f51-07547397be40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:18.134762 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.134737 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80d29792-af25-4e27-8f51-07547397be40-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:18.134762 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.134764 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80d29792-af25-4e27-8f51-07547397be40-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:18.136525 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.136504 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d29792-af25-4e27-8f51-07547397be40-kube-api-access-89rg8" pod "80d29792-af25-4e27-8f51-07547397be40" (UID: "80d29792-af25-4e27-8f51-07547397be40"). InnerVolumeSpecName "kube-api-access-89rg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:18.136599 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.136554 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d29792-af25-4e27-8f51-07547397be40-provider-credentials" pod "80d29792-af25-4e27-8f51-07547397be40" (UID: "80d29792-af25-4e27-8f51-07547397be40"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:18.236026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.235988 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89rg8\" (UniqueName: \"kubernetes.io/projected/80d29792-af25-4e27-8f51-07547397be40-kube-api-access-89rg8\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:18.236026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.236023 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80d29792-af25-4e27-8f51-07547397be40-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:18.249261 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.249228 5549 generic.go:356] "Generic (PLEG): container finished" podID="80d29792-af25-4e27-8f51-07547397be40" containerID="9db22a2b60ba4e03c49a26451a095ffa1346eeff0e42047dc4a1fec243d41f0f" exitCode=0 Nov 02 18:13:18.249723 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.249278 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" Nov 02 18:13:18.249723 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.249323 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" event={"ID":"80d29792-af25-4e27-8f51-07547397be40","Type":"ContainerDied","Data":"9db22a2b60ba4e03c49a26451a095ffa1346eeff0e42047dc4a1fec243d41f0f"} Nov 02 18:13:18.249723 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.249428 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4" event={"ID":"80d29792-af25-4e27-8f51-07547397be40","Type":"ContainerDied","Data":"8b35ad81dd6d02b87ba58ab53386ad09e2af6d28dba090807346ab8ce8b45b51"} Nov 02 18:13:18.249723 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.249466 5549 scope.go:122] "RemoveContainer" containerID="9db22a2b60ba4e03c49a26451a095ffa1346eeff0e42047dc4a1fec243d41f0f" Nov 02 18:13:18.262375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.262354 5549 scope.go:122] "RemoveContainer" containerID="9db22a2b60ba4e03c49a26451a095ffa1346eeff0e42047dc4a1fec243d41f0f" Nov 02 18:13:18.262661 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:13:18.262639 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db22a2b60ba4e03c49a26451a095ffa1346eeff0e42047dc4a1fec243d41f0f\": container with ID starting with 9db22a2b60ba4e03c49a26451a095ffa1346eeff0e42047dc4a1fec243d41f0f not found: ID does not exist" containerID="9db22a2b60ba4e03c49a26451a095ffa1346eeff0e42047dc4a1fec243d41f0f" Nov 02 18:13:18.262710 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.262671 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db22a2b60ba4e03c49a26451a095ffa1346eeff0e42047dc4a1fec243d41f0f"} err="failed to get container status \"9db22a2b60ba4e03c49a26451a095ffa1346eeff0e42047dc4a1fec243d41f0f\": rpc error: code = NotFound desc = could not find container \"9db22a2b60ba4e03c49a26451a095ffa1346eeff0e42047dc4a1fec243d41f0f\": container with ID starting with 9db22a2b60ba4e03c49a26451a095ffa1346eeff0e42047dc4a1fec243d41f0f not found: ID does not exist" Nov 02 18:13:18.275390 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.275360 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4"] Nov 02 18:13:18.279587 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.279565 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7875b97c67-r8hd4"] Nov 02 18:13:18.353981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.344883 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:18.400044 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.396447 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80d29792-af25-4e27-8f51-07547397be40" path="/var/lib/kubelet/pods/80d29792-af25-4e27-8f51-07547397be40/volumes" Nov 02 18:13:18.534628 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.534593 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:18.705151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:18.705121 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:19.186033 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:19.183325 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:19.730027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:19.729996 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75"] Nov 02 18:13:19.730984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:19.730945 5549 cpu_manager.go:414] "containerMap: removing container" podUID="80d29792-af25-4e27-8f51-07547397be40" containerName="payload-processing" Nov 02 18:13:19.730984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:19.730974 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d29792-af25-4e27-8f51-07547397be40" containerName="payload-processing" Nov 02 18:13:19.731154 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:19.731144 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d29792-af25-4e27-8f51-07547397be40" containerName="payload-processing" Nov 02 18:13:19.736602 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:19.736582 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" Nov 02 18:13:19.748852 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:19.746933 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-78383f\"" Nov 02 18:13:19.748852 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:19.748095 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75"] Nov 02 18:13:19.769593 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:19.769562 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:19.788832 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:19.788792 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9cbadb5-683a-46a2-964a-7c144a467125-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75\" (UID: \"d9cbadb5-683a-46a2-964a-7c144a467125\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" Nov 02 18:13:19.789012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:19.788839 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqlxv\" (UniqueName: \"kubernetes.io/projected/d9cbadb5-683a-46a2-964a-7c144a467125-kube-api-access-rqlxv\") pod \"payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75\" (UID: \"d9cbadb5-683a-46a2-964a-7c144a467125\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" Nov 02 18:13:19.809006 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:19.808969 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx"] Nov 02 18:13:19.814178 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:19.814158 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" Nov 02 18:13:19.825641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:19.825438 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-78383f-dockercfg-5b2vx\"" Nov 02 18:13:19.827227 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:19.827202 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx"] Nov 02 18:13:20.022270 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:20.022179 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9cbadb5-683a-46a2-964a-7c144a467125-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75\" (UID: \"d9cbadb5-683a-46a2-964a-7c144a467125\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" Nov 02 18:13:20.108304 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:20.108260 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqlxv\" (UniqueName: \"kubernetes.io/projected/d9cbadb5-683a-46a2-964a-7c144a467125-kube-api-access-rqlxv\") pod \"payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75\" (UID: \"d9cbadb5-683a-46a2-964a-7c144a467125\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" Nov 02 18:13:20.108737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:20.108710 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9blzk\" (UniqueName: \"kubernetes.io/projected/ec19d984-fac8-4ec5-9666-d3c9d3505b11-kube-api-access-9blzk\") pod \"payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx\" (UID: \"ec19d984-fac8-4ec5-9666-d3c9d3505b11\") " pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" Nov 02 18:13:20.108936 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:20.108920 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec19d984-fac8-4ec5-9666-d3c9d3505b11-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx\" (UID: \"ec19d984-fac8-4ec5-9666-d3c9d3505b11\") " pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" Nov 02 18:13:20.109383 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:20.023364 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9cbadb5-683a-46a2-964a-7c144a467125-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75\" (UID: \"d9cbadb5-683a-46a2-964a-7c144a467125\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" Nov 02 18:13:20.159909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:20.159872 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:20.161548 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:20.161487 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqlxv\" (UniqueName: \"kubernetes.io/projected/d9cbadb5-683a-46a2-964a-7c144a467125-kube-api-access-rqlxv\") pod \"payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75\" (UID: \"d9cbadb5-683a-46a2-964a-7c144a467125\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" Nov 02 18:13:20.210711 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:20.210672 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec19d984-fac8-4ec5-9666-d3c9d3505b11-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx\" (UID: \"ec19d984-fac8-4ec5-9666-d3c9d3505b11\") " pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" Nov 02 18:13:20.211408 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:20.211375 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9blzk\" (UniqueName: \"kubernetes.io/projected/ec19d984-fac8-4ec5-9666-d3c9d3505b11-kube-api-access-9blzk\") pod \"payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx\" (UID: \"ec19d984-fac8-4ec5-9666-d3c9d3505b11\") " pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" Nov 02 18:13:20.211664 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:20.211643 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec19d984-fac8-4ec5-9666-d3c9d3505b11-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx\" (UID: \"ec19d984-fac8-4ec5-9666-d3c9d3505b11\") " pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" Nov 02 18:13:20.221307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:20.221287 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9blzk\" (UniqueName: \"kubernetes.io/projected/ec19d984-fac8-4ec5-9666-d3c9d3505b11-kube-api-access-9blzk\") pod \"payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx\" (UID: \"ec19d984-fac8-4ec5-9666-d3c9d3505b11\") " pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" Nov 02 18:13:20.354043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:20.353942 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" Nov 02 18:13:20.446210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:20.432512 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:20.446210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:20.435238 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" Nov 02 18:13:20.711466 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:13:20.711438 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9cbadb5_683a_46a2_964a_7c144a467125.slice/crio-02061ae7934592aa6f52394e92352d4582707a12990dfe9025aad29ac4a0af79 WatchSource:0}: Error finding container 02061ae7934592aa6f52394e92352d4582707a12990dfe9025aad29ac4a0af79: Status 404 returned error can't find the container with id 02061ae7934592aa6f52394e92352d4582707a12990dfe9025aad29ac4a0af79 Nov 02 18:13:20.711626 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:20.711602 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75"] Nov 02 18:13:20.769136 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:20.769107 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx"] Nov 02 18:13:20.770637 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:13:20.770607 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec19d984_fac8_4ec5_9666_d3c9d3505b11.slice/crio-4c902760f1d49b2121b8ead81626e122d78ef9c558dda4f40e8272810cb08c3d WatchSource:0}: Error finding container 4c902760f1d49b2121b8ead81626e122d78ef9c558dda4f40e8272810cb08c3d: Status 404 returned error can't find the container with id 4c902760f1d49b2121b8ead81626e122d78ef9c558dda4f40e8272810cb08c3d Nov 02 18:13:21.133107 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:21.128433 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:21.277109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:21.277069 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" event={"ID":"d9cbadb5-683a-46a2-964a-7c144a467125","Type":"ContainerStarted","Data":"042e9b81a513769b0d5c3da6f9a4ce20ad79243762b70cb2ed9069cc381179d4"} Nov 02 18:13:21.277328 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:21.277131 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" event={"ID":"d9cbadb5-683a-46a2-964a-7c144a467125","Type":"ContainerStarted","Data":"02061ae7934592aa6f52394e92352d4582707a12990dfe9025aad29ac4a0af79"} Nov 02 18:13:21.277328 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:21.277158 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" Nov 02 18:13:21.279046 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:21.279015 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" event={"ID":"ec19d984-fac8-4ec5-9666-d3c9d3505b11","Type":"ContainerStarted","Data":"cc6d3423bc09a5b4d73f8462a7adbd4aa45509e4715662717d5734a93a664e4e"} Nov 02 18:13:21.279189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:21.279062 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" event={"ID":"ec19d984-fac8-4ec5-9666-d3c9d3505b11","Type":"ContainerStarted","Data":"4c902760f1d49b2121b8ead81626e122d78ef9c558dda4f40e8272810cb08c3d"} Nov 02 18:13:21.279189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:21.279117 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" Nov 02 18:13:21.294281 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:21.294239 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" podStartSLOduration=2.294226731 podStartE2EDuration="2.294226731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:13:21.292142576 +0000 UTC m=+2373.080648104" watchObservedRunningTime="2026-11-02 18:13:21.294226731 +0000 UTC m=+2373.082732259" Nov 02 18:13:21.314828 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:21.314778 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" podStartSLOduration=2.314765193 podStartE2EDuration="2.314765193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:13:21.310569239 +0000 UTC m=+2373.099074768" watchObservedRunningTime="2026-11-02 18:13:21.314765193 +0000 UTC m=+2373.103270721" Nov 02 18:13:21.915123 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:21.910185 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:22.579679 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:22.579475 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:23.243557 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:23.243520 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:23.930418 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:23.930382 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:24.618976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:24.614881 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:25.224367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:25.224335 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" Nov 02 18:13:25.277428 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:25.277392 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75456797bd-d542n"] Nov 02 18:13:25.317344 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:25.317310 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:26.034978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:26.022628 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:26.731271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:26.731237 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:27.455979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:27.451286 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:28.124181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:28.124143 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:28.751569 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:28.751528 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:29.473172 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:29.471128 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:30.173978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:30.173264 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:30.277934 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:30.277874 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" podUID="51433153-d037-4839-b460-d447ac7f301a" containerName="payload-processing" containerID="cri-o://3640a97b9995ac1cf0f16437127d71cabeaf2c7201df6f6ad8ad3f7ac94f182a" gracePeriod=25 Nov 02 18:13:30.568673 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:30.568637 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" Nov 02 18:13:30.650070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:30.650038 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51433153-d037-4839-b460-d447ac7f301a-kube-api-access-f8gr6\" (UniqueName: \"kubernetes.io/projected/51433153-d037-4839-b460-d447ac7f301a-kube-api-access-f8gr6\") pod \"51433153-d037-4839-b460-d447ac7f301a\" (UID: \"51433153-d037-4839-b460-d447ac7f301a\") " Nov 02 18:13:30.650250 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:30.650102 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51433153-d037-4839-b460-d447ac7f301a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51433153-d037-4839-b460-d447ac7f301a-plugins-config-volume\") pod \"51433153-d037-4839-b460-d447ac7f301a\" (UID: \"51433153-d037-4839-b460-d447ac7f301a\") " Nov 02 18:13:30.650250 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:30.650241 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51433153-d037-4839-b460-d447ac7f301a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51433153-d037-4839-b460-d447ac7f301a-routing-overlay-volume\") pod \"51433153-d037-4839-b460-d447ac7f301a\" (UID: \"51433153-d037-4839-b460-d447ac7f301a\") " Nov 02 18:13:30.650399 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:30.650273 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51433153-d037-4839-b460-d447ac7f301a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/51433153-d037-4839-b460-d447ac7f301a-provider-credentials\") pod \"51433153-d037-4839-b460-d447ac7f301a\" (UID: \"51433153-d037-4839-b460-d447ac7f301a\") " Nov 02 18:13:30.650641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:30.650606 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51433153-d037-4839-b460-d447ac7f301a-plugins-config-volume" pod "51433153-d037-4839-b460-d447ac7f301a" (UID: "51433153-d037-4839-b460-d447ac7f301a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:30.650762 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:30.650647 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51433153-d037-4839-b460-d447ac7f301a-routing-overlay-volume" pod "51433153-d037-4839-b460-d447ac7f301a" (UID: "51433153-d037-4839-b460-d447ac7f301a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:30.651169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:30.651145 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51433153-d037-4839-b460-d447ac7f301a-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:30.651169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:30.651166 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51433153-d037-4839-b460-d447ac7f301a-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:30.652484 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:30.652455 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51433153-d037-4839-b460-d447ac7f301a-kube-api-access-f8gr6" pod "51433153-d037-4839-b460-d447ac7f301a" (UID: "51433153-d037-4839-b460-d447ac7f301a"). InnerVolumeSpecName "kube-api-access-f8gr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:30.652586 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:30.652510 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51433153-d037-4839-b460-d447ac7f301a-provider-credentials" pod "51433153-d037-4839-b460-d447ac7f301a" (UID: "51433153-d037-4839-b460-d447ac7f301a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:30.753240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:30.753168 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51433153-d037-4839-b460-d447ac7f301a-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:30.753240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:30.753196 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8gr6\" (UniqueName: \"kubernetes.io/projected/51433153-d037-4839-b460-d447ac7f301a-kube-api-access-f8gr6\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:30.821759 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:30.821670 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:31.382215 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:31.382182 5549 generic.go:356] "Generic (PLEG): container finished" podID="51433153-d037-4839-b460-d447ac7f301a" containerID="3640a97b9995ac1cf0f16437127d71cabeaf2c7201df6f6ad8ad3f7ac94f182a" exitCode=0 Nov 02 18:13:31.382663 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:31.382224 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" Nov 02 18:13:31.382663 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:31.382248 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" event={"ID":"51433153-d037-4839-b460-d447ac7f301a","Type":"ContainerDied","Data":"3640a97b9995ac1cf0f16437127d71cabeaf2c7201df6f6ad8ad3f7ac94f182a"} Nov 02 18:13:31.382663 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:31.382290 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75456797bd-d542n" event={"ID":"51433153-d037-4839-b460-d447ac7f301a","Type":"ContainerDied","Data":"7c2af09426b187a099acba0963317c71b85e0d2193c2e2487ece2ab927560c9c"} Nov 02 18:13:31.382663 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:31.382312 5549 scope.go:122] "RemoveContainer" containerID="3640a97b9995ac1cf0f16437127d71cabeaf2c7201df6f6ad8ad3f7ac94f182a" Nov 02 18:13:31.394814 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:31.394791 5549 scope.go:122] "RemoveContainer" containerID="3640a97b9995ac1cf0f16437127d71cabeaf2c7201df6f6ad8ad3f7ac94f182a" Nov 02 18:13:31.395150 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:13:31.395124 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3640a97b9995ac1cf0f16437127d71cabeaf2c7201df6f6ad8ad3f7ac94f182a\": container with ID starting with 3640a97b9995ac1cf0f16437127d71cabeaf2c7201df6f6ad8ad3f7ac94f182a not found: ID does not exist" containerID="3640a97b9995ac1cf0f16437127d71cabeaf2c7201df6f6ad8ad3f7ac94f182a" Nov 02 18:13:31.395252 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:31.395156 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3640a97b9995ac1cf0f16437127d71cabeaf2c7201df6f6ad8ad3f7ac94f182a"} err="failed to get container status \"3640a97b9995ac1cf0f16437127d71cabeaf2c7201df6f6ad8ad3f7ac94f182a\": rpc error: code = NotFound desc = could not find container \"3640a97b9995ac1cf0f16437127d71cabeaf2c7201df6f6ad8ad3f7ac94f182a\": container with ID starting with 3640a97b9995ac1cf0f16437127d71cabeaf2c7201df6f6ad8ad3f7ac94f182a not found: ID does not exist" Nov 02 18:13:31.408506 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:31.408480 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75456797bd-d542n"] Nov 02 18:13:31.412874 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:31.412854 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-75456797bd-d542n"] Nov 02 18:13:31.549355 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:31.548500 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:31.921031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:31.920995 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:32.051262 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:32.051231 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:32.180760 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:32.180682 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:32.290670 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:32.290641 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" Nov 02 18:13:32.290879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:32.290725 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" Nov 02 18:13:32.318285 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:32.318243 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:32.326411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:32.326384 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51433153-d037-4839-b460-d447ac7f301a" path="/var/lib/kubelet/pods/51433153-d037-4839-b460-d447ac7f301a/volumes" Nov 02 18:13:33.119076 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:33.118750 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:33.810514 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:33.810475 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:34.540164 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:34.540127 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:35.338289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:35.336457 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:35.960889 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:35.960852 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:36.637202 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:36.637169 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:36.771278 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:36.771245 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:36.912095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:36.912015 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:37.090143 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:37.090104 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:13:37.629307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:37.629274 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:38.387925 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:38.387889 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:39.325996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:39.324695 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:40.267711 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:40.267676 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:41.321977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:41.321589 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:42.720146 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:42.720106 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:43.401468 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:43.401427 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:44.180063 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:44.177809 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:44.948048 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:44.947616 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:45.616997 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:45.613843 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:46.429304 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:46.429267 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:47.144499 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:47.144458 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:47.590732 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:13:47.590697 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:13:47.869232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:47.869143 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:48.639984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:48.628053 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:49.258983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:49.256357 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:49.790086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:49.790057 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:13:49.918778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:49.918738 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:50.039272 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:50.039245 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:13:50.613910 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:50.613871 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:51.260555 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:51.260511 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:51.964980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:51.960441 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:52.638738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:52.638619 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:53.334533 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:53.334496 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:54.029194 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:54.029147 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:54.662946 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:54.662909 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:55.360984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:55.360621 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:55.590247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:55.590211 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg"] Nov 02 18:13:55.650182 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:55.650102 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:55.862973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:55.862362 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:56.580983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:56.571589 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:57.211540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:57.211497 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:57.987981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:57.986878 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:58.995132 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:58.995094 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:59.142650 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:59.142614 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:59.526165 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:59.526131 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:59.739544 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:59.739508 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:59.992688 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:13:59.992635 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:00.154105 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:00.154072 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:00.590615 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:00.590579 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" podUID="e9fd1072-34d9-4168-9389-d522ae521491" containerName="payload-processing" containerID="cri-o://db32b9f1661fcb7278674379e61de1e1b422630943df77c01267e5fd3dff8840" gracePeriod=25 Nov 02 18:14:00.842803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:00.842550 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:01.049698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.049391 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" Nov 02 18:14:01.094495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.094421 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9fd1072-34d9-4168-9389-d522ae521491-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fd1072-34d9-4168-9389-d522ae521491-plugins-config-volume\") pod \"e9fd1072-34d9-4168-9389-d522ae521491\" (UID: \"e9fd1072-34d9-4168-9389-d522ae521491\") " Nov 02 18:14:01.094743 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.094722 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9fd1072-34d9-4168-9389-d522ae521491-kube-api-access-lkknr\" (UniqueName: \"kubernetes.io/projected/e9fd1072-34d9-4168-9389-d522ae521491-kube-api-access-lkknr\") pod \"e9fd1072-34d9-4168-9389-d522ae521491\" (UID: \"e9fd1072-34d9-4168-9389-d522ae521491\") " Nov 02 18:14:01.094804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.094791 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9fd1072-34d9-4168-9389-d522ae521491-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9fd1072-34d9-4168-9389-d522ae521491-provider-credentials\") pod \"e9fd1072-34d9-4168-9389-d522ae521491\" (UID: \"e9fd1072-34d9-4168-9389-d522ae521491\") " Nov 02 18:14:01.094845 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.094831 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9fd1072-34d9-4168-9389-d522ae521491-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9fd1072-34d9-4168-9389-d522ae521491-routing-overlay-volume\") pod \"e9fd1072-34d9-4168-9389-d522ae521491\" (UID: \"e9fd1072-34d9-4168-9389-d522ae521491\") " Nov 02 18:14:01.094916 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.094848 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9fd1072-34d9-4168-9389-d522ae521491-plugins-config-volume" pod "e9fd1072-34d9-4168-9389-d522ae521491" (UID: "e9fd1072-34d9-4168-9389-d522ae521491"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:14:01.095385 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.095356 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9fd1072-34d9-4168-9389-d522ae521491-routing-overlay-volume" pod "e9fd1072-34d9-4168-9389-d522ae521491" (UID: "e9fd1072-34d9-4168-9389-d522ae521491"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:14:01.095633 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.095616 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9fd1072-34d9-4168-9389-d522ae521491-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:01.095723 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.095635 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fd1072-34d9-4168-9389-d522ae521491-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:01.102398 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.102366 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9fd1072-34d9-4168-9389-d522ae521491-kube-api-access-lkknr" pod "e9fd1072-34d9-4168-9389-d522ae521491" (UID: "e9fd1072-34d9-4168-9389-d522ae521491"). InnerVolumeSpecName "kube-api-access-lkknr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:14:01.103685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.103656 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9fd1072-34d9-4168-9389-d522ae521491-provider-credentials" pod "e9fd1072-34d9-4168-9389-d522ae521491" (UID: "e9fd1072-34d9-4168-9389-d522ae521491"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:14:01.131660 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.131626 5549 generic.go:356] "Generic (PLEG): container finished" podID="e9fd1072-34d9-4168-9389-d522ae521491" containerID="db32b9f1661fcb7278674379e61de1e1b422630943df77c01267e5fd3dff8840" exitCode=0 Nov 02 18:14:01.131800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.131688 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" Nov 02 18:14:01.131800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.131707 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" event={"ID":"e9fd1072-34d9-4168-9389-d522ae521491","Type":"ContainerDied","Data":"db32b9f1661fcb7278674379e61de1e1b422630943df77c01267e5fd3dff8840"} Nov 02 18:14:01.131800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.131782 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg" event={"ID":"e9fd1072-34d9-4168-9389-d522ae521491","Type":"ContainerDied","Data":"3672f80a0d29893796f3cf4cd205ba06e030dc8fcd92c44fe6d017e82cc82ad7"} Nov 02 18:14:01.132019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.131813 5549 scope.go:122] "RemoveContainer" containerID="db32b9f1661fcb7278674379e61de1e1b422630943df77c01267e5fd3dff8840" Nov 02 18:14:01.144574 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.144536 5549 scope.go:122] "RemoveContainer" containerID="db32b9f1661fcb7278674379e61de1e1b422630943df77c01267e5fd3dff8840" Nov 02 18:14:01.144880 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:01.144850 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db32b9f1661fcb7278674379e61de1e1b422630943df77c01267e5fd3dff8840\": container with ID starting with db32b9f1661fcb7278674379e61de1e1b422630943df77c01267e5fd3dff8840 not found: ID does not exist" containerID="db32b9f1661fcb7278674379e61de1e1b422630943df77c01267e5fd3dff8840" Nov 02 18:14:01.145022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.144888 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db32b9f1661fcb7278674379e61de1e1b422630943df77c01267e5fd3dff8840"} err="failed to get container status \"db32b9f1661fcb7278674379e61de1e1b422630943df77c01267e5fd3dff8840\": rpc error: code = NotFound desc = could not find container \"db32b9f1661fcb7278674379e61de1e1b422630943df77c01267e5fd3dff8840\": container with ID starting with db32b9f1661fcb7278674379e61de1e1b422630943df77c01267e5fd3dff8840 not found: ID does not exist" Nov 02 18:14:01.159566 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.159537 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg"] Nov 02 18:14:01.163795 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.163775 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-8685dz4wg"] Nov 02 18:14:01.196916 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.196889 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkknr\" (UniqueName: \"kubernetes.io/projected/e9fd1072-34d9-4168-9389-d522ae521491-kube-api-access-lkknr\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:01.196916 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.196914 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9fd1072-34d9-4168-9389-d522ae521491-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:01.548997 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.548208 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:01.785016 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:01.784988 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:02.322138 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:02.322102 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9fd1072-34d9-4168-9389-d522ae521491" path="/var/lib/kubelet/pods/e9fd1072-34d9-4168-9389-d522ae521491/volumes" Nov 02 18:14:02.544986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:02.536769 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:03.118300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:03.118263 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:03.262124 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:03.262092 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:03.458349 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:03.458314 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:04.323879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.323846 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:04.360637 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.360605 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr"] Nov 02 18:14:04.361572 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.361555 5549 cpu_manager.go:414] "containerMap: removing container" podUID="e9fd1072-34d9-4168-9389-d522ae521491" containerName="payload-processing" Nov 02 18:14:04.361638 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.361573 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9fd1072-34d9-4168-9389-d522ae521491" containerName="payload-processing" Nov 02 18:14:04.361638 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.361628 5549 cpu_manager.go:414] "containerMap: removing container" podUID="51433153-d037-4839-b460-d447ac7f301a" containerName="payload-processing" Nov 02 18:14:04.361638 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.361634 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51433153-d037-4839-b460-d447ac7f301a" containerName="payload-processing" Nov 02 18:14:04.361763 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.361753 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="51433153-d037-4839-b460-d447ac7f301a" containerName="payload-processing" Nov 02 18:14:04.361804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.361768 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9fd1072-34d9-4168-9389-d522ae521491" containerName="payload-processing" Nov 02 18:14:04.367195 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.367176 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" Nov 02 18:14:04.374521 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.374495 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-658b8c\"" Nov 02 18:14:04.374639 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.374591 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-658b8c-dockercfg-fck8w\"" Nov 02 18:14:04.377317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.376842 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr"] Nov 02 18:14:04.389924 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.386456 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr"] Nov 02 18:14:04.393760 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.393715 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" Nov 02 18:14:04.401175 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.401140 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr"] Nov 02 18:14:04.498991 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.485651 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnddj\" (UniqueName: \"kubernetes.io/projected/e0cea7d9-9796-4a08-bf44-93038137646d-kube-api-access-mnddj\") pod \"payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr\" (UID: \"e0cea7d9-9796-4a08-bf44-93038137646d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" Nov 02 18:14:04.531934 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.531637 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:04.553179 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.553145 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afb8728d-ba4e-4787-ac83-e9bbd591ef54-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr\" (UID: \"afb8728d-ba4e-4787-ac83-e9bbd591ef54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" Nov 02 18:14:04.559261 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.559223 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0cea7d9-9796-4a08-bf44-93038137646d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr\" (UID: \"e0cea7d9-9796-4a08-bf44-93038137646d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" Nov 02 18:14:04.559554 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.559521 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zstss\" (UniqueName: \"kubernetes.io/projected/afb8728d-ba4e-4787-ac83-e9bbd591ef54-kube-api-access-zstss\") pod \"payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr\" (UID: \"afb8728d-ba4e-4787-ac83-e9bbd591ef54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" Nov 02 18:14:04.667365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.665226 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnddj\" (UniqueName: \"kubernetes.io/projected/e0cea7d9-9796-4a08-bf44-93038137646d-kube-api-access-mnddj\") pod \"payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr\" (UID: \"e0cea7d9-9796-4a08-bf44-93038137646d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" Nov 02 18:14:04.667365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.665364 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afb8728d-ba4e-4787-ac83-e9bbd591ef54-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr\" (UID: \"afb8728d-ba4e-4787-ac83-e9bbd591ef54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" Nov 02 18:14:04.667365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.665428 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0cea7d9-9796-4a08-bf44-93038137646d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr\" (UID: \"e0cea7d9-9796-4a08-bf44-93038137646d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" Nov 02 18:14:04.667365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.665466 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zstss\" (UniqueName: \"kubernetes.io/projected/afb8728d-ba4e-4787-ac83-e9bbd591ef54-kube-api-access-zstss\") pod \"payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr\" (UID: \"afb8728d-ba4e-4787-ac83-e9bbd591ef54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" Nov 02 18:14:04.667365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.667119 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afb8728d-ba4e-4787-ac83-e9bbd591ef54-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr\" (UID: \"afb8728d-ba4e-4787-ac83-e9bbd591ef54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" Nov 02 18:14:04.681613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.667942 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0cea7d9-9796-4a08-bf44-93038137646d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr\" (UID: \"e0cea7d9-9796-4a08-bf44-93038137646d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" Nov 02 18:14:04.709053 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.708886 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zstss\" (UniqueName: \"kubernetes.io/projected/afb8728d-ba4e-4787-ac83-e9bbd591ef54-kube-api-access-zstss\") pod \"payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr\" (UID: \"afb8728d-ba4e-4787-ac83-e9bbd591ef54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" Nov 02 18:14:04.722744 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.722695 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnddj\" (UniqueName: \"kubernetes.io/projected/e0cea7d9-9796-4a08-bf44-93038137646d-kube-api-access-mnddj\") pod \"payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr\" (UID: \"e0cea7d9-9796-4a08-bf44-93038137646d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" Nov 02 18:14:04.723086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.723065 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" Nov 02 18:14:04.985069 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:04.985033 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" Nov 02 18:14:05.048485 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:05.048447 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr"] Nov 02 18:14:05.169905 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:05.169868 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" event={"ID":"e0cea7d9-9796-4a08-bf44-93038137646d","Type":"ContainerStarted","Data":"ab7704d95d63b9ed548eef8044db6b7e08f2ba3e8660f944c290beb6f987b6e6"} Nov 02 18:14:05.170095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:05.169925 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" event={"ID":"e0cea7d9-9796-4a08-bf44-93038137646d","Type":"ContainerStarted","Data":"cd75d4e1cef3feb47b350ab01095d3eec57070de667ce8275cf724b61f9a3d77"} Nov 02 18:14:05.170142 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:05.170114 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" Nov 02 18:14:05.189444 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:05.189394 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" podStartSLOduration=1.189379394 podStartE2EDuration="1.189379394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:05.188225877 +0000 UTC m=+2416.976731404" watchObservedRunningTime="2026-11-02 18:14:05.189379394 +0000 UTC m=+2416.977884922" Nov 02 18:14:05.223804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:05.223775 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:05.270422 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:05.270384 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr"] Nov 02 18:14:05.270706 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:14:05.270685 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb8728d_ba4e_4787_ac83_e9bbd591ef54.slice/crio-9e30c4e648f42f0af367c96a0ded4f3a3f8f166ea9db26c794506110a68ad6b5 WatchSource:0}: Error finding container 9e30c4e648f42f0af367c96a0ded4f3a3f8f166ea9db26c794506110a68ad6b5: Status 404 returned error can't find the container with id 9e30c4e648f42f0af367c96a0ded4f3a3f8f166ea9db26c794506110a68ad6b5 Nov 02 18:14:05.903641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:05.903592 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:06.179772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:06.179669 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" event={"ID":"afb8728d-ba4e-4787-ac83-e9bbd591ef54","Type":"ContainerStarted","Data":"90de73083bb22c6de76f897bb3ddd0fba72d90d0e72b3fb96df7e4b908cebd75"} Nov 02 18:14:06.179772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:06.179721 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" event={"ID":"afb8728d-ba4e-4787-ac83-e9bbd591ef54","Type":"ContainerStarted","Data":"9e30c4e648f42f0af367c96a0ded4f3a3f8f166ea9db26c794506110a68ad6b5"} Nov 02 18:14:06.180056 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:06.179989 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" Nov 02 18:14:06.203682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:06.203072 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" podStartSLOduration=2.203055884 podStartE2EDuration="2.203055884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:06.196657104 +0000 UTC m=+2417.985162630" watchObservedRunningTime="2026-11-02 18:14:06.203055884 +0000 UTC m=+2417.991561415" Nov 02 18:14:06.628857 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:06.622779 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:07.332069 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:07.332031 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz"] Nov 02 18:14:07.357706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:07.357670 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:07.492285 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:07.492238 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:08.012762 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:08.012725 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:08.586976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:08.584649 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:09.357982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:09.354091 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:10.405082 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:10.405039 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:11.355976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:11.350824 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:12.027505 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:12.024891 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:12.332621 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:12.332534 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" podUID="6ccd2ae1-67d1-495c-af49-b164aecb0d04" containerName="payload-processing" containerID="cri-o://d93cf380bd88d35b5b4ebfac0428f5e98b6f368ea77214898bdf5b3fd352db60" gracePeriod=25 Nov 02 18:14:12.700582 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:12.700141 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" Nov 02 18:14:12.757433 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:12.755228 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:12.852130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:12.845649 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ccd2ae1-67d1-495c-af49-b164aecb0d04-kube-api-access-qckk7\" (UniqueName: \"kubernetes.io/projected/6ccd2ae1-67d1-495c-af49-b164aecb0d04-kube-api-access-qckk7\") pod \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\" (UID: \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\") " Nov 02 18:14:12.852130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:12.845723 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ccd2ae1-67d1-495c-af49-b164aecb0d04-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ccd2ae1-67d1-495c-af49-b164aecb0d04-routing-overlay-volume\") pod \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\" (UID: \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\") " Nov 02 18:14:12.852130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:12.845896 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ccd2ae1-67d1-495c-af49-b164aecb0d04-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ccd2ae1-67d1-495c-af49-b164aecb0d04-provider-credentials\") pod \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\" (UID: \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\") " Nov 02 18:14:12.852130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:12.845937 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ccd2ae1-67d1-495c-af49-b164aecb0d04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ccd2ae1-67d1-495c-af49-b164aecb0d04-plugins-config-volume\") pod \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\" (UID: \"6ccd2ae1-67d1-495c-af49-b164aecb0d04\") " Nov 02 18:14:12.852130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:12.851364 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ccd2ae1-67d1-495c-af49-b164aecb0d04-routing-overlay-volume" pod "6ccd2ae1-67d1-495c-af49-b164aecb0d04" (UID: "6ccd2ae1-67d1-495c-af49-b164aecb0d04"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:14:12.852988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:12.852737 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ccd2ae1-67d1-495c-af49-b164aecb0d04-plugins-config-volume" pod "6ccd2ae1-67d1-495c-af49-b164aecb0d04" (UID: "6ccd2ae1-67d1-495c-af49-b164aecb0d04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:14:12.855499 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:12.855448 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ccd2ae1-67d1-495c-af49-b164aecb0d04-provider-credentials" pod "6ccd2ae1-67d1-495c-af49-b164aecb0d04" (UID: "6ccd2ae1-67d1-495c-af49-b164aecb0d04"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:14:12.857060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:12.857014 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ccd2ae1-67d1-495c-af49-b164aecb0d04-kube-api-access-qckk7" pod "6ccd2ae1-67d1-495c-af49-b164aecb0d04" (UID: "6ccd2ae1-67d1-495c-af49-b164aecb0d04"). InnerVolumeSpecName "kube-api-access-qckk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:14:12.948404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:12.948375 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ccd2ae1-67d1-495c-af49-b164aecb0d04-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:12.948404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:12.948402 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ccd2ae1-67d1-495c-af49-b164aecb0d04-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:12.948603 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:12.948415 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qckk7\" (UniqueName: \"kubernetes.io/projected/6ccd2ae1-67d1-495c-af49-b164aecb0d04-kube-api-access-qckk7\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:12.948603 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:12.948425 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ccd2ae1-67d1-495c-af49-b164aecb0d04-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:13.247154 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:13.247068 5549 generic.go:356] "Generic (PLEG): container finished" podID="6ccd2ae1-67d1-495c-af49-b164aecb0d04" containerID="d93cf380bd88d35b5b4ebfac0428f5e98b6f368ea77214898bdf5b3fd352db60" exitCode=0 Nov 02 18:14:13.247154 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:13.247123 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" Nov 02 18:14:13.247154 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:13.247148 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" event={"ID":"6ccd2ae1-67d1-495c-af49-b164aecb0d04","Type":"ContainerDied","Data":"d93cf380bd88d35b5b4ebfac0428f5e98b6f368ea77214898bdf5b3fd352db60"} Nov 02 18:14:13.247683 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:13.247201 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz" event={"ID":"6ccd2ae1-67d1-495c-af49-b164aecb0d04","Type":"ContainerDied","Data":"20341c9284b0972f02c6178093ea92154db5605c8501675ccf82b79c259b4d0c"} Nov 02 18:14:13.247683 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:13.247231 5549 scope.go:122] "RemoveContainer" containerID="d93cf380bd88d35b5b4ebfac0428f5e98b6f368ea77214898bdf5b3fd352db60" Nov 02 18:14:13.260082 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:13.260052 5549 scope.go:122] "RemoveContainer" containerID="d93cf380bd88d35b5b4ebfac0428f5e98b6f368ea77214898bdf5b3fd352db60" Nov 02 18:14:13.260391 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:13.260366 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93cf380bd88d35b5b4ebfac0428f5e98b6f368ea77214898bdf5b3fd352db60\": container with ID starting with d93cf380bd88d35b5b4ebfac0428f5e98b6f368ea77214898bdf5b3fd352db60 not found: ID does not exist" containerID="d93cf380bd88d35b5b4ebfac0428f5e98b6f368ea77214898bdf5b3fd352db60" Nov 02 18:14:13.260490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:13.260398 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93cf380bd88d35b5b4ebfac0428f5e98b6f368ea77214898bdf5b3fd352db60"} err="failed to get container status \"d93cf380bd88d35b5b4ebfac0428f5e98b6f368ea77214898bdf5b3fd352db60\": rpc error: code = NotFound desc = could not find container \"d93cf380bd88d35b5b4ebfac0428f5e98b6f368ea77214898bdf5b3fd352db60\": container with ID starting with d93cf380bd88d35b5b4ebfac0428f5e98b6f368ea77214898bdf5b3fd352db60 not found: ID does not exist" Nov 02 18:14:13.273562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:13.273537 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz"] Nov 02 18:14:13.277929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:13.277904 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-586d595wz"] Nov 02 18:14:13.405917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:13.404457 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:14.256060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:14.255754 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:14.333309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:14.333273 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ccd2ae1-67d1-495c-af49-b164aecb0d04" path="/var/lib/kubelet/pods/6ccd2ae1-67d1-495c-af49-b164aecb0d04/volumes" Nov 02 18:14:14.749812 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:14.748734 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:14.921148 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:14.921106 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:15.098054 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:15.097942 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:15.284908 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:15.284874 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:15.508648 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:15.508591 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:16.015024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.014992 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:16.175183 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.175102 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:16.183662 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.183433 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" Nov 02 18:14:16.764483 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.764432 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:16.814972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.814928 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r"] Nov 02 18:14:16.815999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.815981 5549 cpu_manager.go:414] "containerMap: removing container" podUID="6ccd2ae1-67d1-495c-af49-b164aecb0d04" containerName="payload-processing" Nov 02 18:14:16.815999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.815999 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ccd2ae1-67d1-495c-af49-b164aecb0d04" containerName="payload-processing" Nov 02 18:14:16.816175 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.816163 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ccd2ae1-67d1-495c-af49-b164aecb0d04" containerName="payload-processing" Nov 02 18:14:16.820138 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.820121 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" Nov 02 18:14:16.825534 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.825511 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"agc-cred-c8cfcf1ad238\"" Nov 02 18:14:16.825784 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.825768 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"payload-processing-external-model-e2e--69602eb2-dockercfg-v22cl\"" Nov 02 18:14:16.826030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.826017 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"kube-root-ca.crt\"" Nov 02 18:14:16.829296 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.829276 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"routing-overlay\"" Nov 02 18:14:16.829984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.829934 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"payload-processing-external-model-plugins-e2e-extmod-b-327608\"" Nov 02 18:14:16.839274 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.839244 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r"] Nov 02 18:14:16.892983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.887300 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r\" (UID: \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" Nov 02 18:14:16.892983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.887472 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c4wn\" (UniqueName: \"kubernetes.io/projected/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-kube-api-access-2c4wn\") pod \"payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r\" (UID: \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" Nov 02 18:14:16.892983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.887596 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r\" (UID: \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" Nov 02 18:14:16.892983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:16.887641 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r\" (UID: \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" Nov 02 18:14:17.008978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.006379 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r\" (UID: \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" Nov 02 18:14:17.008978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.006556 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c4wn\" (UniqueName: \"kubernetes.io/projected/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-kube-api-access-2c4wn\") pod \"payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r\" (UID: \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" Nov 02 18:14:17.008978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.006676 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r\" (UID: \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" Nov 02 18:14:17.008978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.006725 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r\" (UID: \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" Nov 02 18:14:17.023964 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.022993 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"openshift-service-ca.crt\"" Nov 02 18:14:17.039458 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.039419 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r\" (UID: \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" Nov 02 18:14:17.041123 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.041066 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r\" (UID: \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" Nov 02 18:14:17.044404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.042845 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r\" (UID: \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" Nov 02 18:14:17.062309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.061694 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c4wn\" (UniqueName: \"kubernetes.io/projected/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-kube-api-access-2c4wn\") pod \"payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r\" (UID: \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" Nov 02 18:14:17.146292 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.146248 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" Nov 02 18:14:17.193271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.193233 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" Nov 02 18:14:17.285237 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.285146 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:17.368042 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.368004 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks"] Nov 02 18:14:17.375420 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.375398 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" Nov 02 18:14:17.382413 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.382384 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"routing-overlay\"" Nov 02 18:14:17.382720 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.382677 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"kube-root-ca.crt\"" Nov 02 18:14:17.383059 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.382573 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"openshift-service-ca.crt\"" Nov 02 18:14:17.383570 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.383182 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"payload-processing-external-model-plugins-e2e-extmod-a-42d786\"" Nov 02 18:14:17.386080 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.386056 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks"] Nov 02 18:14:17.386332 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.386318 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"agc-cred-e222b9ddb927\"" Nov 02 18:14:17.386845 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.386729 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-42d786\"/\"payload-processing-external-model-e2e--cd90e2f4-dockercfg-cj4pn\"" Nov 02 18:14:17.450297 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.450264 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6gzf\" (UniqueName: \"kubernetes.io/projected/a6e9e87c-d4b0-4191-990a-56a9531c627e-kube-api-access-d6gzf\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks\" (UID: \"a6e9e87c-d4b0-4191-990a-56a9531c627e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" Nov 02 18:14:17.450555 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.450531 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e9e87c-d4b0-4191-990a-56a9531c627e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks\" (UID: \"a6e9e87c-d4b0-4191-990a-56a9531c627e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" Nov 02 18:14:17.450635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.450584 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6e9e87c-d4b0-4191-990a-56a9531c627e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks\" (UID: \"a6e9e87c-d4b0-4191-990a-56a9531c627e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" Nov 02 18:14:17.450794 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.450762 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6e9e87c-d4b0-4191-990a-56a9531c627e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks\" (UID: \"a6e9e87c-d4b0-4191-990a-56a9531c627e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" Nov 02 18:14:17.577156 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.554244 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e9e87c-d4b0-4191-990a-56a9531c627e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks\" (UID: \"a6e9e87c-d4b0-4191-990a-56a9531c627e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" Nov 02 18:14:17.577156 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.554327 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6e9e87c-d4b0-4191-990a-56a9531c627e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks\" (UID: \"a6e9e87c-d4b0-4191-990a-56a9531c627e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" Nov 02 18:14:17.577156 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.554427 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6e9e87c-d4b0-4191-990a-56a9531c627e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks\" (UID: \"a6e9e87c-d4b0-4191-990a-56a9531c627e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" Nov 02 18:14:17.577156 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.554551 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6gzf\" (UniqueName: \"kubernetes.io/projected/a6e9e87c-d4b0-4191-990a-56a9531c627e-kube-api-access-d6gzf\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks\" (UID: \"a6e9e87c-d4b0-4191-990a-56a9531c627e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" Nov 02 18:14:17.577156 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.573705 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6e9e87c-d4b0-4191-990a-56a9531c627e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks\" (UID: \"a6e9e87c-d4b0-4191-990a-56a9531c627e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" Nov 02 18:14:17.597354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.597253 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6gzf\" (UniqueName: \"kubernetes.io/projected/a6e9e87c-d4b0-4191-990a-56a9531c627e-kube-api-access-d6gzf\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks\" (UID: \"a6e9e87c-d4b0-4191-990a-56a9531c627e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" Nov 02 18:14:17.600539 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.600482 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6e9e87c-d4b0-4191-990a-56a9531c627e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks\" (UID: \"a6e9e87c-d4b0-4191-990a-56a9531c627e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" Nov 02 18:14:17.609031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.608999 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r"] Nov 02 18:14:17.609555 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.609459 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e9e87c-d4b0-4191-990a-56a9531c627e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks\" (UID: \"a6e9e87c-d4b0-4191-990a-56a9531c627e\") " pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" Nov 02 18:14:17.631224 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:17.630665 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:14:17.696133 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.696105 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" Nov 02 18:14:17.866940 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:17.845718 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:18.007839 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:14:18.007809 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e9e87c_d4b0_4191_990a_56a9531c627e.slice/crio-9e63ed70c3404fcd547efc2921f91f12c37353daabcbe2e3cfb278ac8f4e2047 WatchSource:0}: Error finding container 9e63ed70c3404fcd547efc2921f91f12c37353daabcbe2e3cfb278ac8f4e2047: Status 404 returned error can't find the container with id 9e63ed70c3404fcd547efc2921f91f12c37353daabcbe2e3cfb278ac8f4e2047 Nov 02 18:14:18.010499 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:18.010463 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks"] Nov 02 18:14:18.282157 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:18.282109 5549 patch_prober.go:29] interesting pod/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:14:18.282329 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:18.282191 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:14:18.351254 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:18.351207 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" event={"ID":"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7","Type":"ContainerStarted","Data":"eb6376ea42db355084eeb9f5f1bde51d49ef3a5ad17e06841b0aff76ce244527"} Nov 02 18:14:18.351455 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:18.351282 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" Nov 02 18:14:18.351455 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:18.351307 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" event={"ID":"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7","Type":"ContainerStarted","Data":"035cc30a67ca92460a864ece70b7ab2988c7514523610a31de312efa6f8f53be"} Nov 02 18:14:18.353318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:18.353285 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" event={"ID":"a6e9e87c-d4b0-4191-990a-56a9531c627e","Type":"ContainerStarted","Data":"cfd6e64dc4d4bbcb68887168cc57c336130b428005b58a3643343d88ebcf865e"} Nov 02 18:14:18.353459 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:18.353339 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" event={"ID":"a6e9e87c-d4b0-4191-990a-56a9531c627e","Type":"ContainerStarted","Data":"9e63ed70c3404fcd547efc2921f91f12c37353daabcbe2e3cfb278ac8f4e2047"} Nov 02 18:14:18.353459 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:18.353372 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" Nov 02 18:14:18.372178 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:18.372106 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" podStartSLOduration=2.372088477 podStartE2EDuration="2.372088477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:18.367944527 +0000 UTC m=+2430.156450057" watchObservedRunningTime="2026-11-02 18:14:18.372088477 +0000 UTC m=+2430.160594005" Nov 02 18:14:18.387469 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:18.387409 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" podStartSLOduration=1.387395446 podStartE2EDuration="1.387395446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:18.383941895 +0000 UTC m=+2430.172447424" watchObservedRunningTime="2026-11-02 18:14:18.387395446 +0000 UTC m=+2430.175900992" Nov 02 18:14:18.600034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:18.593172 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:19.395615 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:19.395574 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:20.049517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:20.049479 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:20.721686 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:20.721506 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:21.464251 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:21.464197 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:21.477263 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:21.477226 5549 generic.go:356] "Generic (PLEG): container finished" podID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerID="aac067ee553514a799a2413e2799512f06b8fd446137d39748b46c57248a089b" exitCode=0 Nov 02 18:14:21.477435 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:21.477282 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" event={"ID":"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef","Type":"ContainerDied","Data":"aac067ee553514a799a2413e2799512f06b8fd446137d39748b46c57248a089b"} Nov 02 18:14:21.477997 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:21.477937 5549 scope.go:122] "RemoveContainer" containerID="aac067ee553514a799a2413e2799512f06b8fd446137d39748b46c57248a089b" Nov 02 18:14:21.480239 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:21.480203 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:14:21.480365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:21.480290 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:14:21.480365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:21.480342 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:14:22.141979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:22.141478 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:22.488355 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:22.488320 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" event={"ID":"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef","Type":"ContainerStarted","Data":"a90f923d31d7b3b16c3121acba8214aa159d7185be731660e85b7738ab6e3478"} Nov 02 18:14:22.809221 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:22.809052 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:22.940335 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:22.940301 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:14:22.940542 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:22.940450 5549 patch_prober.go:29] interesting pod/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 18:14:22.940542 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:22.940492 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 18:14:23.565281 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:23.565248 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:23.940178 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:23.940088 5549 patch_prober.go:29] interesting pod/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 18:14:23.940178 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:23.940154 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 18:14:24.238864 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:24.233107 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:24.896531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:24.896490 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv"] Nov 02 18:14:24.912895 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:24.912860 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" Nov 02 18:14:24.918632 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:24.918598 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-658b8c-serving-cert\"" Nov 02 18:14:24.924835 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:24.924801 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv"] Nov 02 18:14:24.940772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:24.940741 5549 patch_prober.go:29] interesting pod/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 18:14:24.941305 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:24.940799 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 18:14:24.960410 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:24.960377 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p98nw\" (UniqueName: \"kubernetes.io/projected/f2e000bf-d401-45c8-8450-a0a6fbf2043b-kube-api-access-p98nw\") pod \"maas-api-e2e-inf-b-658b8c-644849974c-6xnwv\" (UID: \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" Nov 02 18:14:24.960826 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:24.960799 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e000bf-d401-45c8-8450-a0a6fbf2043b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-658b8c-644849974c-6xnwv\" (UID: \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" Nov 02 18:14:24.961015 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:24.960994 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2e000bf-d401-45c8-8450-a0a6fbf2043b-maas-api-tls\") pod \"maas-api-e2e-inf-b-658b8c-644849974c-6xnwv\" (UID: \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" Nov 02 18:14:24.961103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:24.961030 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2e000bf-d401-45c8-8450-a0a6fbf2043b-metrics-certs\") pod \"maas-api-e2e-inf-b-658b8c-644849974c-6xnwv\" (UID: \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" Nov 02 18:14:25.018228 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:25.018177 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:25.089983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:25.088078 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e000bf-d401-45c8-8450-a0a6fbf2043b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-658b8c-644849974c-6xnwv\" (UID: \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" Nov 02 18:14:25.089983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:25.088225 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2e000bf-d401-45c8-8450-a0a6fbf2043b-maas-api-tls\") pod \"maas-api-e2e-inf-b-658b8c-644849974c-6xnwv\" (UID: \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" Nov 02 18:14:25.089983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:25.088256 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2e000bf-d401-45c8-8450-a0a6fbf2043b-metrics-certs\") pod \"maas-api-e2e-inf-b-658b8c-644849974c-6xnwv\" (UID: \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" Nov 02 18:14:25.089983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:25.088322 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p98nw\" (UniqueName: \"kubernetes.io/projected/f2e000bf-d401-45c8-8450-a0a6fbf2043b-kube-api-access-p98nw\") pod \"maas-api-e2e-inf-b-658b8c-644849974c-6xnwv\" (UID: \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" Nov 02 18:14:25.105983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:25.101484 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e000bf-d401-45c8-8450-a0a6fbf2043b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-658b8c-644849974c-6xnwv\" (UID: \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" Nov 02 18:14:25.124102 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:25.122001 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2e000bf-d401-45c8-8450-a0a6fbf2043b-metrics-certs\") pod \"maas-api-e2e-inf-b-658b8c-644849974c-6xnwv\" (UID: \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" Nov 02 18:14:25.134441 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:25.134138 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p98nw\" (UniqueName: \"kubernetes.io/projected/f2e000bf-d401-45c8-8450-a0a6fbf2043b-kube-api-access-p98nw\") pod \"maas-api-e2e-inf-b-658b8c-644849974c-6xnwv\" (UID: \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" Nov 02 18:14:25.152877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:25.143928 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2e000bf-d401-45c8-8450-a0a6fbf2043b-maas-api-tls\") pod \"maas-api-e2e-inf-b-658b8c-644849974c-6xnwv\" (UID: \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" Nov 02 18:14:25.155642 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:25.155606 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:25.247041 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:25.247003 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" Nov 02 18:14:25.639223 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:25.637218 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv"] Nov 02 18:14:25.645527 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:25.645495 5549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:14:25.940220 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:25.940138 5549 patch_prober.go:29] interesting pod/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 18:14:25.940220 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:25.940191 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 18:14:26.029701 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:26.029655 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:26.638162 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:26.638111 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" event={"ID":"f2e000bf-d401-45c8-8450-a0a6fbf2043b","Type":"ContainerStarted","Data":"5c18389b452e090484bc23b8c15c285ccbfcef2837f77b1427e18c1605968d43"} Nov 02 18:14:26.638366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:26.638181 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" event={"ID":"f2e000bf-d401-45c8-8450-a0a6fbf2043b","Type":"ContainerStarted","Data":"bc17a1104f5f9b3272e22d5e0098fb8950f5fcd8b032710bcbb8ae1ed00fc529"} Nov 02 18:14:26.638366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:26.638208 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" Nov 02 18:14:26.664100 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:26.664046 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" podStartSLOduration=2.198351351 podStartE2EDuration="2.664032252s" totalImagesPullingTime="465.680901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:26.659254183 +0000 UTC m=+2438.447759713" watchObservedRunningTime="2026-11-02 18:14:26.664032252 +0000 UTC m=+2438.452537780" Nov 02 18:14:26.742098 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:26.742067 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:26.940288 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:26.940211 5549 patch_prober.go:29] interesting pod/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 18:14:26.940288 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:26.940277 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 18:14:27.520973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:27.515105 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:27.940326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:27.940236 5549 patch_prober.go:29] interesting pod/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 18:14:27.940326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:27.940288 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 18:14:28.168411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:28.149799 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:28.896994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:28.889266 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:28.962978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:28.952301 5549 patch_prober.go:29] interesting pod/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 18:14:28.962978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:28.952360 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 18:14:29.409014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:29.408926 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" Nov 02 18:14:29.409206 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:29.409037 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" Nov 02 18:14:29.516674 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:29.516591 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:29.940211 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:29.940172 5549 patch_prober.go:29] interesting pod/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 18:14:29.940544 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:29.940230 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 18:14:30.269038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:30.268789 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:30.960976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:30.960607 5549 patch_prober.go:29] interesting pod/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 18:14:30.960976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:30.960663 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 18:14:30.972975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:30.963218 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:31.609974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:31.608966 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:31.940800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:31.940710 5549 patch_prober.go:29] interesting pod/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 18:14:31.940800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:31.940762 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 18:14:32.266011 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:32.262908 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:14:32.275295 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:32.272517 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:32.652896 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:32.652819 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" Nov 02 18:14:32.940567 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:32.940492 5549 patch_prober.go:29] interesting pod/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 18:14:32.940786 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:32.940762 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 18:14:33.144992 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:33.143701 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:33.838746 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:33.838672 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:33.941002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:33.940965 5549 patch_prober.go:29] interesting pod/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 18:14:33.941187 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:33.941028 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 18:14:34.530294 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:34.530254 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:34.945553 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:34.945527 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:14:34.946738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:34.946718 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:14:35.311704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:35.308399 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:35.959226 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:35.959183 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:36.821979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:36.821552 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:37.534555 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:37.534502 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:38.239672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:38.239634 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:38.924403 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:38.924369 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:39.654922 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:39.654804 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:40.379566 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:40.379530 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:41.130787 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:41.130750 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:41.740486 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:41.737279 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:42.456977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:42.456928 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:43.219976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:43.218531 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:43.882347 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:43.882298 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:44.566012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:44.565969 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:45.240105 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:45.240065 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:46.274984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:46.270451 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:47.021617 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:47.021509 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:47.591727 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:47.591690 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:14:47.747975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:47.747883 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:48.470086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:48.470000 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:48.704741 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:48.704682 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:49.331082 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.331032 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:49.776073 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.776035 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7"] Nov 02 18:14:49.795474 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.795441 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7"] Nov 02 18:14:49.795825 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.795744 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:49.802463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.802433 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-78383f\"/\"default-dockercfg-fzt7t\"" Nov 02 18:14:49.802654 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.802479 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-78383f\"/\"kube-root-ca.crt\"" Nov 02 18:14:49.802770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.802599 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-78383f\"/\"openshift-service-ca.crt\"" Nov 02 18:14:49.803448 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.803421 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-78383f\"/\"test-model-78383f-kserve-self-signed-certs\"" Nov 02 18:14:49.818850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.818819 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190289c2-9f3a-4201-9cf6-7b38e5edfa86-tls-certs\") pod \"test-model-78383f-kserve-845bf778df-8n6x7\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:49.819084 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.819029 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-tmp-dir\") pod \"test-model-78383f-kserve-845bf778df-8n6x7\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:49.819198 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.819176 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-home\") pod \"test-model-78383f-kserve-845bf778df-8n6x7\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:49.819267 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.819213 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsbsp\" (UniqueName: \"kubernetes.io/projected/190289c2-9f3a-4201-9cf6-7b38e5edfa86-kube-api-access-bsbsp\") pod \"test-model-78383f-kserve-845bf778df-8n6x7\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:49.819388 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.819365 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-model-cache\") pod \"test-model-78383f-kserve-845bf778df-8n6x7\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:49.819479 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.819419 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-dshm\") pod \"test-model-78383f-kserve-845bf778df-8n6x7\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:49.856357 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.856317 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:49.920563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.920523 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-home\") pod \"test-model-78383f-kserve-845bf778df-8n6x7\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:49.920563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.920575 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsbsp\" (UniqueName: \"kubernetes.io/projected/190289c2-9f3a-4201-9cf6-7b38e5edfa86-kube-api-access-bsbsp\") pod \"test-model-78383f-kserve-845bf778df-8n6x7\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:49.920563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.920632 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-model-cache\") pod \"test-model-78383f-kserve-845bf778df-8n6x7\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:49.920563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.920685 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-dshm\") pod \"test-model-78383f-kserve-845bf778df-8n6x7\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:49.920563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.920788 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190289c2-9f3a-4201-9cf6-7b38e5edfa86-tls-certs\") pod \"test-model-78383f-kserve-845bf778df-8n6x7\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:49.920563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.920895 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-tmp-dir\") pod \"test-model-78383f-kserve-845bf778df-8n6x7\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:49.920563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.921331 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-home\") pod \"test-model-78383f-kserve-845bf778df-8n6x7\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:49.920563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.921385 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-tmp-dir\") pod \"test-model-78383f-kserve-845bf778df-8n6x7\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:49.922519 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.922481 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-model-cache\") pod \"test-model-78383f-kserve-845bf778df-8n6x7\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:49.942567 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.942304 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-dshm\") pod \"test-model-78383f-kserve-845bf778df-8n6x7\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:49.983084 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:49.977668 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190289c2-9f3a-4201-9cf6-7b38e5edfa86-tls-certs\") pod \"test-model-78383f-kserve-845bf778df-8n6x7\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:50.043025 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:50.039117 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsbsp\" (UniqueName: \"kubernetes.io/projected/190289c2-9f3a-4201-9cf6-7b38e5edfa86-kube-api-access-bsbsp\") pod \"test-model-78383f-kserve-845bf778df-8n6x7\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:50.120125 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:50.120094 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:50.430473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:50.426727 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:50.462484 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:14:50.462445 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod190289c2_9f3a_4201_9cf6_7b38e5edfa86.slice/crio-7493ee72c8339ef9b8afbb4d6ee8df22312fadfe1ab5bcf3d2e62eed4d173dbd WatchSource:0}: Error finding container 7493ee72c8339ef9b8afbb4d6ee8df22312fadfe1ab5bcf3d2e62eed4d173dbd: Status 404 returned error can't find the container with id 7493ee72c8339ef9b8afbb4d6ee8df22312fadfe1ab5bcf3d2e62eed4d173dbd Nov 02 18:14:50.462484 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:50.462463 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7"] Nov 02 18:14:51.050471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:51.050432 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" event={"ID":"190289c2-9f3a-4201-9cf6-7b38e5edfa86","Type":"ContainerStarted","Data":"ddb5028765cde3a42e1aa34d6e5ed19d961f19572550fffdb588febff10b73af"} Nov 02 18:14:51.050897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:51.050481 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" event={"ID":"190289c2-9f3a-4201-9cf6-7b38e5edfa86","Type":"ContainerStarted","Data":"7493ee72c8339ef9b8afbb4d6ee8df22312fadfe1ab5bcf3d2e62eed4d173dbd"} Nov 02 18:14:51.069862 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:51.069807 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" podStartSLOduration=2.069792768 podStartE2EDuration="2.069792768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:51.067351919 +0000 UTC m=+2462.855857448" watchObservedRunningTime="2026-11-02 18:14:51.069792768 +0000 UTC m=+2462.858298296" Nov 02 18:14:51.120687 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:51.120647 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:14:51.392026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:51.386780 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:52.120166 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:52.117611 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:52.835976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:52.835685 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:53.525450 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:53.519834 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:54.238995 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:54.237726 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:55.120979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:55.119850 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:56.040983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:56.038878 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:56.835483 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:56.832315 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:57.187328 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.187247 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz"] Nov 02 18:14:57.194667 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.194640 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" Nov 02 18:14:57.200205 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.200176 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"agc-cred-41558d72238a\"" Nov 02 18:14:57.201177 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.201147 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"openshift-service-ca.crt\"" Nov 02 18:14:57.201177 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.201155 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"routing-overlay\"" Nov 02 18:14:57.201360 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.201155 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"kube-root-ca.crt\"" Nov 02 18:14:57.201360 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.201159 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"payload-processing-external-model-plugins-e2e-extmod-a-807c08\"" Nov 02 18:14:57.201360 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.201278 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"payload-processing-external-model-e2e--d2178d1f-dockercfg-8gtsz\"" Nov 02 18:14:57.201774 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.201756 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz"] Nov 02 18:14:57.229627 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.229593 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:57.335979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.320304 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz\" (UID: \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" Nov 02 18:14:57.335979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.320404 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz\" (UID: \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" Nov 02 18:14:57.335979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.320439 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn4g7\" (UniqueName: \"kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-kube-api-access-bn4g7\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz\" (UID: \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" Nov 02 18:14:57.335979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.320707 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz\" (UID: \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" Nov 02 18:14:57.381967 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.381904 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:57.424297 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.424251 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz\" (UID: \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" Nov 02 18:14:57.424558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.424534 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz\" (UID: \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" Nov 02 18:14:57.424641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.424625 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz\" (UID: \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" Nov 02 18:14:57.424681 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.424667 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn4g7\" (UniqueName: \"kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-kube-api-access-bn4g7\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz\" (UID: \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" Nov 02 18:14:57.426361 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.426334 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz\" (UID: \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" Nov 02 18:14:57.426996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.426974 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz\" (UID: \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" Nov 02 18:14:57.430414 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.430388 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz\" (UID: \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" Nov 02 18:14:57.434001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.433974 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn4g7\" (UniqueName: \"kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-kube-api-access-bn4g7\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz\" (UID: \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" Nov 02 18:14:57.515373 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.515337 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" Nov 02 18:14:57.654613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.654574 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r"] Nov 02 18:14:57.673694 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.668937 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" Nov 02 18:14:57.673694 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.668832 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r"] Nov 02 18:14:57.673945 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.673854 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"routing-overlay\"" Nov 02 18:14:57.691982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.674200 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"kube-root-ca.crt\"" Nov 02 18:14:57.691982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.674332 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"openshift-service-ca.crt\"" Nov 02 18:14:57.691982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.689613 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"agc-cred-5df426b52b49\"" Nov 02 18:14:57.691982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.689726 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"payload-processing-external-model-e2e--24159a0a-dockercfg-lk82g\"" Nov 02 18:14:57.691982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.689854 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"payload-processing-external-model-plugins-e2e-extmod-b-bee8f6\"" Nov 02 18:14:57.774087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.768313 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r\" (UID: \"76dbe462-e831-40a3-a735-79ce6c63ed41\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" Nov 02 18:14:57.774087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.768409 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bn29\" (UniqueName: \"kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-kube-api-access-2bn29\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r\" (UID: \"76dbe462-e831-40a3-a735-79ce6c63ed41\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" Nov 02 18:14:57.774087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.768531 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r\" (UID: \"76dbe462-e831-40a3-a735-79ce6c63ed41\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" Nov 02 18:14:57.774087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.768623 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r\" (UID: \"76dbe462-e831-40a3-a735-79ce6c63ed41\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" Nov 02 18:14:57.926552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.914717 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r\" (UID: \"76dbe462-e831-40a3-a735-79ce6c63ed41\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" Nov 02 18:14:57.926552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.914836 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r\" (UID: \"76dbe462-e831-40a3-a735-79ce6c63ed41\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" Nov 02 18:14:57.926552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.915224 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r\" (UID: \"76dbe462-e831-40a3-a735-79ce6c63ed41\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" Nov 02 18:14:57.926552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.915314 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bn29\" (UniqueName: \"kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-kube-api-access-2bn29\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r\" (UID: \"76dbe462-e831-40a3-a735-79ce6c63ed41\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" Nov 02 18:14:57.926552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.916590 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r\" (UID: \"76dbe462-e831-40a3-a735-79ce6c63ed41\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" Nov 02 18:14:57.951975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:57.935141 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r\" (UID: \"76dbe462-e831-40a3-a735-79ce6c63ed41\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" Nov 02 18:14:58.262026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:58.258496 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz"] Nov 02 18:14:58.262026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:58.259366 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bn29\" (UniqueName: \"kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-kube-api-access-2bn29\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r\" (UID: \"76dbe462-e831-40a3-a735-79ce6c63ed41\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" Nov 02 18:14:58.271173 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:58.271139 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r\" (UID: \"76dbe462-e831-40a3-a735-79ce6c63ed41\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" Nov 02 18:14:58.273143 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:58.273047 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" event={"ID":"22fe51da-2ea2-4ab3-8b74-aa303cab4be2","Type":"ContainerStarted","Data":"0d7bbffd70090c08e6837c975e8596718541af7326bf8fcf9bd08a2a1ccc08c7"} Nov 02 18:14:58.525981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:58.513826 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:58.525981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:58.514486 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" Nov 02 18:14:58.833695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:58.833655 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz"] Nov 02 18:14:58.858798 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:14:58.858765 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76dbe462_e831_40a3_a735_79ce6c63ed41.slice/crio-e473152d3cf4791ef819d75d8912783a812aeedf503a6d1f88410c6b7ca5bdd8 WatchSource:0}: Error finding container e473152d3cf4791ef819d75d8912783a812aeedf503a6d1f88410c6b7ca5bdd8: Status 404 returned error can't find the container with id e473152d3cf4791ef819d75d8912783a812aeedf503a6d1f88410c6b7ca5bdd8 Nov 02 18:14:58.859220 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:58.859195 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r"] Nov 02 18:14:58.997857 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:58.997825 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:59.165137 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:59.165028 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:59.319019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:59.316929 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" event={"ID":"76dbe462-e831-40a3-a735-79ce6c63ed41","Type":"ContainerStarted","Data":"b59975749f6f7732734f4a0f469303d3e5226189079632cce2cf17fcd7f45624"} Nov 02 18:14:59.319019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:59.317010 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" event={"ID":"76dbe462-e831-40a3-a735-79ce6c63ed41","Type":"ContainerStarted","Data":"e473152d3cf4791ef819d75d8912783a812aeedf503a6d1f88410c6b7ca5bdd8"} Nov 02 18:14:59.319019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:59.318030 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" Nov 02 18:14:59.326384 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:59.326339 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" event={"ID":"22fe51da-2ea2-4ab3-8b74-aa303cab4be2","Type":"ContainerStarted","Data":"48e33a32d4bba7172bf4a8fbd5bb817b2961412f061419f79a383fc9d8832f6f"} Nov 02 18:14:59.326769 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:59.326740 5549 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" secret="" err="secret \"payload-processing-external-model-e2e--d2178d1f-dockercfg-8gtsz\" not found" Nov 02 18:14:59.327297 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:59.327270 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" Nov 02 18:14:59.346826 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:59.346739 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" podStartSLOduration=2.346718017 podStartE2EDuration="2.346718017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:59.342477377 +0000 UTC m=+2471.130982905" watchObservedRunningTime="2026-11-02 18:14:59.346718017 +0000 UTC m=+2471.135223546" Nov 02 18:14:59.368275 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:59.368217 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" podStartSLOduration=2.368200403 podStartE2EDuration="2.368200403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:59.364842885 +0000 UTC m=+2471.153348412" watchObservedRunningTime="2026-11-02 18:14:59.368200403 +0000 UTC m=+2471.156705930" Nov 02 18:14:59.385600 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:59.385565 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r"] Nov 02 18:14:59.423030 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.422874 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-bee8f6/agc-cred-5df426b52b49: secret "agc-cred-5df426b52b49" not found Nov 02 18:14:59.423030 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.422924 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r: secret "agc-cred-5df426b52b49" not found Nov 02 18:14:59.423030 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.422906 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-plugins-e2e-extmod-b-bee8f6: configmap "payload-processing-external-model-plugins-e2e-extmod-b-bee8f6" not found Nov 02 18:14:59.423311 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.423073 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-plugins-config-volume podName:76dbe462-e831-40a3-a735-79ce6c63ed41 nodeName:}" failed. No retries permitted until 2026-11-02 18:14:59.923055397 +0000 UTC m=+2471.711560922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" (UID: "76dbe462-e831-40a3-a735-79ce6c63ed41") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-bee8f6" not found Nov 02 18:14:59.423311 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.423100 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-provider-credentials podName:76dbe462-e831-40a3-a735-79ce6c63ed41 nodeName:}" failed. No retries permitted until 2026-11-02 18:14:59.923091218 +0000 UTC m=+2471.711596724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" (UID: "76dbe462-e831-40a3-a735-79ce6c63ed41") : secret "agc-cred-5df426b52b49" not found Nov 02 18:14:59.423311 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.423143 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-plugins-e2e-extmod-a-807c08: configmap "payload-processing-external-model-plugins-e2e-extmod-a-807c08" not found Nov 02 18:14:59.423311 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.423282 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-plugins-config-volume podName:22fe51da-2ea2-4ab3-8b74-aa303cab4be2 nodeName:}" failed. No retries permitted until 2026-11-02 18:14:59.923252494 +0000 UTC m=+2471.711758017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" (UID: "22fe51da-2ea2-4ab3-8b74-aa303cab4be2") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-807c08" not found Nov 02 18:14:59.423929 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.423843 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-807c08/agc-cred-41558d72238a: secret "agc-cred-41558d72238a" not found Nov 02 18:14:59.423929 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.423863 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz: secret "agc-cred-41558d72238a" not found Nov 02 18:14:59.423929 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.423905 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-provider-credentials podName:22fe51da-2ea2-4ab3-8b74-aa303cab4be2 nodeName:}" failed. No retries permitted until 2026-11-02 18:14:59.923891244 +0000 UTC m=+2471.712396763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" (UID: "22fe51da-2ea2-4ab3-8b74-aa303cab4be2") : secret "agc-cred-41558d72238a" not found Nov 02 18:14:59.481842 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:59.481811 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:59.665239 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:14:59.654822 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:59.959431 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.959398 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-807c08/agc-cred-41558d72238a: secret "agc-cred-41558d72238a" not found Nov 02 18:14:59.959431 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.959429 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz: secret "agc-cred-41558d72238a" not found Nov 02 18:14:59.959431 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.959431 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-plugins-e2e-extmod-b-bee8f6: configmap "payload-processing-external-model-plugins-e2e-extmod-b-bee8f6" not found Nov 02 18:14:59.959720 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.959487 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-provider-credentials podName:22fe51da-2ea2-4ab3-8b74-aa303cab4be2 nodeName:}" failed. No retries permitted until 2026-11-02 18:15:00.959468985 +0000 UTC m=+2472.747974511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" (UID: "22fe51da-2ea2-4ab3-8b74-aa303cab4be2") : secret "agc-cred-41558d72238a" not found Nov 02 18:14:59.959720 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.959488 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-bee8f6/agc-cred-5df426b52b49: secret "agc-cred-5df426b52b49" not found Nov 02 18:14:59.959720 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.959504 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-plugins-e2e-extmod-a-807c08: configmap "payload-processing-external-model-plugins-e2e-extmod-a-807c08" not found Nov 02 18:14:59.959720 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.959511 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r: secret "agc-cred-5df426b52b49" not found Nov 02 18:14:59.959720 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.959540 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-plugins-config-volume podName:76dbe462-e831-40a3-a735-79ce6c63ed41 nodeName:}" failed. No retries permitted until 2026-11-02 18:15:00.959521148 +0000 UTC m=+2472.748026654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" (UID: "76dbe462-e831-40a3-a735-79ce6c63ed41") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-bee8f6" not found Nov 02 18:14:59.959720 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.959565 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-plugins-config-volume podName:22fe51da-2ea2-4ab3-8b74-aa303cab4be2 nodeName:}" failed. No retries permitted until 2026-11-02 18:15:00.95955852 +0000 UTC m=+2472.748064027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" (UID: "22fe51da-2ea2-4ab3-8b74-aa303cab4be2") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-807c08" not found Nov 02 18:14:59.959720 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:14:59.959578 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-provider-credentials podName:76dbe462-e831-40a3-a735-79ce6c63ed41 nodeName:}" failed. No retries permitted until 2026-11-02 18:15:00.959572759 +0000 UTC m=+2472.748078264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" (UID: "76dbe462-e831-40a3-a735-79ce6c63ed41") : secret "agc-cred-5df426b52b49" not found Nov 02 18:15:00.121482 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.121438 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:15:00.135008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.134978 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:15:00.191017 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.190986 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749"] Nov 02 18:15:00.201517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.201483 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" Nov 02 18:15:00.204248 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.204214 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749"] Nov 02 18:15:00.219022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.215778 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht"] Nov 02 18:15:00.223684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.223659 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" Nov 02 18:15:00.224702 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.224671 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht"] Nov 02 18:15:00.259903 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.259867 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\"" Nov 02 18:15:00.336382 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.336346 5549 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" secret="" err="secret \"payload-processing-external-model-e2e--24159a0a-dockercfg-lk82g\" not found" Nov 02 18:15:00.348144 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.348111 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:15:00.373014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.369140 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2a76db-edd0-46a7-b117-2d06e99f3840-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749\" (UID: \"ca2a76db-edd0-46a7-b117-2d06e99f3840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" Nov 02 18:15:00.373014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.369205 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea777210-3c0f-4e8b-be3b-4b07895990b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht\" (UID: \"ea777210-3c0f-4e8b-be3b-4b07895990b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" Nov 02 18:15:00.373014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.369314 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwx52\" (UniqueName: \"kubernetes.io/projected/ea777210-3c0f-4e8b-be3b-4b07895990b7-kube-api-access-gwx52\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht\" (UID: \"ea777210-3c0f-4e8b-be3b-4b07895990b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" Nov 02 18:15:00.373014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.369370 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wq6q\" (UniqueName: \"kubernetes.io/projected/ca2a76db-edd0-46a7-b117-2d06e99f3840-kube-api-access-8wq6q\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749\" (UID: \"ca2a76db-edd0-46a7-b117-2d06e99f3840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" Nov 02 18:15:00.434975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.432024 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:00.497589 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.497499 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2a76db-edd0-46a7-b117-2d06e99f3840-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749\" (UID: \"ca2a76db-edd0-46a7-b117-2d06e99f3840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" Nov 02 18:15:00.497589 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.497561 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea777210-3c0f-4e8b-be3b-4b07895990b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht\" (UID: \"ea777210-3c0f-4e8b-be3b-4b07895990b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" Nov 02 18:15:00.497825 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.497679 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwx52\" (UniqueName: \"kubernetes.io/projected/ea777210-3c0f-4e8b-be3b-4b07895990b7-kube-api-access-gwx52\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht\" (UID: \"ea777210-3c0f-4e8b-be3b-4b07895990b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" Nov 02 18:15:00.497825 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.497793 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wq6q\" (UniqueName: \"kubernetes.io/projected/ca2a76db-edd0-46a7-b117-2d06e99f3840-kube-api-access-8wq6q\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749\" (UID: \"ca2a76db-edd0-46a7-b117-2d06e99f3840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" Nov 02 18:15:00.499027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.498436 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2a76db-edd0-46a7-b117-2d06e99f3840-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749\" (UID: \"ca2a76db-edd0-46a7-b117-2d06e99f3840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" Nov 02 18:15:00.499631 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.499602 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea777210-3c0f-4e8b-be3b-4b07895990b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht\" (UID: \"ea777210-3c0f-4e8b-be3b-4b07895990b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" Nov 02 18:15:00.509613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.509586 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wq6q\" (UniqueName: \"kubernetes.io/projected/ca2a76db-edd0-46a7-b117-2d06e99f3840-kube-api-access-8wq6q\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749\" (UID: \"ca2a76db-edd0-46a7-b117-2d06e99f3840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" Nov 02 18:15:00.510614 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.510591 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwx52\" (UniqueName: \"kubernetes.io/projected/ea777210-3c0f-4e8b-be3b-4b07895990b7-kube-api-access-gwx52\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht\" (UID: \"ea777210-3c0f-4e8b-be3b-4b07895990b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" Nov 02 18:15:00.538111 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.538080 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" Nov 02 18:15:00.551735 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.551693 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" Nov 02 18:15:00.848704 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:15:00.848678 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea777210_3c0f_4e8b_be3b_4b07895990b7.slice/crio-691bd1b3232e507a261888efe6c0636fefe7e63b92c025ded80d471952419351 WatchSource:0}: Error finding container 691bd1b3232e507a261888efe6c0636fefe7e63b92c025ded80d471952419351: Status 404 returned error can't find the container with id 691bd1b3232e507a261888efe6c0636fefe7e63b92c025ded80d471952419351 Nov 02 18:15:00.848838 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.848820 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht"] Nov 02 18:15:00.851419 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:15:00.851392 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2a76db_edd0_46a7_b117_2d06e99f3840.slice/crio-8e808a9192dddf8ecf05d384600a6fa26e5d26d1d91145ac040f8df389f42bf3 WatchSource:0}: Error finding container 8e808a9192dddf8ecf05d384600a6fa26e5d26d1d91145ac040f8df389f42bf3: Status 404 returned error can't find the container with id 8e808a9192dddf8ecf05d384600a6fa26e5d26d1d91145ac040f8df389f42bf3 Nov 02 18:15:00.854850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.854822 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749"] Nov 02 18:15:00.920873 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.920837 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:15:00.928219 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.928187 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm"] Nov 02 18:15:00.936524 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.936494 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:00.946298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.946262 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-658b8c\"/\"openshift-service-ca.crt\"" Nov 02 18:15:00.946298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.946290 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-658b8c\"/\"kube-root-ca.crt\"" Nov 02 18:15:00.947101 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.947074 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-658b8c\"/\"test-model-658b8c-kserve-self-signed-certs\"" Nov 02 18:15:00.947254 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.947175 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-658b8c\"/\"default-dockercfg-nlr29\"" Nov 02 18:15:00.948507 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:00.948478 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm"] Nov 02 18:15:01.007915 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.007885 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-home\") pod \"test-model-658b8c-kserve-5d9f6d89cf-df8fm\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.008126 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.008040 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-dshm\") pod \"test-model-658b8c-kserve-5d9f6d89cf-df8fm\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.008126 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.008100 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-tls-certs\") pod \"test-model-658b8c-kserve-5d9f6d89cf-df8fm\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.008269 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:01.008251 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-807c08/agc-cred-41558d72238a: secret "agc-cred-41558d72238a" not found Nov 02 18:15:01.008314 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.008264 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szz9n\" (UniqueName: \"kubernetes.io/projected/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-kube-api-access-szz9n\") pod \"test-model-658b8c-kserve-5d9f6d89cf-df8fm\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.008358 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:01.008271 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz: secret "agc-cred-41558d72238a" not found Nov 02 18:15:01.008358 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.008350 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-tmp-dir\") pod \"test-model-658b8c-kserve-5d9f6d89cf-df8fm\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.008432 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:01.008367 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-provider-credentials podName:22fe51da-2ea2-4ab3-8b74-aa303cab4be2 nodeName:}" failed. No retries permitted until 2026-11-02 18:15:03.008353604 +0000 UTC m=+2474.796859110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" (UID: "22fe51da-2ea2-4ab3-8b74-aa303cab4be2") : secret "agc-cred-41558d72238a" not found Nov 02 18:15:01.008432 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:01.008314 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-plugins-e2e-extmod-b-bee8f6: configmap "payload-processing-external-model-plugins-e2e-extmod-b-bee8f6" not found Nov 02 18:15:01.008432 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:01.008408 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-plugins-config-volume podName:76dbe462-e831-40a3-a735-79ce6c63ed41 nodeName:}" failed. No retries permitted until 2026-11-02 18:15:03.008397263 +0000 UTC m=+2474.796902768 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" (UID: "76dbe462-e831-40a3-a735-79ce6c63ed41") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-bee8f6" not found Nov 02 18:15:01.008568 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:01.008452 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-bee8f6/agc-cred-5df426b52b49: secret "agc-cred-5df426b52b49" not found Nov 02 18:15:01.008568 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:01.008460 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r: secret "agc-cred-5df426b52b49" not found Nov 02 18:15:01.008568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.008469 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-model-cache\") pod \"test-model-658b8c-kserve-5d9f6d89cf-df8fm\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.008568 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:01.008480 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-provider-credentials podName:76dbe462-e831-40a3-a735-79ce6c63ed41 nodeName:}" failed. No retries permitted until 2026-11-02 18:15:03.008473621 +0000 UTC m=+2474.796979155 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" (UID: "76dbe462-e831-40a3-a735-79ce6c63ed41") : secret "agc-cred-5df426b52b49" not found Nov 02 18:15:01.008783 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:01.008771 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-plugins-e2e-extmod-a-807c08: configmap "payload-processing-external-model-plugins-e2e-extmod-a-807c08" not found Nov 02 18:15:01.008839 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:01.008820 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-plugins-config-volume podName:22fe51da-2ea2-4ab3-8b74-aa303cab4be2 nodeName:}" failed. No retries permitted until 2026-11-02 18:15:03.008808859 +0000 UTC m=+2474.797314382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" (UID: "22fe51da-2ea2-4ab3-8b74-aa303cab4be2") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-807c08" not found Nov 02 18:15:01.046760 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.046729 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:01.110261 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.110217 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-home\") pod \"test-model-658b8c-kserve-5d9f6d89cf-df8fm\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.110399 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.110362 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-dshm\") pod \"test-model-658b8c-kserve-5d9f6d89cf-df8fm\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.110469 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.110444 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-tls-certs\") pod \"test-model-658b8c-kserve-5d9f6d89cf-df8fm\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.110574 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.110553 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szz9n\" (UniqueName: \"kubernetes.io/projected/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-kube-api-access-szz9n\") pod \"test-model-658b8c-kserve-5d9f6d89cf-df8fm\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.110729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.110618 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-tmp-dir\") pod \"test-model-658b8c-kserve-5d9f6d89cf-df8fm\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.110729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.110560 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-home\") pod \"test-model-658b8c-kserve-5d9f6d89cf-df8fm\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.110729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.110704 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-model-cache\") pod \"test-model-658b8c-kserve-5d9f6d89cf-df8fm\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.110922 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.110893 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-tmp-dir\") pod \"test-model-658b8c-kserve-5d9f6d89cf-df8fm\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.111008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.110996 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-model-cache\") pod \"test-model-658b8c-kserve-5d9f6d89cf-df8fm\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.112533 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.112516 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-dshm\") pod \"test-model-658b8c-kserve-5d9f6d89cf-df8fm\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.112878 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.112858 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-tls-certs\") pod \"test-model-658b8c-kserve-5d9f6d89cf-df8fm\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.119789 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.119767 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szz9n\" (UniqueName: \"kubernetes.io/projected/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-kube-api-access-szz9n\") pod \"test-model-658b8c-kserve-5d9f6d89cf-df8fm\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.281975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.281504 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:01.376027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.373634 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" event={"ID":"ca2a76db-edd0-46a7-b117-2d06e99f3840","Type":"ContainerStarted","Data":"8e808a9192dddf8ecf05d384600a6fa26e5d26d1d91145ac040f8df389f42bf3"} Nov 02 18:15:01.391423 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.377880 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" event={"ID":"ea777210-3c0f-4e8b-be3b-4b07895990b7","Type":"ContainerStarted","Data":"691bd1b3232e507a261888efe6c0636fefe7e63b92c025ded80d471952419351"} Nov 02 18:15:01.696605 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:15:01.696569 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c2b7cf4_52fc_4044_ad57_34fb04d241c3.slice/crio-034d1ecf7590c3e77611a0a84ae3545cea4b5c6f5c011ac4042828417e62d692 WatchSource:0}: Error finding container 034d1ecf7590c3e77611a0a84ae3545cea4b5c6f5c011ac4042828417e62d692: Status 404 returned error can't find the container with id 034d1ecf7590c3e77611a0a84ae3545cea4b5c6f5c011ac4042828417e62d692 Nov 02 18:15:01.697564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.697534 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm"] Nov 02 18:15:01.983300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:01.983245 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:02.135322 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:02.135289 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:02.412461 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:02.412365 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" event={"ID":"3c2b7cf4-52fc-4044-ad57-34fb04d241c3","Type":"ContainerStarted","Data":"322ff85d759e9dce77e6afa6a005485a7f82b301897387940c3a1e0a7f3f8a2a"} Nov 02 18:15:02.412461 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:02.412415 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" event={"ID":"3c2b7cf4-52fc-4044-ad57-34fb04d241c3","Type":"ContainerStarted","Data":"034d1ecf7590c3e77611a0a84ae3545cea4b5c6f5c011ac4042828417e62d692"} Nov 02 18:15:02.414832 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:02.414805 5549 generic.go:356] "Generic (PLEG): container finished" podID="ca2a76db-edd0-46a7-b117-2d06e99f3840" containerID="f77c67c7c3b7cf6c2b380b068901e5bd326393782dc2e203ceb0ce901d044dab" exitCode=60 Nov 02 18:15:02.415020 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:02.415003 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" event={"ID":"ca2a76db-edd0-46a7-b117-2d06e99f3840","Type":"ContainerDied","Data":"f77c67c7c3b7cf6c2b380b068901e5bd326393782dc2e203ceb0ce901d044dab"} Nov 02 18:15:02.416803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:02.416783 5549 scope.go:122] "RemoveContainer" containerID="f77c67c7c3b7cf6c2b380b068901e5bd326393782dc2e203ceb0ce901d044dab" Nov 02 18:15:02.421409 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:02.421381 5549 generic.go:356] "Generic (PLEG): container finished" podID="ea777210-3c0f-4e8b-be3b-4b07895990b7" containerID="284f2931c31189428fb2d3f0a6fdb8ae5b86014684686147950e8d80276af614" exitCode=60 Nov 02 18:15:02.421534 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:02.421449 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" event={"ID":"ea777210-3c0f-4e8b-be3b-4b07895990b7","Type":"ContainerDied","Data":"284f2931c31189428fb2d3f0a6fdb8ae5b86014684686147950e8d80276af614"} Nov 02 18:15:02.421771 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:02.421752 5549 scope.go:122] "RemoveContainer" containerID="284f2931c31189428fb2d3f0a6fdb8ae5b86014684686147950e8d80276af614" Nov 02 18:15:02.437400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:02.437327 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" podStartSLOduration=2.437310793 podStartE2EDuration="2.437310793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:02.435311751 +0000 UTC m=+2474.223817280" watchObservedRunningTime="2026-11-02 18:15:02.437310793 +0000 UTC m=+2474.225816325" Nov 02 18:15:03.101058 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:03.101020 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-plugins-e2e-extmod-b-bee8f6: configmap "payload-processing-external-model-plugins-e2e-extmod-b-bee8f6" not found Nov 02 18:15:03.101200 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:03.101070 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-bee8f6/agc-cred-5df426b52b49: secret "agc-cred-5df426b52b49" not found Nov 02 18:15:03.101200 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:03.101091 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r: secret "agc-cred-5df426b52b49" not found Nov 02 18:15:03.101200 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:03.101108 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-plugins-e2e-extmod-a-807c08: configmap "payload-processing-external-model-plugins-e2e-extmod-a-807c08" not found Nov 02 18:15:03.101200 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:03.101093 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-plugins-config-volume podName:76dbe462-e831-40a3-a735-79ce6c63ed41 nodeName:}" failed. No retries permitted until 2026-11-02 18:15:07.101073965 +0000 UTC m=+2478.889579502 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" (UID: "76dbe462-e831-40a3-a735-79ce6c63ed41") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-bee8f6" not found Nov 02 18:15:03.101200 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:03.101164 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-provider-credentials podName:76dbe462-e831-40a3-a735-79ce6c63ed41 nodeName:}" failed. No retries permitted until 2026-11-02 18:15:07.101144271 +0000 UTC m=+2478.889649794 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" (UID: "76dbe462-e831-40a3-a735-79ce6c63ed41") : secret "agc-cred-5df426b52b49" not found Nov 02 18:15:03.101200 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:03.101192 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-plugins-config-volume podName:22fe51da-2ea2-4ab3-8b74-aa303cab4be2 nodeName:}" failed. No retries permitted until 2026-11-02 18:15:07.101182911 +0000 UTC m=+2478.889688417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" (UID: "22fe51da-2ea2-4ab3-8b74-aa303cab4be2") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-807c08" not found Nov 02 18:15:03.101547 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:03.101466 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-807c08/agc-cred-41558d72238a: secret "agc-cred-41558d72238a" not found Nov 02 18:15:03.101547 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:03.101483 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz: secret "agc-cred-41558d72238a" not found Nov 02 18:15:03.101547 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:03.101524 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-provider-credentials podName:22fe51da-2ea2-4ab3-8b74-aa303cab4be2 nodeName:}" failed. No retries permitted until 2026-11-02 18:15:07.101513437 +0000 UTC m=+2478.890018959 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" (UID: "22fe51da-2ea2-4ab3-8b74-aa303cab4be2") : secret "agc-cred-41558d72238a" not found Nov 02 18:15:03.288980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:03.285103 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:03.401029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:03.400605 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:03.515983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:03.509630 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" event={"ID":"ca2a76db-edd0-46a7-b117-2d06e99f3840","Type":"ContainerStarted","Data":"99c7b43b0dbfd6eab1988f27cb3ff6a2563ef696196df9a89d296b6d1099a3e1"} Nov 02 18:15:03.530341 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:03.520074 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" event={"ID":"ea777210-3c0f-4e8b-be3b-4b07895990b7","Type":"ContainerStarted","Data":"d2d114fc414dbb75c9822ed2657d8bbbc2a1d156a24d900672e7c55f44eaa281"} Nov 02 18:15:03.557968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:03.546005 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" podStartSLOduration=3.545984884 podStartE2EDuration="3.545984884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.543841194 +0000 UTC m=+2475.332346740" watchObservedRunningTime="2026-11-02 18:15:03.545984884 +0000 UTC m=+2475.334490413" Nov 02 18:15:03.611378 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:03.603648 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" podStartSLOduration=3.603627902 podStartE2EDuration="3.603627902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.572570582 +0000 UTC m=+2475.361076111" watchObservedRunningTime="2026-11-02 18:15:03.603627902 +0000 UTC m=+2475.392133430" Nov 02 18:15:04.536185 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:04.536126 5549 generic.go:356] "Generic (PLEG): container finished" podID="ca2a76db-edd0-46a7-b117-2d06e99f3840" containerID="99c7b43b0dbfd6eab1988f27cb3ff6a2563ef696196df9a89d296b6d1099a3e1" exitCode=60 Nov 02 18:15:04.536999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:04.536341 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" event={"ID":"ca2a76db-edd0-46a7-b117-2d06e99f3840","Type":"ContainerDied","Data":"99c7b43b0dbfd6eab1988f27cb3ff6a2563ef696196df9a89d296b6d1099a3e1"} Nov 02 18:15:04.536999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:04.536403 5549 scope.go:122] "RemoveContainer" containerID="f77c67c7c3b7cf6c2b380b068901e5bd326393782dc2e203ceb0ce901d044dab" Nov 02 18:15:04.537257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:04.537009 5549 scope.go:122] "RemoveContainer" containerID="99c7b43b0dbfd6eab1988f27cb3ff6a2563ef696196df9a89d296b6d1099a3e1" Nov 02 18:15:04.537596 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:04.537566 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749_odh-ai-gateway-infra(ca2a76db-edd0-46a7-b117-2d06e99f3840)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" podUID="ca2a76db-edd0-46a7-b117-2d06e99f3840" Nov 02 18:15:04.544313 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:04.542629 5549 generic.go:356] "Generic (PLEG): container finished" podID="ea777210-3c0f-4e8b-be3b-4b07895990b7" containerID="d2d114fc414dbb75c9822ed2657d8bbbc2a1d156a24d900672e7c55f44eaa281" exitCode=60 Nov 02 18:15:04.544313 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:04.543860 5549 scope.go:122] "RemoveContainer" containerID="d2d114fc414dbb75c9822ed2657d8bbbc2a1d156a24d900672e7c55f44eaa281" Nov 02 18:15:04.544313 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:04.544203 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht_odh-ai-gateway-infra(ea777210-3c0f-4e8b-be3b-4b07895990b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" podUID="ea777210-3c0f-4e8b-be3b-4b07895990b7" Nov 02 18:15:04.544313 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:04.544252 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" event={"ID":"ea777210-3c0f-4e8b-be3b-4b07895990b7","Type":"ContainerDied","Data":"d2d114fc414dbb75c9822ed2657d8bbbc2a1d156a24d900672e7c55f44eaa281"} Nov 02 18:15:04.562451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:04.562423 5549 scope.go:122] "RemoveContainer" containerID="284f2931c31189428fb2d3f0a6fdb8ae5b86014684686147950e8d80276af614" Nov 02 18:15:04.982704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:04.982669 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:05.420031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:05.399349 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" podUID="22fe51da-2ea2-4ab3-8b74-aa303cab4be2" containerName="payload-processing" containerID="cri-o://48e33a32d4bba7172bf4a8fbd5bb817b2961412f061419f79a383fc9d8832f6f" gracePeriod=25 Nov 02 18:15:05.766977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:05.766847 5549 scope.go:122] "RemoveContainer" containerID="d2d114fc414dbb75c9822ed2657d8bbbc2a1d156a24d900672e7c55f44eaa281" Nov 02 18:15:05.767508 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:05.767368 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht_odh-ai-gateway-infra(ea777210-3c0f-4e8b-be3b-4b07895990b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" podUID="ea777210-3c0f-4e8b-be3b-4b07895990b7" Nov 02 18:15:05.778716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:05.776539 5549 scope.go:122] "RemoveContainer" containerID="99c7b43b0dbfd6eab1988f27cb3ff6a2563ef696196df9a89d296b6d1099a3e1" Nov 02 18:15:05.778716 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:05.777048 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749_odh-ai-gateway-infra(ca2a76db-edd0-46a7-b117-2d06e99f3840)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" podUID="ca2a76db-edd0-46a7-b117-2d06e99f3840" Nov 02 18:15:06.277985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:06.277705 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:06.497978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:06.480224 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" podUID="76dbe462-e831-40a3-a735-79ce6c63ed41" containerName="payload-processing" containerID="cri-o://b59975749f6f7732734f4a0f469303d3e5226189079632cce2cf17fcd7f45624" gracePeriod=25 Nov 02 18:15:06.674428 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:06.674384 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 18:15:07.119639 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:07.119283 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-bee8f6/agc-cred-5df426b52b49: secret "agc-cred-5df426b52b49" not found Nov 02 18:15:07.119639 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:07.119311 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r: secret "agc-cred-5df426b52b49" not found Nov 02 18:15:07.119639 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:07.119367 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-provider-credentials podName:76dbe462-e831-40a3-a735-79ce6c63ed41 nodeName:}" failed. No retries permitted until 2026-11-02 18:15:15.119348401 +0000 UTC m=+2486.907853921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" (UID: "76dbe462-e831-40a3-a735-79ce6c63ed41") : secret "agc-cred-5df426b52b49" not found Nov 02 18:15:07.120232 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:07.119727 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-plugins-e2e-extmod-a-807c08: configmap "payload-processing-external-model-plugins-e2e-extmod-a-807c08" not found Nov 02 18:15:07.120232 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:07.119775 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-plugins-config-volume podName:22fe51da-2ea2-4ab3-8b74-aa303cab4be2 nodeName:}" failed. No retries permitted until 2026-11-02 18:15:15.119762372 +0000 UTC m=+2486.908267891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" (UID: "22fe51da-2ea2-4ab3-8b74-aa303cab4be2") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-807c08" not found Nov 02 18:15:07.123806 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.123783 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" Nov 02 18:15:07.124150 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:07.124131 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-807c08/agc-cred-41558d72238a: secret "agc-cred-41558d72238a" not found Nov 02 18:15:07.124267 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:07.124153 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz: secret "agc-cred-41558d72238a" not found Nov 02 18:15:07.124267 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:07.124217 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-provider-credentials podName:22fe51da-2ea2-4ab3-8b74-aa303cab4be2 nodeName:}" failed. No retries permitted until 2026-11-02 18:15:15.124199943 +0000 UTC m=+2486.912705463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" (UID: "22fe51da-2ea2-4ab3-8b74-aa303cab4be2") : secret "agc-cred-41558d72238a" not found Nov 02 18:15:07.124813 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:07.124783 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-plugins-e2e-extmod-b-bee8f6: configmap "payload-processing-external-model-plugins-e2e-extmod-b-bee8f6" not found Nov 02 18:15:07.125091 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:07.125075 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-plugins-config-volume podName:76dbe462-e831-40a3-a735-79ce6c63ed41 nodeName:}" failed. No retries permitted until 2026-11-02 18:15:15.125054584 +0000 UTC m=+2486.913560106 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" (UID: "76dbe462-e831-40a3-a735-79ce6c63ed41") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-bee8f6" not found Nov 02 18:15:07.145370 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.145319 5549 generic.go:356] "Generic (PLEG): container finished" podID="76dbe462-e831-40a3-a735-79ce6c63ed41" containerID="b59975749f6f7732734f4a0f469303d3e5226189079632cce2cf17fcd7f45624" exitCode=0 Nov 02 18:15:07.145679 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.145582 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" event={"ID":"76dbe462-e831-40a3-a735-79ce6c63ed41","Type":"ContainerDied","Data":"b59975749f6f7732734f4a0f469303d3e5226189079632cce2cf17fcd7f45624"} Nov 02 18:15:07.149442 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.149408 5549 generic.go:356] "Generic (PLEG): container finished" podID="22fe51da-2ea2-4ab3-8b74-aa303cab4be2" containerID="48e33a32d4bba7172bf4a8fbd5bb817b2961412f061419f79a383fc9d8832f6f" exitCode=0 Nov 02 18:15:07.149662 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.149633 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" event={"ID":"22fe51da-2ea2-4ab3-8b74-aa303cab4be2","Type":"ContainerDied","Data":"48e33a32d4bba7172bf4a8fbd5bb817b2961412f061419f79a383fc9d8832f6f"} Nov 02 18:15:07.149737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.149697 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" event={"ID":"22fe51da-2ea2-4ab3-8b74-aa303cab4be2","Type":"ContainerDied","Data":"0d7bbffd70090c08e6837c975e8596718541af7326bf8fcf9bd08a2a1ccc08c7"} Nov 02 18:15:07.149737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.149731 5549 scope.go:122] "RemoveContainer" containerID="48e33a32d4bba7172bf4a8fbd5bb817b2961412f061419f79a383fc9d8832f6f" Nov 02 18:15:07.149897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.149882 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz" Nov 02 18:15:07.220562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.220522 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-plugins-config-volume\") pod \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\" (UID: \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\") " Nov 02 18:15:07.221506 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.221479 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-provider-credentials\") pod \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\" (UID: \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\") " Nov 02 18:15:07.222103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.222083 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-kube-api-access-bn4g7\" (UniqueName: \"kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-kube-api-access-bn4g7\") pod \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\" (UID: \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\") " Nov 02 18:15:07.222550 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.222528 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-routing-overlay-volume\") pod \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\" (UID: \"22fe51da-2ea2-4ab3-8b74-aa303cab4be2\") " Nov 02 18:15:07.223859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.221232 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-plugins-config-volume" pod "22fe51da-2ea2-4ab3-8b74-aa303cab4be2" (UID: "22fe51da-2ea2-4ab3-8b74-aa303cab4be2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:07.224547 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.224524 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-routing-overlay-volume" pod "22fe51da-2ea2-4ab3-8b74-aa303cab4be2" (UID: "22fe51da-2ea2-4ab3-8b74-aa303cab4be2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:07.235506 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.235480 5549 scope.go:122] "RemoveContainer" containerID="48e33a32d4bba7172bf4a8fbd5bb817b2961412f061419f79a383fc9d8832f6f" Nov 02 18:15:07.236178 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:07.236134 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e33a32d4bba7172bf4a8fbd5bb817b2961412f061419f79a383fc9d8832f6f\": container with ID starting with 48e33a32d4bba7172bf4a8fbd5bb817b2961412f061419f79a383fc9d8832f6f not found: ID does not exist" containerID="48e33a32d4bba7172bf4a8fbd5bb817b2961412f061419f79a383fc9d8832f6f" Nov 02 18:15:07.236356 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.236328 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e33a32d4bba7172bf4a8fbd5bb817b2961412f061419f79a383fc9d8832f6f"} err="failed to get container status \"48e33a32d4bba7172bf4a8fbd5bb817b2961412f061419f79a383fc9d8832f6f\": rpc error: code = NotFound desc = could not find container \"48e33a32d4bba7172bf4a8fbd5bb817b2961412f061419f79a383fc9d8832f6f\": container with ID starting with 48e33a32d4bba7172bf4a8fbd5bb817b2961412f061419f79a383fc9d8832f6f not found: ID does not exist" Nov 02 18:15:07.259971 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.259921 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-provider-credentials" pod "22fe51da-2ea2-4ab3-8b74-aa303cab4be2" (UID: "22fe51da-2ea2-4ab3-8b74-aa303cab4be2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:07.272264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.272227 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-kube-api-access-bn4g7" pod "22fe51da-2ea2-4ab3-8b74-aa303cab4be2" (UID: "22fe51da-2ea2-4ab3-8b74-aa303cab4be2"). InnerVolumeSpecName "kube-api-access-bn4g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:07.325319 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.325287 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:07.325509 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.325494 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:07.325575 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.325567 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:07.325634 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.325626 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn4g7\" (UniqueName: \"kubernetes.io/projected/22fe51da-2ea2-4ab3-8b74-aa303cab4be2-kube-api-access-bn4g7\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:07.378707 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.378680 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" Nov 02 18:15:07.426719 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.426682 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-routing-overlay-volume\") pod \"76dbe462-e831-40a3-a735-79ce6c63ed41\" (UID: \"76dbe462-e831-40a3-a735-79ce6c63ed41\") " Nov 02 18:15:07.426926 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.426842 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-plugins-config-volume\") pod \"76dbe462-e831-40a3-a735-79ce6c63ed41\" (UID: \"76dbe462-e831-40a3-a735-79ce6c63ed41\") " Nov 02 18:15:07.426926 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.426904 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-kube-api-access-2bn29\" (UniqueName: \"kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-kube-api-access-2bn29\") pod \"76dbe462-e831-40a3-a735-79ce6c63ed41\" (UID: \"76dbe462-e831-40a3-a735-79ce6c63ed41\") " Nov 02 18:15:07.427123 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.426936 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-provider-credentials\" (UniqueName: \"kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-provider-credentials\") pod \"76dbe462-e831-40a3-a735-79ce6c63ed41\" (UID: \"76dbe462-e831-40a3-a735-79ce6c63ed41\") " Nov 02 18:15:07.427245 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.427218 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-routing-overlay-volume" pod "76dbe462-e831-40a3-a735-79ce6c63ed41" (UID: "76dbe462-e831-40a3-a735-79ce6c63ed41"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:07.427384 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.427357 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-plugins-config-volume" pod "76dbe462-e831-40a3-a735-79ce6c63ed41" (UID: "76dbe462-e831-40a3-a735-79ce6c63ed41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:07.428316 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.428286 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:07.428316 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.428317 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76dbe462-e831-40a3-a735-79ce6c63ed41-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:07.429513 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.429485 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-provider-credentials" pod "76dbe462-e831-40a3-a735-79ce6c63ed41" (UID: "76dbe462-e831-40a3-a735-79ce6c63ed41"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:07.429586 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.429533 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-kube-api-access-2bn29" pod "76dbe462-e831-40a3-a735-79ce6c63ed41" (UID: "76dbe462-e831-40a3-a735-79ce6c63ed41"). InnerVolumeSpecName "kube-api-access-2bn29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:07.504219 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.504126 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:07.508741 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.508664 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz"] Nov 02 18:15:07.515846 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.515815 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-8579wd2xz"] Nov 02 18:15:07.529836 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.529805 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bn29\" (UniqueName: \"kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-kube-api-access-2bn29\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:07.529836 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.529835 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76dbe462-e831-40a3-a735-79ce6c63ed41-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:07.724037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.723937 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:07.890290 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:07.890261 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:08.152874 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:08.152791 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:08.161149 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:08.161124 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" Nov 02 18:15:08.161288 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:08.161151 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r" event={"ID":"76dbe462-e831-40a3-a735-79ce6c63ed41","Type":"ContainerDied","Data":"e473152d3cf4791ef819d75d8912783a812aeedf503a6d1f88410c6b7ca5bdd8"} Nov 02 18:15:08.161288 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:08.161214 5549 scope.go:122] "RemoveContainer" containerID="b59975749f6f7732734f4a0f469303d3e5226189079632cce2cf17fcd7f45624" Nov 02 18:15:08.189321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:08.189293 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r"] Nov 02 18:15:08.199183 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:08.199157 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758b8kn4r"] Nov 02 18:15:08.323944 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:08.323912 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22fe51da-2ea2-4ab3-8b74-aa303cab4be2" path="/var/lib/kubelet/pods/22fe51da-2ea2-4ab3-8b74-aa303cab4be2/volumes" Nov 02 18:15:08.324593 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:08.324561 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76dbe462-e831-40a3-a735-79ce6c63ed41" path="/var/lib/kubelet/pods/76dbe462-e831-40a3-a735-79ce6c63ed41/volumes" Nov 02 18:15:08.606979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:08.599445 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:09.342245 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:09.342212 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:09.924614 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:09.924581 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:11.283999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:11.279248 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:11.284512 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:11.284358 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:11.329021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:11.328944 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:12.158593 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:12.158546 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:12.227585 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:12.227554 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:15:13.017630 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:13.017592 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:13.921438 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:13.921380 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:14.119568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.119530 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:14.237745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.237713 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm"] Nov 02 18:15:14.239002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.238977 5549 cpu_manager.go:414] "containerMap: removing container" podUID="22fe51da-2ea2-4ab3-8b74-aa303cab4be2" containerName="payload-processing" Nov 02 18:15:14.239002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.239000 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fe51da-2ea2-4ab3-8b74-aa303cab4be2" containerName="payload-processing" Nov 02 18:15:14.239169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.239044 5549 cpu_manager.go:414] "containerMap: removing container" podUID="76dbe462-e831-40a3-a735-79ce6c63ed41" containerName="payload-processing" Nov 02 18:15:14.239169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.239049 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76dbe462-e831-40a3-a735-79ce6c63ed41" containerName="payload-processing" Nov 02 18:15:14.239247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.239228 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="76dbe462-e831-40a3-a735-79ce6c63ed41" containerName="payload-processing" Nov 02 18:15:14.239294 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.239257 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fe51da-2ea2-4ab3-8b74-aa303cab4be2" containerName="payload-processing" Nov 02 18:15:14.245068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.245044 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm" Nov 02 18:15:14.249139 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.249116 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:15:14.249480 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.249461 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm"] Nov 02 18:15:14.278236 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.278200 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:14.355635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.355592 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l67hg\" (UniqueName: \"kubernetes.io/projected/90437a6d-ed10-429a-a0bd-2ec2d891579e-kube-api-access-l67hg\") pod \"maas-api-revoke-sub-480b0c62-wlrpm\" (UID: \"90437a6d-ed10-429a-a0bd-2ec2d891579e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm" Nov 02 18:15:14.355836 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.355770 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90437a6d-ed10-429a-a0bd-2ec2d891579e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-480b0c62-wlrpm\" (UID: \"90437a6d-ed10-429a-a0bd-2ec2d891579e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm" Nov 02 18:15:14.355836 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.355816 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90437a6d-ed10-429a-a0bd-2ec2d891579e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-480b0c62-wlrpm\" (UID: \"90437a6d-ed10-429a-a0bd-2ec2d891579e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm" Nov 02 18:15:14.458139 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.458097 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90437a6d-ed10-429a-a0bd-2ec2d891579e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-480b0c62-wlrpm\" (UID: \"90437a6d-ed10-429a-a0bd-2ec2d891579e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm" Nov 02 18:15:14.458139 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.458150 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90437a6d-ed10-429a-a0bd-2ec2d891579e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-480b0c62-wlrpm\" (UID: \"90437a6d-ed10-429a-a0bd-2ec2d891579e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm" Nov 02 18:15:14.458638 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.458603 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l67hg\" (UniqueName: \"kubernetes.io/projected/90437a6d-ed10-429a-a0bd-2ec2d891579e-kube-api-access-l67hg\") pod \"maas-api-revoke-sub-480b0c62-wlrpm\" (UID: \"90437a6d-ed10-429a-a0bd-2ec2d891579e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm" Nov 02 18:15:14.459915 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.459386 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90437a6d-ed10-429a-a0bd-2ec2d891579e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-480b0c62-wlrpm\" (UID: \"90437a6d-ed10-429a-a0bd-2ec2d891579e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm" Nov 02 18:15:14.461552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.461529 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90437a6d-ed10-429a-a0bd-2ec2d891579e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-480b0c62-wlrpm\" (UID: \"90437a6d-ed10-429a-a0bd-2ec2d891579e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm" Nov 02 18:15:14.467657 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.467630 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l67hg\" (UniqueName: \"kubernetes.io/projected/90437a6d-ed10-429a-a0bd-2ec2d891579e-kube-api-access-l67hg\") pod \"maas-api-revoke-sub-480b0c62-wlrpm\" (UID: \"90437a6d-ed10-429a-a0bd-2ec2d891579e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm" Nov 02 18:15:14.570171 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.570078 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm" Nov 02 18:15:14.859737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.856739 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm"] Nov 02 18:15:14.859737 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:15:14.858100 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90437a6d_ed10_429a_a0bd_2ec2d891579e.slice/crio-9521a4ad0601162dce838b255a27ddb2263492a15a980c0c53c46cee64a78889 WatchSource:0}: Error finding container 9521a4ad0601162dce838b255a27ddb2263492a15a980c0c53c46cee64a78889: Status 404 returned error can't find the container with id 9521a4ad0601162dce838b255a27ddb2263492a15a980c0c53c46cee64a78889 Nov 02 18:15:14.928103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:14.927881 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:15.241273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:15.241224 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm" event={"ID":"90437a6d-ed10-429a-a0bd-2ec2d891579e","Type":"ContainerStarted","Data":"9521a4ad0601162dce838b255a27ddb2263492a15a980c0c53c46cee64a78889"} Nov 02 18:15:15.312857 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:15.312827 5549 scope.go:122] "RemoveContainer" containerID="99c7b43b0dbfd6eab1988f27cb3ff6a2563ef696196df9a89d296b6d1099a3e1" Nov 02 18:15:15.578211 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:15.578177 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:15.728649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:15.725515 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:15.873535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:15.873493 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:16.251771 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:16.251733 5549 generic.go:356] "Generic (PLEG): container finished" podID="90437a6d-ed10-429a-a0bd-2ec2d891579e" containerID="31819625a2d7ec8d9093073cbf61a5836615e99c76bab68e58a6f5637711dfed" exitCode=0 Nov 02 18:15:16.252224 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:16.251815 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm" event={"ID":"90437a6d-ed10-429a-a0bd-2ec2d891579e","Type":"ContainerDied","Data":"31819625a2d7ec8d9093073cbf61a5836615e99c76bab68e58a6f5637711dfed"} Nov 02 18:15:16.253891 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:16.253869 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" event={"ID":"ca2a76db-edd0-46a7-b117-2d06e99f3840","Type":"ContainerStarted","Data":"a2d6b23055b5ed5b3ef630b03f40b8e6a6a1e5f9f21a69d52a3dea83acf77a65"} Nov 02 18:15:16.729988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:16.729591 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:17.265575 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:17.265540 5549 generic.go:356] "Generic (PLEG): container finished" podID="ca2a76db-edd0-46a7-b117-2d06e99f3840" containerID="a2d6b23055b5ed5b3ef630b03f40b8e6a6a1e5f9f21a69d52a3dea83acf77a65" exitCode=60 Nov 02 18:15:17.266057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:17.265620 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" event={"ID":"ca2a76db-edd0-46a7-b117-2d06e99f3840","Type":"ContainerDied","Data":"a2d6b23055b5ed5b3ef630b03f40b8e6a6a1e5f9f21a69d52a3dea83acf77a65"} Nov 02 18:15:17.266057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:17.265730 5549 scope.go:122] "RemoveContainer" containerID="99c7b43b0dbfd6eab1988f27cb3ff6a2563ef696196df9a89d296b6d1099a3e1" Nov 02 18:15:17.266547 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:17.266194 5549 scope.go:122] "RemoveContainer" containerID="a2d6b23055b5ed5b3ef630b03f40b8e6a6a1e5f9f21a69d52a3dea83acf77a65" Nov 02 18:15:17.266653 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:17.266550 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749_odh-ai-gateway-infra(ca2a76db-edd0-46a7-b117-2d06e99f3840)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" podUID="ca2a76db-edd0-46a7-b117-2d06e99f3840" Nov 02 18:15:17.305239 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:17.305204 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749"] Nov 02 18:15:17.313353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:17.313126 5549 scope.go:122] "RemoveContainer" containerID="d2d114fc414dbb75c9822ed2657d8bbbc2a1d156a24d900672e7c55f44eaa281" Nov 02 18:15:17.514816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:17.514780 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:17.559685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:17.559610 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm" Nov 02 18:15:17.610357 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:17.610303 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:15:17.670889 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:17.670850 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90437a6d-ed10-429a-a0bd-2ec2d891579e-kube-api-access-l67hg\" (UniqueName: \"kubernetes.io/projected/90437a6d-ed10-429a-a0bd-2ec2d891579e-kube-api-access-l67hg\") pod \"90437a6d-ed10-429a-a0bd-2ec2d891579e\" (UID: \"90437a6d-ed10-429a-a0bd-2ec2d891579e\") " Nov 02 18:15:17.671206 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:17.671181 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90437a6d-ed10-429a-a0bd-2ec2d891579e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90437a6d-ed10-429a-a0bd-2ec2d891579e-maas-api-xks-ca\") pod \"90437a6d-ed10-429a-a0bd-2ec2d891579e\" (UID: \"90437a6d-ed10-429a-a0bd-2ec2d891579e\") " Nov 02 18:15:17.671316 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:17.671293 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90437a6d-ed10-429a-a0bd-2ec2d891579e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90437a6d-ed10-429a-a0bd-2ec2d891579e-maas-api-openshift-ca\") pod \"90437a6d-ed10-429a-a0bd-2ec2d891579e\" (UID: \"90437a6d-ed10-429a-a0bd-2ec2d891579e\") " Nov 02 18:15:17.671725 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:17.671701 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90437a6d-ed10-429a-a0bd-2ec2d891579e-maas-api-openshift-ca" pod "90437a6d-ed10-429a-a0bd-2ec2d891579e" (UID: "90437a6d-ed10-429a-a0bd-2ec2d891579e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:17.672139 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:17.672118 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90437a6d-ed10-429a-a0bd-2ec2d891579e-maas-api-openshift-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:17.673270 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:17.673248 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90437a6d-ed10-429a-a0bd-2ec2d891579e-kube-api-access-l67hg" pod "90437a6d-ed10-429a-a0bd-2ec2d891579e" (UID: "90437a6d-ed10-429a-a0bd-2ec2d891579e"). InnerVolumeSpecName "kube-api-access-l67hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:17.673393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:17.673362 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90437a6d-ed10-429a-a0bd-2ec2d891579e-maas-api-xks-ca" pod "90437a6d-ed10-429a-a0bd-2ec2d891579e" (UID: "90437a6d-ed10-429a-a0bd-2ec2d891579e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:15:17.773768 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:17.773728 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90437a6d-ed10-429a-a0bd-2ec2d891579e-maas-api-xks-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:17.773768 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:17.773761 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l67hg\" (UniqueName: \"kubernetes.io/projected/90437a6d-ed10-429a-a0bd-2ec2d891579e-kube-api-access-l67hg\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:18.244976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:18.241009 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:18.280106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:18.280068 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" event={"ID":"ea777210-3c0f-4e8b-be3b-4b07895990b7","Type":"ContainerStarted","Data":"96bd8ed2825c53ede282a1c44514a3ee6fa117a7aa2d426abf708d06c7e58856"} Nov 02 18:15:18.282318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:18.282291 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm" Nov 02 18:15:18.282489 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:18.282327 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm" event={"ID":"90437a6d-ed10-429a-a0bd-2ec2d891579e","Type":"ContainerDied","Data":"9521a4ad0601162dce838b255a27ddb2263492a15a980c0c53c46cee64a78889"} Nov 02 18:15:18.282489 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:18.282376 5549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9521a4ad0601162dce838b255a27ddb2263492a15a980c0c53c46cee64a78889" Nov 02 18:15:18.470071 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:18.470042 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" Nov 02 18:15:18.593257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:18.593178 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca2a76db-edd0-46a7-b117-2d06e99f3840-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2a76db-edd0-46a7-b117-2d06e99f3840-maas-api-service-ca\") pod \"ca2a76db-edd0-46a7-b117-2d06e99f3840\" (UID: \"ca2a76db-edd0-46a7-b117-2d06e99f3840\") " Nov 02 18:15:18.593411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:18.593317 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2a76db-edd0-46a7-b117-2d06e99f3840-kube-api-access-8wq6q\" (UniqueName: \"kubernetes.io/projected/ca2a76db-edd0-46a7-b117-2d06e99f3840-kube-api-access-8wq6q\") pod \"ca2a76db-edd0-46a7-b117-2d06e99f3840\" (UID: \"ca2a76db-edd0-46a7-b117-2d06e99f3840\") " Nov 02 18:15:18.593536 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:18.593513 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2a76db-edd0-46a7-b117-2d06e99f3840-maas-api-service-ca" pod "ca2a76db-edd0-46a7-b117-2d06e99f3840" (UID: "ca2a76db-edd0-46a7-b117-2d06e99f3840"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:18.594184 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:18.594165 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2a76db-edd0-46a7-b117-2d06e99f3840-maas-api-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:18.595539 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:18.595519 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2a76db-edd0-46a7-b117-2d06e99f3840-kube-api-access-8wq6q" pod "ca2a76db-edd0-46a7-b117-2d06e99f3840" (UID: "ca2a76db-edd0-46a7-b117-2d06e99f3840"). InnerVolumeSpecName "kube-api-access-8wq6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:18.695707 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:18.695671 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wq6q\" (UniqueName: \"kubernetes.io/projected/ca2a76db-edd0-46a7-b117-2d06e99f3840-kube-api-access-8wq6q\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:19.173103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:19.169192 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:19.292681 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:19.292645 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" Nov 02 18:15:19.292681 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:19.292656 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749" event={"ID":"ca2a76db-edd0-46a7-b117-2d06e99f3840","Type":"ContainerDied","Data":"8e808a9192dddf8ecf05d384600a6fa26e5d26d1d91145ac040f8df389f42bf3"} Nov 02 18:15:19.293225 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:19.292748 5549 scope.go:122] "RemoveContainer" containerID="a2d6b23055b5ed5b3ef630b03f40b8e6a6a1e5f9f21a69d52a3dea83acf77a65" Nov 02 18:15:19.295095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:19.295065 5549 generic.go:356] "Generic (PLEG): container finished" podID="ea777210-3c0f-4e8b-be3b-4b07895990b7" containerID="96bd8ed2825c53ede282a1c44514a3ee6fa117a7aa2d426abf708d06c7e58856" exitCode=60 Nov 02 18:15:19.295213 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:19.295120 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" event={"ID":"ea777210-3c0f-4e8b-be3b-4b07895990b7","Type":"ContainerDied","Data":"96bd8ed2825c53ede282a1c44514a3ee6fa117a7aa2d426abf708d06c7e58856"} Nov 02 18:15:19.295604 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:19.295585 5549 scope.go:122] "RemoveContainer" containerID="96bd8ed2825c53ede282a1c44514a3ee6fa117a7aa2d426abf708d06c7e58856" Nov 02 18:15:19.295933 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:19.295911 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht_odh-ai-gateway-infra(ea777210-3c0f-4e8b-be3b-4b07895990b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" podUID="ea777210-3c0f-4e8b-be3b-4b07895990b7" Nov 02 18:15:19.305350 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:19.305329 5549 scope.go:122] "RemoveContainer" containerID="d2d114fc414dbb75c9822ed2657d8bbbc2a1d156a24d900672e7c55f44eaa281" Nov 02 18:15:19.338676 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:19.338646 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749"] Nov 02 18:15:19.343919 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:19.343896 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht"] Nov 02 18:15:19.349124 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:19.349101 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894055-g8749"] Nov 02 18:15:19.957976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:19.957448 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:20.340517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:20.340477 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca2a76db-edd0-46a7-b117-2d06e99f3840" path="/var/lib/kubelet/pods/ca2a76db-edd0-46a7-b117-2d06e99f3840/volumes" Nov 02 18:15:20.511264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:20.511230 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" Nov 02 18:15:20.626969 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:20.626858 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea777210-3c0f-4e8b-be3b-4b07895990b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea777210-3c0f-4e8b-be3b-4b07895990b7-maas-api-service-ca\") pod \"ea777210-3c0f-4e8b-be3b-4b07895990b7\" (UID: \"ea777210-3c0f-4e8b-be3b-4b07895990b7\") " Nov 02 18:15:20.627236 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:20.627220 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea777210-3c0f-4e8b-be3b-4b07895990b7-kube-api-access-gwx52\" (UniqueName: \"kubernetes.io/projected/ea777210-3c0f-4e8b-be3b-4b07895990b7-kube-api-access-gwx52\") pod \"ea777210-3c0f-4e8b-be3b-4b07895990b7\" (UID: \"ea777210-3c0f-4e8b-be3b-4b07895990b7\") " Nov 02 18:15:20.627318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:20.627255 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea777210-3c0f-4e8b-be3b-4b07895990b7-maas-api-service-ca" pod "ea777210-3c0f-4e8b-be3b-4b07895990b7" (UID: "ea777210-3c0f-4e8b-be3b-4b07895990b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:20.627786 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:20.627768 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea777210-3c0f-4e8b-be3b-4b07895990b7-maas-api-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:20.629380 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:20.629360 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea777210-3c0f-4e8b-be3b-4b07895990b7-kube-api-access-gwx52" pod "ea777210-3c0f-4e8b-be3b-4b07895990b7" (UID: "ea777210-3c0f-4e8b-be3b-4b07895990b7"). InnerVolumeSpecName "kube-api-access-gwx52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:20.729024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:20.728988 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwx52\" (UniqueName: \"kubernetes.io/projected/ea777210-3c0f-4e8b-be3b-4b07895990b7-kube-api-access-gwx52\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:20.823976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:20.813970 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:21.331807 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:21.331773 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" Nov 02 18:15:21.332177 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:21.331894 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht" event={"ID":"ea777210-3c0f-4e8b-be3b-4b07895990b7","Type":"ContainerDied","Data":"691bd1b3232e507a261888efe6c0636fefe7e63b92c025ded80d471952419351"} Nov 02 18:15:21.332293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:21.332261 5549 scope.go:122] "RemoveContainer" containerID="96bd8ed2825c53ede282a1c44514a3ee6fa117a7aa2d426abf708d06c7e58856" Nov 02 18:15:21.365272 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:21.365235 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht"] Nov 02 18:15:21.369891 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:21.369861 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894055-wfmht"] Nov 02 18:15:21.700692 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:21.700660 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:22.327339 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:22.327299 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea777210-3c0f-4e8b-be3b-4b07895990b7" path="/var/lib/kubelet/pods/ea777210-3c0f-4e8b-be3b-4b07895990b7/volumes" Nov 02 18:15:22.643561 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:22.643478 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:23.512627 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:23.512390 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:24.355974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:24.347857 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:25.233457 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:25.233425 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:25.268037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:25.268000 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fb475f76-m7d62"] Nov 02 18:15:25.368489 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:25.368452 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:26.642975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:26.639365 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:27.222262 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:27.222229 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:27.364976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:27.364073 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:27.661627 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:27.654179 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:28.517087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:28.516992 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:28.658356 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:28.658316 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:29.326135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:29.326088 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:30.122544 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:30.122499 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:30.268745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:30.268690 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" podUID="46a34c22-53fc-4da2-a5ae-60d86f7160e7" containerName="payload-processing" containerID="cri-o://2bb2d9f3690bc8d871e60e3645ec88deea0a131e7230139832568a9717722c9d" gracePeriod=25 Nov 02 18:15:30.532654 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:30.532612 5549 generic.go:356] "Generic (PLEG): container finished" podID="46a34c22-53fc-4da2-a5ae-60d86f7160e7" containerID="2bb2d9f3690bc8d871e60e3645ec88deea0a131e7230139832568a9717722c9d" exitCode=0 Nov 02 18:15:30.533094 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:30.532675 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" event={"ID":"46a34c22-53fc-4da2-a5ae-60d86f7160e7","Type":"ContainerDied","Data":"2bb2d9f3690bc8d871e60e3645ec88deea0a131e7230139832568a9717722c9d"} Nov 02 18:15:30.582496 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:30.582470 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" Nov 02 18:15:30.661852 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:30.661748 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a34c22-53fc-4da2-a5ae-60d86f7160e7-kube-api-access-l9m6m\" (UniqueName: \"kubernetes.io/projected/46a34c22-53fc-4da2-a5ae-60d86f7160e7-kube-api-access-l9m6m\") pod \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\" (UID: \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\") " Nov 02 18:15:30.662070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:30.661893 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a34c22-53fc-4da2-a5ae-60d86f7160e7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/46a34c22-53fc-4da2-a5ae-60d86f7160e7-provider-credentials\") pod \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\" (UID: \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\") " Nov 02 18:15:30.662070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:30.661941 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46a34c22-53fc-4da2-a5ae-60d86f7160e7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46a34c22-53fc-4da2-a5ae-60d86f7160e7-routing-overlay-volume\") pod \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\" (UID: \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\") " Nov 02 18:15:30.662322 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:30.662293 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46a34c22-53fc-4da2-a5ae-60d86f7160e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a34c22-53fc-4da2-a5ae-60d86f7160e7-plugins-config-volume\") pod \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\" (UID: \"46a34c22-53fc-4da2-a5ae-60d86f7160e7\") " Nov 02 18:15:30.662430 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:30.662375 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a34c22-53fc-4da2-a5ae-60d86f7160e7-routing-overlay-volume" pod "46a34c22-53fc-4da2-a5ae-60d86f7160e7" (UID: "46a34c22-53fc-4da2-a5ae-60d86f7160e7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:30.662668 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:30.662645 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a34c22-53fc-4da2-a5ae-60d86f7160e7-plugins-config-volume" pod "46a34c22-53fc-4da2-a5ae-60d86f7160e7" (UID: "46a34c22-53fc-4da2-a5ae-60d86f7160e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:30.663385 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:30.663360 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46a34c22-53fc-4da2-a5ae-60d86f7160e7-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:30.663463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:30.663392 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a34c22-53fc-4da2-a5ae-60d86f7160e7-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:30.664701 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:30.664665 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a34c22-53fc-4da2-a5ae-60d86f7160e7-kube-api-access-l9m6m" pod "46a34c22-53fc-4da2-a5ae-60d86f7160e7" (UID: "46a34c22-53fc-4da2-a5ae-60d86f7160e7"). InnerVolumeSpecName "kube-api-access-l9m6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:30.664833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:30.664807 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a34c22-53fc-4da2-a5ae-60d86f7160e7-provider-credentials" pod "46a34c22-53fc-4da2-a5ae-60d86f7160e7" (UID: "46a34c22-53fc-4da2-a5ae-60d86f7160e7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:30.765285 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:30.765249 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9m6m\" (UniqueName: \"kubernetes.io/projected/46a34c22-53fc-4da2-a5ae-60d86f7160e7-kube-api-access-l9m6m\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:30.765285 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:30.765280 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46a34c22-53fc-4da2-a5ae-60d86f7160e7-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:31.047668 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:31.047632 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:31.453031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:31.452995 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:31.543592 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:31.543559 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" Nov 02 18:15:31.543592 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:31.543590 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56fb475f76-m7d62" event={"ID":"46a34c22-53fc-4da2-a5ae-60d86f7160e7","Type":"ContainerDied","Data":"e1991df61459ea6cddc901d914c0dc00e25dee75f9650bffefcb8f9218cc7b8a"} Nov 02 18:15:31.544373 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:31.543658 5549 scope.go:122] "RemoveContainer" containerID="2bb2d9f3690bc8d871e60e3645ec88deea0a131e7230139832568a9717722c9d" Nov 02 18:15:31.574128 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:31.574093 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fb475f76-m7d62"] Nov 02 18:15:31.579276 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:31.579250 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fb475f76-m7d62"] Nov 02 18:15:31.600803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:31.600774 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:31.840113 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:31.836493 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:32.323602 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:32.323570 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46a34c22-53fc-4da2-a5ae-60d86f7160e7" path="/var/lib/kubelet/pods/46a34c22-53fc-4da2-a5ae-60d86f7160e7/volumes" Nov 02 18:15:32.548141 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:32.548107 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:32.683352 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:32.683270 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:33.506708 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:33.506669 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:34.149648 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:34.149594 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:34.834876 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:34.834825 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:35.547432 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:35.547385 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:36.247339 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:36.247305 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:37.339329 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:37.339241 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:37.482283 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:37.482252 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:38.296983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:38.296653 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:39.012947 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:39.012913 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:39.711037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:39.711000 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:40.208252 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:40.208219 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:40.352342 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:40.352303 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:40.494767 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:40.494687 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:41.065264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:41.065192 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:41.836716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:41.836638 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:42.503229 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:42.503166 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:43.151571 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:43.151529 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:43.864135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:43.862777 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:44.529355 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:44.529316 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:45.220523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:45.220488 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:45.908491 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:45.906336 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:46.557979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:46.553028 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:47.238975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:47.238505 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:47.591123 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:15:47.591044 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:15:47.898769 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:47.898667 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:48.565352 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:48.565294 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:49.302516 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:49.302449 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:49.925993 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:49.918179 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:50.701036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:50.700975 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:51.438637 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:51.438598 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:52.115088 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:52.115058 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:52.244131 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:52.244098 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:52.454463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:52.454428 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:52.996882 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:52.996844 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:53.743435 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:53.743399 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:54.458836 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:54.458800 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:55.214056 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:55.214015 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:55.917237 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:55.913411 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:56.557978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:56.553124 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:57.208819 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:57.208008 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:57.947034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:57.946998 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:58.608945 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:58.608656 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:59.378537 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:59.378471 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:59.736424 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:15:59.736383 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:00.129299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:00.129211 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:00.316912 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:00.306923 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:00.609976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:00.603282 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:01.195739 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:01.195698 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:01.508723 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:01.508438 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:01.936320 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:01.936232 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.390899 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv"] Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392446 5549 cpu_manager.go:414] "containerMap: removing container" podUID="ea777210-3c0f-4e8b-be3b-4b07895990b7" containerName="cleanup" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392471 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea777210-3c0f-4e8b-be3b-4b07895990b7" containerName="cleanup" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392500 5549 cpu_manager.go:414] "containerMap: removing container" podUID="ca2a76db-edd0-46a7-b117-2d06e99f3840" containerName="cleanup" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392507 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca2a76db-edd0-46a7-b117-2d06e99f3840" containerName="cleanup" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392538 5549 cpu_manager.go:414] "containerMap: removing container" podUID="ca2a76db-edd0-46a7-b117-2d06e99f3840" containerName="cleanup" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392546 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca2a76db-edd0-46a7-b117-2d06e99f3840" containerName="cleanup" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392558 5549 cpu_manager.go:414] "containerMap: removing container" podUID="ea777210-3c0f-4e8b-be3b-4b07895990b7" containerName="cleanup" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392566 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea777210-3c0f-4e8b-be3b-4b07895990b7" containerName="cleanup" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392585 5549 cpu_manager.go:414] "containerMap: removing container" podUID="ca2a76db-edd0-46a7-b117-2d06e99f3840" containerName="cleanup" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392591 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca2a76db-edd0-46a7-b117-2d06e99f3840" containerName="cleanup" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392624 5549 cpu_manager.go:414] "containerMap: removing container" podUID="90437a6d-ed10-429a-a0bd-2ec2d891579e" containerName="revoke-keys" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392633 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90437a6d-ed10-429a-a0bd-2ec2d891579e" containerName="revoke-keys" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392655 5549 cpu_manager.go:414] "containerMap: removing container" podUID="46a34c22-53fc-4da2-a5ae-60d86f7160e7" containerName="payload-processing" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392662 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a34c22-53fc-4da2-a5ae-60d86f7160e7" containerName="payload-processing" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392679 5549 cpu_manager.go:414] "containerMap: removing container" podUID="ea777210-3c0f-4e8b-be3b-4b07895990b7" containerName="cleanup" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392686 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea777210-3c0f-4e8b-be3b-4b07895990b7" containerName="cleanup" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392910 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca2a76db-edd0-46a7-b117-2d06e99f3840" containerName="cleanup" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392928 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca2a76db-edd0-46a7-b117-2d06e99f3840" containerName="cleanup" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392938 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea777210-3c0f-4e8b-be3b-4b07895990b7" containerName="cleanup" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392967 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca2a76db-edd0-46a7-b117-2d06e99f3840" containerName="cleanup" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392985 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="90437a6d-ed10-429a-a0bd-2ec2d891579e" containerName="revoke-keys" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.392998 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a34c22-53fc-4da2-a5ae-60d86f7160e7" containerName="payload-processing" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.393007 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea777210-3c0f-4e8b-be3b-4b07895990b7" containerName="cleanup" Nov 02 18:16:02.404975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.400679 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:02.431009 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.430947 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" Nov 02 18:16:02.431009 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.430998 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv"] Nov 02 18:16:02.495582 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.495547 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"agc-cred-0ebc5b098a14\"" Nov 02 18:16:02.495811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.495785 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"payload-processing-external-model-e2e--59b59218-dockercfg-lp4xp\"" Nov 02 18:16:02.495891 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.495866 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"agc-cred-811d350f868d\"" Nov 02 18:16:02.496018 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.495943 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"openshift-service-ca.crt\"" Nov 02 18:16:02.496087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.496040 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"payload-processing-external-model-plugins-e2e-extmod-a-36a991\"" Nov 02 18:16:02.498984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.495683 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"routing-overlay\"" Nov 02 18:16:02.500114 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.500092 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"kube-root-ca.crt\"" Nov 02 18:16:02.542315 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.542278 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:02.598879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.598835 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7tx9\" (UniqueName: \"kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-kube-api-access-n7tx9\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv\" (UID: \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" Nov 02 18:16:02.599217 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.599196 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv\" (UID: \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" Nov 02 18:16:02.599404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.599382 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv\" (UID: \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" Nov 02 18:16:02.599482 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.599439 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv\" (UID: \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" Nov 02 18:16:02.701441 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.701402 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv\" (UID: \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" Nov 02 18:16:02.701639 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.701527 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv\" (UID: \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" Nov 02 18:16:02.701639 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.701558 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv\" (UID: \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" Nov 02 18:16:02.701639 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.701610 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7tx9\" (UniqueName: \"kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-kube-api-access-n7tx9\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv\" (UID: \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" Nov 02 18:16:02.702209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.702184 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv\" (UID: \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" Nov 02 18:16:02.702353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.702334 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv\" (UID: \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" Nov 02 18:16:02.704099 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.704079 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv\" (UID: \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" Nov 02 18:16:02.709650 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.709623 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7tx9\" (UniqueName: \"kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-kube-api-access-n7tx9\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv\" (UID: \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" Nov 02 18:16:02.726097 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.726069 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm"] Nov 02 18:16:02.727295 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.727276 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea777210-3c0f-4e8b-be3b-4b07895990b7" containerName="cleanup" Nov 02 18:16:02.731749 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.731732 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" Nov 02 18:16:02.736595 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.736563 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"routing-overlay\"" Nov 02 18:16:02.738575 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.738542 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"agc-cred-ff45ed7b3970\"" Nov 02 18:16:02.738717 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.738594 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"payload-processing-external-model-e2e--96b2e561-dockercfg-v9qrb\"" Nov 02 18:16:02.738717 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.738646 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"agc-cred-12ef34d6c6c7\"" Nov 02 18:16:02.739500 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.739477 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm"] Nov 02 18:16:02.821977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.819663 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2db162\"" Nov 02 18:16:02.821977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.820400 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"kube-root-ca.crt\"" Nov 02 18:16:02.838988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.828165 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"openshift-service-ca.crt\"" Nov 02 18:16:02.846976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.840088 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" Nov 02 18:16:02.953021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.950808 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm\" (UID: \"31153884-0178-422c-b089-b06f4e6b149b\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" Nov 02 18:16:02.953021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.950929 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31153884-0178-422c-b089-b06f4e6b149b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm\" (UID: \"31153884-0178-422c-b089-b06f4e6b149b\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" Nov 02 18:16:02.953021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.951038 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31153884-0178-422c-b089-b06f4e6b149b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm\" (UID: \"31153884-0178-422c-b089-b06f4e6b149b\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" Nov 02 18:16:02.953021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.951213 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sshx\" (UniqueName: \"kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-kube-api-access-6sshx\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm\" (UID: \"31153884-0178-422c-b089-b06f4e6b149b\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" Nov 02 18:16:02.965143 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.963900 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5"] Nov 02 18:16:02.965143 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:02.964122 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5" podUID="7b3ed03b-5899-4984-b221-12059d4ea928" containerName="sim" containerID="cri-o://500ebcc6450a5aba18847ea5901cbb35487a2e1a89b49d3e73f7b432bb4cd242" gracePeriod=30 Nov 02 18:16:03.068096 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.068058 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:03.071126 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.071096 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sshx\" (UniqueName: \"kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-kube-api-access-6sshx\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm\" (UID: \"31153884-0178-422c-b089-b06f4e6b149b\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" Nov 02 18:16:03.071316 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.071296 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm\" (UID: \"31153884-0178-422c-b089-b06f4e6b149b\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" Nov 02 18:16:03.071412 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.071394 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31153884-0178-422c-b089-b06f4e6b149b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm\" (UID: \"31153884-0178-422c-b089-b06f4e6b149b\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" Nov 02 18:16:03.071506 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.071493 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31153884-0178-422c-b089-b06f4e6b149b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm\" (UID: \"31153884-0178-422c-b089-b06f4e6b149b\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" Nov 02 18:16:03.072504 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.072476 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31153884-0178-422c-b089-b06f4e6b149b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm\" (UID: \"31153884-0178-422c-b089-b06f4e6b149b\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" Nov 02 18:16:03.091456 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.084324 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm\" (UID: \"31153884-0178-422c-b089-b06f4e6b149b\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" Nov 02 18:16:03.092592 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.092020 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31153884-0178-422c-b089-b06f4e6b149b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm\" (UID: \"31153884-0178-422c-b089-b06f4e6b149b\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" Nov 02 18:16:03.100670 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.096201 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sshx\" (UniqueName: \"kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-kube-api-access-6sshx\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm\" (UID: \"31153884-0178-422c-b089-b06f4e6b149b\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" Nov 02 18:16:03.101857 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.101727 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" Nov 02 18:16:03.278047 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:03.274623 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b3ed03b_5899_4984_b221_12059d4ea928.slice/crio-conmon-500ebcc6450a5aba18847ea5901cbb35487a2e1a89b49d3e73f7b432bb4cd242.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b3ed03b_5899_4984_b221_12059d4ea928.slice/crio-500ebcc6450a5aba18847ea5901cbb35487a2e1a89b49d3e73f7b432bb4cd242.scope\": RecentStats: unable to find data in memory cache]" Nov 02 18:16:03.278047 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:03.274915 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b3ed03b_5899_4984_b221_12059d4ea928.slice/crio-500ebcc6450a5aba18847ea5901cbb35487a2e1a89b49d3e73f7b432bb4cd242.scope\": RecentStats: unable to find data in memory cache]" Nov 02 18:16:03.452363 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.452327 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv"] Nov 02 18:16:03.453912 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:16:03.453835 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86914c05_ecb6_4c59_b4f9_ed49aa828caf.slice/crio-e1c2c9b2844f8c8fa0b4ca95716b095775a2e1d7aadd7f94ca4eff82cdb09bbc WatchSource:0}: Error finding container e1c2c9b2844f8c8fa0b4ca95716b095775a2e1d7aadd7f94ca4eff82cdb09bbc: Status 404 returned error can't find the container with id e1c2c9b2844f8c8fa0b4ca95716b095775a2e1d7aadd7f94ca4eff82cdb09bbc Nov 02 18:16:03.548073 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.547739 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5" Nov 02 18:16:03.558534 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:16:03.558505 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31153884_0178_422c_b089_b06f4e6b149b.slice/crio-815bceeace7a844a0aac5928e3f579216e8f0949485705882d942ac65b16fb91 WatchSource:0}: Error finding container 815bceeace7a844a0aac5928e3f579216e8f0949485705882d942ac65b16fb91: Status 404 returned error can't find the container with id 815bceeace7a844a0aac5928e3f579216e8f0949485705882d942ac65b16fb91 Nov 02 18:16:03.560349 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.560326 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm"] Nov 02 18:16:03.589196 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.589165 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b3ed03b-5899-4984-b221-12059d4ea928-tls\" (UniqueName: \"kubernetes.io/secret/7b3ed03b-5899-4984-b221-12059d4ea928-tls\") pod \"7b3ed03b-5899-4984-b221-12059d4ea928\" (UID: \"7b3ed03b-5899-4984-b221-12059d4ea928\") " Nov 02 18:16:03.589549 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.589523 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b3ed03b-5899-4984-b221-12059d4ea928-kube-api-access-clqms\" (UniqueName: \"kubernetes.io/projected/7b3ed03b-5899-4984-b221-12059d4ea928-kube-api-access-clqms\") pod \"7b3ed03b-5899-4984-b221-12059d4ea928\" (UID: \"7b3ed03b-5899-4984-b221-12059d4ea928\") " Nov 02 18:16:03.593856 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.593829 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b3ed03b-5899-4984-b221-12059d4ea928-tls" pod "7b3ed03b-5899-4984-b221-12059d4ea928" (UID: "7b3ed03b-5899-4984-b221-12059d4ea928"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:16:03.594027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.593887 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3ed03b-5899-4984-b221-12059d4ea928-kube-api-access-clqms" pod "7b3ed03b-5899-4984-b221-12059d4ea928" (UID: "7b3ed03b-5899-4984-b221-12059d4ea928"). InnerVolumeSpecName "kube-api-access-clqms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:03.692008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.691976 5549 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7b3ed03b-5899-4984-b221-12059d4ea928-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:03.692008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.692007 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clqms\" (UniqueName: \"kubernetes.io/projected/7b3ed03b-5899-4984-b221-12059d4ea928-kube-api-access-clqms\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:03.787056 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.787026 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:03.975292 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:03.974893 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:04.116297 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.116258 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" event={"ID":"86914c05-ecb6-4c59-b4f9-ed49aa828caf","Type":"ContainerStarted","Data":"892a00c0d92f82a423dbcf62f38f68dd04a2960f8467e2767f52f26c96448de2"} Nov 02 18:16:04.116596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.116564 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" event={"ID":"86914c05-ecb6-4c59-b4f9-ed49aa828caf","Type":"ContainerStarted","Data":"e1c2c9b2844f8c8fa0b4ca95716b095775a2e1d7aadd7f94ca4eff82cdb09bbc"} Nov 02 18:16:04.117969 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.117932 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" Nov 02 18:16:04.129974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.127996 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" event={"ID":"31153884-0178-422c-b089-b06f4e6b149b","Type":"ContainerStarted","Data":"16ed40459457eec5967fc1db8ab4db764ee63e9afed892372a4c4dc1e4d00738"} Nov 02 18:16:04.129974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.128054 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" event={"ID":"31153884-0178-422c-b089-b06f4e6b149b","Type":"ContainerStarted","Data":"815bceeace7a844a0aac5928e3f579216e8f0949485705882d942ac65b16fb91"} Nov 02 18:16:04.129974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.129008 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" Nov 02 18:16:04.133055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.133021 5549 generic.go:356] "Generic (PLEG): container finished" podID="7b3ed03b-5899-4984-b221-12059d4ea928" containerID="500ebcc6450a5aba18847ea5901cbb35487a2e1a89b49d3e73f7b432bb4cd242" exitCode=0 Nov 02 18:16:04.133248 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.133229 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5" event={"ID":"7b3ed03b-5899-4984-b221-12059d4ea928","Type":"ContainerDied","Data":"500ebcc6450a5aba18847ea5901cbb35487a2e1a89b49d3e73f7b432bb4cd242"} Nov 02 18:16:04.133352 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.133268 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5" event={"ID":"7b3ed03b-5899-4984-b221-12059d4ea928","Type":"ContainerDied","Data":"680b83365af99be53554bf6c6b21776c5c49eec42b03e3ba4147df56b5134f11"} Nov 02 18:16:04.133352 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.133292 5549 scope.go:122] "RemoveContainer" containerID="500ebcc6450a5aba18847ea5901cbb35487a2e1a89b49d3e73f7b432bb4cd242" Nov 02 18:16:04.133476 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.133453 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5" Nov 02 18:16:04.145428 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.144810 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" podStartSLOduration=2.144791505 podStartE2EDuration="2.144791505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:04.144196764 +0000 UTC m=+2535.932702294" watchObservedRunningTime="2026-11-02 18:16:04.144791505 +0000 UTC m=+2535.933297034" Nov 02 18:16:04.184605 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.184554 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" podStartSLOduration=2.184539179 podStartE2EDuration="2.184539179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:04.173545481 +0000 UTC m=+2535.962051008" watchObservedRunningTime="2026-11-02 18:16:04.184539179 +0000 UTC m=+2535.973044706" Nov 02 18:16:04.186213 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.186189 5549 scope.go:122] "RemoveContainer" containerID="500ebcc6450a5aba18847ea5901cbb35487a2e1a89b49d3e73f7b432bb4cd242" Nov 02 18:16:04.186579 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:04.186552 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"500ebcc6450a5aba18847ea5901cbb35487a2e1a89b49d3e73f7b432bb4cd242\": container with ID starting with 500ebcc6450a5aba18847ea5901cbb35487a2e1a89b49d3e73f7b432bb4cd242 not found: ID does not exist" containerID="500ebcc6450a5aba18847ea5901cbb35487a2e1a89b49d3e73f7b432bb4cd242" Nov 02 18:16:04.186702 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.186590 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"500ebcc6450a5aba18847ea5901cbb35487a2e1a89b49d3e73f7b432bb4cd242"} err="failed to get container status \"500ebcc6450a5aba18847ea5901cbb35487a2e1a89b49d3e73f7b432bb4cd242\": rpc error: code = NotFound desc = could not find container \"500ebcc6450a5aba18847ea5901cbb35487a2e1a89b49d3e73f7b432bb4cd242\": container with ID starting with 500ebcc6450a5aba18847ea5901cbb35487a2e1a89b49d3e73f7b432bb4cd242 not found: ID does not exist" Nov 02 18:16:04.200472 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.200443 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5"] Nov 02 18:16:04.204975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.204938 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-a208ef/e2e-usage-sim-7c2607-5f95848c6b-79wt5"] Nov 02 18:16:04.323610 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.323527 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b3ed03b-5899-4984-b221-12059d4ea928" path="/var/lib/kubelet/pods/7b3ed03b-5899-4984-b221-12059d4ea928/volumes" Nov 02 18:16:04.457348 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.457311 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:04.590030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.581852 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm"] Nov 02 18:16:04.604794 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.604759 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:04.607046 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.607020 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd"] Nov 02 18:16:04.609617 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.609239 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7b3ed03b-5899-4984-b221-12059d4ea928" containerName="sim" Nov 02 18:16:04.609807 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.609787 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b3ed03b-5899-4984-b221-12059d4ea928" containerName="sim" Nov 02 18:16:04.610501 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.610456 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b3ed03b-5899-4984-b221-12059d4ea928" containerName="sim" Nov 02 18:16:04.642593 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.642563 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" Nov 02 18:16:04.685991 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.676110 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98q7n\" (UniqueName: \"kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-kube-api-access-98q7n\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd\" (UID: \"183637b2-b2f8-458e-a20a-326c7d5c6333\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" Nov 02 18:16:04.685991 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.676417 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd\" (UID: \"183637b2-b2f8-458e-a20a-326c7d5c6333\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" Nov 02 18:16:04.685991 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.676543 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd\" (UID: \"183637b2-b2f8-458e-a20a-326c7d5c6333\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" Nov 02 18:16:04.685991 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.676611 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd\" (UID: \"183637b2-b2f8-458e-a20a-326c7d5c6333\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" Nov 02 18:16:04.704976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.704630 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd"] Nov 02 18:16:04.778569 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.778528 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd\" (UID: \"183637b2-b2f8-458e-a20a-326c7d5c6333\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" Nov 02 18:16:04.778757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.778683 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd\" (UID: \"183637b2-b2f8-458e-a20a-326c7d5c6333\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" Nov 02 18:16:04.778804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.778756 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd\" (UID: \"183637b2-b2f8-458e-a20a-326c7d5c6333\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" Nov 02 18:16:04.778868 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.778826 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98q7n\" (UniqueName: \"kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-kube-api-access-98q7n\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd\" (UID: \"183637b2-b2f8-458e-a20a-326c7d5c6333\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" Nov 02 18:16:04.779642 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.779613 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd\" (UID: \"183637b2-b2f8-458e-a20a-326c7d5c6333\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" Nov 02 18:16:04.780426 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.780399 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd\" (UID: \"183637b2-b2f8-458e-a20a-326c7d5c6333\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" Nov 02 18:16:04.781716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.781691 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd\" (UID: \"183637b2-b2f8-458e-a20a-326c7d5c6333\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" Nov 02 18:16:04.789610 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:04.789587 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98q7n\" (UniqueName: \"kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-kube-api-access-98q7n\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd\" (UID: \"183637b2-b2f8-458e-a20a-326c7d5c6333\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" Nov 02 18:16:05.063345 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:05.063304 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" Nov 02 18:16:05.446409 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:05.446364 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv"] Nov 02 18:16:05.579057 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:05.579025 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2db162/agc-cred-12ef34d6c6c7: secret "agc-cred-12ef34d6c6c7" not found Nov 02 18:16:05.579294 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:05.579277 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm: secret "agc-cred-12ef34d6c6c7" not found Nov 02 18:16:05.579535 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:05.579516 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-provider-credentials podName:31153884-0178-422c-b089-b06f4e6b149b nodeName:}" failed. No retries permitted until 2026-11-02 18:16:06.079494794 +0000 UTC m=+2537.868000314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" (UID: "31153884-0178-422c-b089-b06f4e6b149b") : secret "agc-cred-12ef34d6c6c7" not found Nov 02 18:16:05.587652 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:05.587215 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-36a991/agc-cred-811d350f868d: secret "agc-cred-811d350f868d" not found Nov 02 18:16:05.587652 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:05.587240 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv: secret "agc-cred-811d350f868d" not found Nov 02 18:16:05.587652 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:05.587309 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-provider-credentials podName:86914c05-ecb6-4c59-b4f9-ed49aa828caf nodeName:}" failed. No retries permitted until 2026-11-02 18:16:06.087290081 +0000 UTC m=+2537.875795604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" (UID: "86914c05-ecb6-4c59-b4f9-ed49aa828caf") : secret "agc-cred-811d350f868d" not found Nov 02 18:16:05.755335 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:05.752250 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:05.790017 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:05.789983 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd"] Nov 02 18:16:05.793160 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:16:05.793132 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183637b2_b2f8_458e_a20a_326c7d5c6333.slice/crio-ce02e394184d862cceba4bb444ad489cbf82c91ee45125d80ecdd87c097da7db WatchSource:0}: Error finding container ce02e394184d862cceba4bb444ad489cbf82c91ee45125d80ecdd87c097da7db: Status 404 returned error can't find the container with id ce02e394184d862cceba4bb444ad489cbf82c91ee45125d80ecdd87c097da7db Nov 02 18:16:06.004280 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:06.004248 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:06.088638 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:06.088554 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2db162/agc-cred-12ef34d6c6c7: secret "agc-cred-12ef34d6c6c7" not found Nov 02 18:16:06.088638 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:06.088583 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2db162/agc-cred-ff45ed7b3970: secret "agc-cred-ff45ed7b3970" not found Nov 02 18:16:06.088638 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:06.088595 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm: [secret "agc-cred-12ef34d6c6c7" not found, secret "agc-cred-ff45ed7b3970" not found] Nov 02 18:16:06.089165 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:06.088647 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-provider-credentials podName:31153884-0178-422c-b089-b06f4e6b149b nodeName:}" failed. No retries permitted until 2026-11-02 18:16:07.088633359 +0000 UTC m=+2538.877138865 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" (UID: "31153884-0178-422c-b089-b06f4e6b149b") : [secret "agc-cred-12ef34d6c6c7" not found, secret "agc-cred-ff45ed7b3970" not found] Nov 02 18:16:06.089165 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:06.088678 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-36a991/agc-cred-0ebc5b098a14: secret "agc-cred-0ebc5b098a14" not found Nov 02 18:16:06.089165 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:06.088706 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-36a991/agc-cred-811d350f868d: secret "agc-cred-811d350f868d" not found Nov 02 18:16:06.089165 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:06.088718 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv: [secret "agc-cred-0ebc5b098a14" not found, secret "agc-cred-811d350f868d" not found] Nov 02 18:16:06.089165 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:06.088785 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-provider-credentials podName:86914c05-ecb6-4c59-b4f9-ed49aa828caf nodeName:}" failed. No retries permitted until 2026-11-02 18:16:07.088766555 +0000 UTC m=+2538.877272080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" (UID: "86914c05-ecb6-4c59-b4f9-ed49aa828caf") : [secret "agc-cred-0ebc5b098a14" not found, secret "agc-cred-811d350f868d" not found] Nov 02 18:16:06.203931 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:06.203572 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:06.385065 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:06.384978 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:06.604072 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:06.604026 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" event={"ID":"183637b2-b2f8-458e-a20a-326c7d5c6333","Type":"ContainerStarted","Data":"0a3a9345e93fb71a77534fe215a5c9408c6b260ea12ac3fb6367dc9758b58477"} Nov 02 18:16:06.604335 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:06.604094 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" event={"ID":"183637b2-b2f8-458e-a20a-326c7d5c6333","Type":"ContainerStarted","Data":"ce02e394184d862cceba4bb444ad489cbf82c91ee45125d80ecdd87c097da7db"} Nov 02 18:16:06.604335 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:06.604125 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" Nov 02 18:16:06.605079 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:06.605046 5549 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" secret="" err="secret \"payload-processing-external-model-e2e--59b59218-dockercfg-lp4xp\" not found" Nov 02 18:16:06.619174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:06.619129 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:06.635485 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:06.635382 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" podStartSLOduration=2.635366275 podStartE2EDuration="2.635366275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:06.623320574 +0000 UTC m=+2538.411826104" watchObservedRunningTime="2026-11-02 18:16:06.635366275 +0000 UTC m=+2538.423871802" Nov 02 18:16:06.690154 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:06.690123 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-plugins-e2e-extmod-a-36a991: configmap "payload-processing-external-model-plugins-e2e-extmod-a-36a991" not found Nov 02 18:16:06.690387 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:06.690199 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-plugins-config-volume podName:86914c05-ecb6-4c59-b4f9-ed49aa828caf nodeName:}" failed. No retries permitted until 2026-11-02 18:16:07.190181747 +0000 UTC m=+2538.978687267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" (UID: "86914c05-ecb6-4c59-b4f9-ed49aa828caf") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-36a991" not found Nov 02 18:16:06.690569 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:06.690544 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2db162/agc-cred-ff45ed7b3970: secret "agc-cred-ff45ed7b3970" not found Nov 02 18:16:06.690667 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:06.690571 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd: secret "agc-cred-ff45ed7b3970" not found Nov 02 18:16:06.690730 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:06.690701 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-provider-credentials podName:183637b2-b2f8-458e-a20a-326c7d5c6333 nodeName:}" failed. No retries permitted until 2026-11-02 18:16:07.190681751 +0000 UTC m=+2538.979187277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" (UID: "183637b2-b2f8-458e-a20a-326c7d5c6333") : secret "agc-cred-ff45ed7b3970" not found Nov 02 18:16:06.905302 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:06.905216 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd"] Nov 02 18:16:06.995931 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:06.995886 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:07.203806 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:07.195640 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-plugins-e2e-extmod-a-36a991: configmap "payload-processing-external-model-plugins-e2e-extmod-a-36a991" not found Nov 02 18:16:07.203806 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:07.195743 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-plugins-config-volume podName:86914c05-ecb6-4c59-b4f9-ed49aa828caf nodeName:}" failed. No retries permitted until 2026-11-02 18:16:08.195719747 +0000 UTC m=+2539.984225276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" (UID: "86914c05-ecb6-4c59-b4f9-ed49aa828caf") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-36a991" not found Nov 02 18:16:07.203806 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:07.196311 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2db162/agc-cred-ff45ed7b3970: secret "agc-cred-ff45ed7b3970" not found Nov 02 18:16:07.203806 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:07.196335 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd: secret "agc-cred-ff45ed7b3970" not found Nov 02 18:16:07.203806 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:07.196398 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-provider-credentials podName:183637b2-b2f8-458e-a20a-326c7d5c6333 nodeName:}" failed. No retries permitted until 2026-11-02 18:16:08.196371536 +0000 UTC m=+2539.984877047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" (UID: "183637b2-b2f8-458e-a20a-326c7d5c6333") : secret "agc-cred-ff45ed7b3970" not found Nov 02 18:16:07.203806 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:07.196470 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2db162/agc-cred-12ef34d6c6c7: secret "agc-cred-12ef34d6c6c7" not found Nov 02 18:16:07.203806 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:07.196486 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2db162/agc-cred-ff45ed7b3970: secret "agc-cred-ff45ed7b3970" not found Nov 02 18:16:07.203806 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:07.196504 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm: [secret "agc-cred-12ef34d6c6c7" not found, secret "agc-cred-ff45ed7b3970" not found] Nov 02 18:16:07.203806 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:07.196546 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-provider-credentials podName:31153884-0178-422c-b089-b06f4e6b149b nodeName:}" failed. No retries permitted until 2026-11-02 18:16:09.196527798 +0000 UTC m=+2540.985033305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" (UID: "31153884-0178-422c-b089-b06f4e6b149b") : [secret "agc-cred-12ef34d6c6c7" not found, secret "agc-cred-ff45ed7b3970" not found] Nov 02 18:16:07.203806 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:07.196612 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-36a991/agc-cred-0ebc5b098a14: secret "agc-cred-0ebc5b098a14" not found Nov 02 18:16:07.203806 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:07.196624 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-36a991/agc-cred-811d350f868d: secret "agc-cred-811d350f868d" not found Nov 02 18:16:07.203806 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:07.196632 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv: [secret "agc-cred-0ebc5b098a14" not found, secret "agc-cred-811d350f868d" not found] Nov 02 18:16:07.203806 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:07.196673 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-provider-credentials podName:86914c05-ecb6-4c59-b4f9-ed49aa828caf nodeName:}" failed. No retries permitted until 2026-11-02 18:16:09.196661485 +0000 UTC m=+2540.985166993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" (UID: "86914c05-ecb6-4c59-b4f9-ed49aa828caf") : [secret "agc-cred-0ebc5b098a14" not found, secret "agc-cred-811d350f868d" not found] Nov 02 18:16:07.613916 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:07.613807 5549 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" secret="" err="secret \"payload-processing-external-model-e2e--96b2e561-dockercfg-v9qrb\" not found" Nov 02 18:16:07.705917 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:07.705885 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-plugins-e2e-extmod-b-2db162: configmap "payload-processing-external-model-plugins-e2e-extmod-b-2db162" not found Nov 02 18:16:07.706133 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:07.705967 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-plugins-config-volume podName:183637b2-b2f8-458e-a20a-326c7d5c6333 nodeName:}" failed. No retries permitted until 2026-11-02 18:16:08.205939651 +0000 UTC m=+2539.994445158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" (UID: "183637b2-b2f8-458e-a20a-326c7d5c6333") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-2db162" not found Nov 02 18:16:07.825492 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:07.825425 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:08.232230 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:08.232182 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-plugins-e2e-extmod-a-36a991: configmap "payload-processing-external-model-plugins-e2e-extmod-a-36a991" not found Nov 02 18:16:08.232672 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:08.232318 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2db162/agc-cred-ff45ed7b3970: secret "agc-cred-ff45ed7b3970" not found Nov 02 18:16:08.232672 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:08.232346 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd: secret "agc-cred-ff45ed7b3970" not found Nov 02 18:16:08.232672 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:08.232358 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-plugins-e2e-extmod-b-2db162: configmap "payload-processing-external-model-plugins-e2e-extmod-b-2db162" not found Nov 02 18:16:08.232672 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:08.232400 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-plugins-config-volume podName:86914c05-ecb6-4c59-b4f9-ed49aa828caf nodeName:}" failed. No retries permitted until 2026-11-02 18:16:10.232379714 +0000 UTC m=+2542.020885235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" (UID: "86914c05-ecb6-4c59-b4f9-ed49aa828caf") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-36a991" not found Nov 02 18:16:08.232672 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:08.232427 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-provider-credentials podName:183637b2-b2f8-458e-a20a-326c7d5c6333 nodeName:}" failed. No retries permitted until 2026-11-02 18:16:10.232419191 +0000 UTC m=+2542.020924701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" (UID: "183637b2-b2f8-458e-a20a-326c7d5c6333") : secret "agc-cred-ff45ed7b3970" not found Nov 02 18:16:08.232672 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:08.232440 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-plugins-config-volume podName:183637b2-b2f8-458e-a20a-326c7d5c6333 nodeName:}" failed. No retries permitted until 2026-11-02 18:16:09.232435087 +0000 UTC m=+2541.020940598 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" (UID: "183637b2-b2f8-458e-a20a-326c7d5c6333") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-2db162" not found Nov 02 18:16:08.539031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:08.538636 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:09.296196 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:09.294437 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2db162/agc-cred-12ef34d6c6c7: secret "agc-cred-12ef34d6c6c7" not found Nov 02 18:16:09.296196 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:09.294470 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2db162/agc-cred-ff45ed7b3970: secret "agc-cred-ff45ed7b3970" not found Nov 02 18:16:09.296196 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:09.294483 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm: [secret "agc-cred-12ef34d6c6c7" not found, secret "agc-cred-ff45ed7b3970" not found] Nov 02 18:16:09.296196 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:09.294545 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-provider-credentials podName:31153884-0178-422c-b089-b06f4e6b149b nodeName:}" failed. No retries permitted until 2026-11-02 18:16:13.294525894 +0000 UTC m=+2545.083031416 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" (UID: "31153884-0178-422c-b089-b06f4e6b149b") : [secret "agc-cred-12ef34d6c6c7" not found, secret "agc-cred-ff45ed7b3970" not found] Nov 02 18:16:09.297267 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:09.296425 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-36a991/agc-cred-0ebc5b098a14: secret "agc-cred-0ebc5b098a14" not found Nov 02 18:16:09.297267 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:09.296452 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-36a991/agc-cred-811d350f868d: secret "agc-cred-811d350f868d" not found Nov 02 18:16:09.297267 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:09.296465 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv: [secret "agc-cred-0ebc5b098a14" not found, secret "agc-cred-811d350f868d" not found] Nov 02 18:16:09.297267 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:09.296534 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-provider-credentials podName:86914c05-ecb6-4c59-b4f9-ed49aa828caf nodeName:}" failed. No retries permitted until 2026-11-02 18:16:13.296514051 +0000 UTC m=+2545.085019557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" (UID: "86914c05-ecb6-4c59-b4f9-ed49aa828caf") : [secret "agc-cred-0ebc5b098a14" not found, secret "agc-cred-811d350f868d" not found] Nov 02 18:16:09.298048 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:09.294945 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-plugins-e2e-extmod-b-2db162: configmap "payload-processing-external-model-plugins-e2e-extmod-b-2db162" not found Nov 02 18:16:09.298140 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:09.298119 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-plugins-config-volume podName:183637b2-b2f8-458e-a20a-326c7d5c6333 nodeName:}" failed. No retries permitted until 2026-11-02 18:16:11.298096336 +0000 UTC m=+2543.086601856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" (UID: "183637b2-b2f8-458e-a20a-326c7d5c6333") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-2db162" not found Nov 02 18:16:09.298808 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:09.298461 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:10.116967 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:10.113684 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:10.327490 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:10.327462 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-plugins-e2e-extmod-a-36a991: configmap "payload-processing-external-model-plugins-e2e-extmod-a-36a991" not found Nov 02 18:16:10.327864 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:10.327520 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-plugins-config-volume podName:86914c05-ecb6-4c59-b4f9-ed49aa828caf nodeName:}" failed. No retries permitted until 2026-11-02 18:16:14.327506792 +0000 UTC m=+2546.116012298 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" (UID: "86914c05-ecb6-4c59-b4f9-ed49aa828caf") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-36a991" not found Nov 02 18:16:10.327864 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:10.327580 5549 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2db162/agc-cred-ff45ed7b3970: secret "agc-cred-ff45ed7b3970" not found Nov 02 18:16:10.327864 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:10.327602 5549 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd: secret "agc-cred-ff45ed7b3970" not found Nov 02 18:16:10.327864 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:10.327648 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-provider-credentials podName:183637b2-b2f8-458e-a20a-326c7d5c6333 nodeName:}" failed. No retries permitted until 2026-11-02 18:16:14.327636598 +0000 UTC m=+2546.116142134 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" (UID: "183637b2-b2f8-458e-a20a-326c7d5c6333") : secret "agc-cred-ff45ed7b3970" not found Nov 02 18:16:10.780987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:10.776293 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:11.347555 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:11.347518 5549 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-plugins-e2e-extmod-b-2db162: configmap "payload-processing-external-model-plugins-e2e-extmod-b-2db162" not found Nov 02 18:16:11.347989 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:11.347610 5549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-plugins-config-volume podName:183637b2-b2f8-458e-a20a-326c7d5c6333 nodeName:}" failed. No retries permitted until 2026-11-02 18:16:15.347590638 +0000 UTC m=+2547.136096166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" (UID: "183637b2-b2f8-458e-a20a-326c7d5c6333") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-2db162" not found Nov 02 18:16:11.475213 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:11.475171 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:11.603768 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:11.603682 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" podUID="31153884-0178-422c-b089-b06f4e6b149b" containerName="payload-processing" containerID="cri-o://16ed40459457eec5967fc1db8ab4db764ee63e9afed892372a4c4dc1e4d00738" gracePeriod=25 Nov 02 18:16:11.904619 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:11.904588 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" Nov 02 18:16:11.981527 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:11.981491 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31153884-0178-422c-b089-b06f4e6b149b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31153884-0178-422c-b089-b06f4e6b149b-routing-overlay-volume\") pod \"31153884-0178-422c-b089-b06f4e6b149b\" (UID: \"31153884-0178-422c-b089-b06f4e6b149b\") " Nov 02 18:16:11.981695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:11.981586 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-kube-api-access-6sshx\" (UniqueName: \"kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-kube-api-access-6sshx\") pod \"31153884-0178-422c-b089-b06f4e6b149b\" (UID: \"31153884-0178-422c-b089-b06f4e6b149b\") " Nov 02 18:16:11.981822 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:11.981804 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31153884-0178-422c-b089-b06f4e6b149b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31153884-0178-422c-b089-b06f4e6b149b-plugins-config-volume\") pod \"31153884-0178-422c-b089-b06f4e6b149b\" (UID: \"31153884-0178-422c-b089-b06f4e6b149b\") " Nov 02 18:16:11.981883 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:11.981867 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-provider-credentials\") pod \"31153884-0178-422c-b089-b06f4e6b149b\" (UID: \"31153884-0178-422c-b089-b06f4e6b149b\") " Nov 02 18:16:11.981933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:11.981892 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31153884-0178-422c-b089-b06f4e6b149b-routing-overlay-volume" pod "31153884-0178-422c-b089-b06f4e6b149b" (UID: "31153884-0178-422c-b089-b06f4e6b149b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:16:11.982228 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:11.982197 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31153884-0178-422c-b089-b06f4e6b149b-plugins-config-volume" pod "31153884-0178-422c-b089-b06f4e6b149b" (UID: "31153884-0178-422c-b089-b06f4e6b149b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:16:11.982698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:11.982675 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31153884-0178-422c-b089-b06f4e6b149b-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:11.982765 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:11.982702 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31153884-0178-422c-b089-b06f4e6b149b-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:11.983817 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:11.983792 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-kube-api-access-6sshx" pod "31153884-0178-422c-b089-b06f4e6b149b" (UID: "31153884-0178-422c-b089-b06f4e6b149b"). InnerVolumeSpecName "kube-api-access-6sshx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:11.983912 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:11.983874 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-provider-credentials" pod "31153884-0178-422c-b089-b06f4e6b149b" (UID: "31153884-0178-422c-b089-b06f4e6b149b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:12.111138 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:12.085936 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:12.111138 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:12.110655 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sshx\" (UniqueName: \"kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-kube-api-access-6sshx\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:12.111138 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:12.110693 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31153884-0178-422c-b089-b06f4e6b149b-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:12.613838 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:12.613784 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" podUID="86914c05-ecb6-4c59-b4f9-ed49aa828caf" containerName="payload-processing" containerID="cri-o://892a00c0d92f82a423dbcf62f38f68dd04a2960f8467e2767f52f26c96448de2" gracePeriod=25 Nov 02 18:16:12.672606 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:12.672577 5549 generic.go:356] "Generic (PLEG): container finished" podID="31153884-0178-422c-b089-b06f4e6b149b" containerID="16ed40459457eec5967fc1db8ab4db764ee63e9afed892372a4c4dc1e4d00738" exitCode=0 Nov 02 18:16:12.672772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:12.672621 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" Nov 02 18:16:12.672772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:12.672650 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" event={"ID":"31153884-0178-422c-b089-b06f4e6b149b","Type":"ContainerDied","Data":"16ed40459457eec5967fc1db8ab4db764ee63e9afed892372a4c4dc1e4d00738"} Nov 02 18:16:12.672772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:12.672701 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm" event={"ID":"31153884-0178-422c-b089-b06f4e6b149b","Type":"ContainerDied","Data":"815bceeace7a844a0aac5928e3f579216e8f0949485705882d942ac65b16fb91"} Nov 02 18:16:12.672772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:12.672730 5549 scope.go:122] "RemoveContainer" containerID="16ed40459457eec5967fc1db8ab4db764ee63e9afed892372a4c4dc1e4d00738" Nov 02 18:16:12.720319 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:12.720288 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm"] Nov 02 18:16:12.729054 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:12.729029 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-7fd5mcjqm"] Nov 02 18:16:12.729660 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:12.729640 5549 scope.go:122] "RemoveContainer" containerID="16ed40459457eec5967fc1db8ab4db764ee63e9afed892372a4c4dc1e4d00738" Nov 02 18:16:12.729919 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:12.729902 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ed40459457eec5967fc1db8ab4db764ee63e9afed892372a4c4dc1e4d00738\": container with ID starting with 16ed40459457eec5967fc1db8ab4db764ee63e9afed892372a4c4dc1e4d00738 not found: ID does not exist" containerID="16ed40459457eec5967fc1db8ab4db764ee63e9afed892372a4c4dc1e4d00738" Nov 02 18:16:12.730011 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:12.729925 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ed40459457eec5967fc1db8ab4db764ee63e9afed892372a4c4dc1e4d00738"} err="failed to get container status \"16ed40459457eec5967fc1db8ab4db764ee63e9afed892372a4c4dc1e4d00738\": rpc error: code = NotFound desc = could not find container \"16ed40459457eec5967fc1db8ab4db764ee63e9afed892372a4c4dc1e4d00738\": container with ID starting with 16ed40459457eec5967fc1db8ab4db764ee63e9afed892372a4c4dc1e4d00738 not found: ID does not exist" Nov 02 18:16:12.850380 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:12.843549 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:12.998876 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:12.998849 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" Nov 02 18:16:13.030685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.030652 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-routing-overlay-volume\") pod \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\" (UID: \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\") " Nov 02 18:16:13.030882 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.030822 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-plugins-config-volume\") pod \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\" (UID: \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\") " Nov 02 18:16:13.030948 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.030889 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-kube-api-access-n7tx9\" (UniqueName: \"kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-kube-api-access-n7tx9\") pod \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\" (UID: \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\") " Nov 02 18:16:13.031062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.030997 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-routing-overlay-volume" pod "86914c05-ecb6-4c59-b4f9-ed49aa828caf" (UID: "86914c05-ecb6-4c59-b4f9-ed49aa828caf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:16:13.031190 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.031170 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-provider-credentials\") pod \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\" (UID: \"86914c05-ecb6-4c59-b4f9-ed49aa828caf\") " Nov 02 18:16:13.031337 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.031312 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-plugins-config-volume" pod "86914c05-ecb6-4c59-b4f9-ed49aa828caf" (UID: "86914c05-ecb6-4c59-b4f9-ed49aa828caf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:16:13.031811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.031788 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:13.031811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.031810 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86914c05-ecb6-4c59-b4f9-ed49aa828caf-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:13.033159 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.033130 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-provider-credentials" pod "86914c05-ecb6-4c59-b4f9-ed49aa828caf" (UID: "86914c05-ecb6-4c59-b4f9-ed49aa828caf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:13.033246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.033205 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-kube-api-access-n7tx9" pod "86914c05-ecb6-4c59-b4f9-ed49aa828caf" (UID: "86914c05-ecb6-4c59-b4f9-ed49aa828caf"). InnerVolumeSpecName "kube-api-access-n7tx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:13.134032 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.133932 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7tx9\" (UniqueName: \"kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-kube-api-access-n7tx9\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:13.134032 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.133980 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86914c05-ecb6-4c59-b4f9-ed49aa828caf-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:13.415345 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.414444 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:13.586147 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.586109 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:13.640229 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.633985 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" podUID="183637b2-b2f8-458e-a20a-326c7d5c6333" containerName="payload-processing" containerID="cri-o://0a3a9345e93fb71a77534fe215a5c9408c6b260ea12ac3fb6367dc9758b58477" gracePeriod=25 Nov 02 18:16:13.695920 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.695885 5549 generic.go:356] "Generic (PLEG): container finished" podID="86914c05-ecb6-4c59-b4f9-ed49aa828caf" containerID="892a00c0d92f82a423dbcf62f38f68dd04a2960f8467e2767f52f26c96448de2" exitCode=0 Nov 02 18:16:13.696125 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.696028 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" event={"ID":"86914c05-ecb6-4c59-b4f9-ed49aa828caf","Type":"ContainerDied","Data":"892a00c0d92f82a423dbcf62f38f68dd04a2960f8467e2767f52f26c96448de2"} Nov 02 18:16:13.696224 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.696193 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" Nov 02 18:16:13.696749 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.696702 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv" event={"ID":"86914c05-ecb6-4c59-b4f9-ed49aa828caf","Type":"ContainerDied","Data":"e1c2c9b2844f8c8fa0b4ca95716b095775a2e1d7aadd7f94ca4eff82cdb09bbc"} Nov 02 18:16:13.696891 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.696787 5549 scope.go:122] "RemoveContainer" containerID="892a00c0d92f82a423dbcf62f38f68dd04a2960f8467e2767f52f26c96448de2" Nov 02 18:16:13.744780 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.744748 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:13.776030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.775999 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv"] Nov 02 18:16:13.781047 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.781021 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-78bdjzbcv"] Nov 02 18:16:13.791592 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.791570 5549 scope.go:122] "RemoveContainer" containerID="892a00c0d92f82a423dbcf62f38f68dd04a2960f8467e2767f52f26c96448de2" Nov 02 18:16:13.791896 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:13.791876 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"892a00c0d92f82a423dbcf62f38f68dd04a2960f8467e2767f52f26c96448de2\": container with ID starting with 892a00c0d92f82a423dbcf62f38f68dd04a2960f8467e2767f52f26c96448de2 not found: ID does not exist" containerID="892a00c0d92f82a423dbcf62f38f68dd04a2960f8467e2767f52f26c96448de2" Nov 02 18:16:13.791943 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.791906 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"892a00c0d92f82a423dbcf62f38f68dd04a2960f8467e2767f52f26c96448de2"} err="failed to get container status \"892a00c0d92f82a423dbcf62f38f68dd04a2960f8467e2767f52f26c96448de2\": rpc error: code = NotFound desc = could not find container \"892a00c0d92f82a423dbcf62f38f68dd04a2960f8467e2767f52f26c96448de2\": container with ID starting with 892a00c0d92f82a423dbcf62f38f68dd04a2960f8467e2767f52f26c96448de2 not found: ID does not exist" Nov 02 18:16:13.951662 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.951627 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:13.983193 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:13.983165 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" Nov 02 18:16:14.063691 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.063653 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-provider-credentials\" (UniqueName: \"kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-provider-credentials\") pod \"183637b2-b2f8-458e-a20a-326c7d5c6333\" (UID: \"183637b2-b2f8-458e-a20a-326c7d5c6333\") " Nov 02 18:16:14.063871 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.063766 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-kube-api-access-98q7n\" (UniqueName: \"kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-kube-api-access-98q7n\") pod \"183637b2-b2f8-458e-a20a-326c7d5c6333\" (UID: \"183637b2-b2f8-458e-a20a-326c7d5c6333\") " Nov 02 18:16:14.063871 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.063836 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-plugins-config-volume\") pod \"183637b2-b2f8-458e-a20a-326c7d5c6333\" (UID: \"183637b2-b2f8-458e-a20a-326c7d5c6333\") " Nov 02 18:16:14.064075 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.063898 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-routing-overlay-volume\") pod \"183637b2-b2f8-458e-a20a-326c7d5c6333\" (UID: \"183637b2-b2f8-458e-a20a-326c7d5c6333\") " Nov 02 18:16:14.064353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.064320 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-plugins-config-volume" pod "183637b2-b2f8-458e-a20a-326c7d5c6333" (UID: "183637b2-b2f8-458e-a20a-326c7d5c6333"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:16:14.064452 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.064347 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-routing-overlay-volume" pod "183637b2-b2f8-458e-a20a-326c7d5c6333" (UID: "183637b2-b2f8-458e-a20a-326c7d5c6333"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:16:14.065056 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.065029 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:14.065168 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.065059 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/183637b2-b2f8-458e-a20a-326c7d5c6333-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:14.065974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.065938 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-provider-credentials" pod "183637b2-b2f8-458e-a20a-326c7d5c6333" (UID: "183637b2-b2f8-458e-a20a-326c7d5c6333"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:14.066262 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.066242 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-kube-api-access-98q7n" pod "183637b2-b2f8-458e-a20a-326c7d5c6333" (UID: "183637b2-b2f8-458e-a20a-326c7d5c6333"). InnerVolumeSpecName "kube-api-access-98q7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:14.166792 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.166757 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98q7n\" (UniqueName: \"kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-kube-api-access-98q7n\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:14.166792 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.166788 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/183637b2-b2f8-458e-a20a-326c7d5c6333-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:14.222459 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.222422 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr"] Nov 02 18:16:14.224244 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.223918 5549 cpu_manager.go:414] "containerMap: removing container" podUID="183637b2-b2f8-458e-a20a-326c7d5c6333" containerName="payload-processing" Nov 02 18:16:14.224244 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.223944 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="183637b2-b2f8-458e-a20a-326c7d5c6333" containerName="payload-processing" Nov 02 18:16:14.224244 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.224039 5549 cpu_manager.go:414] "containerMap: removing container" podUID="86914c05-ecb6-4c59-b4f9-ed49aa828caf" containerName="payload-processing" Nov 02 18:16:14.224244 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.224047 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86914c05-ecb6-4c59-b4f9-ed49aa828caf" containerName="payload-processing" Nov 02 18:16:14.224244 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.224059 5549 cpu_manager.go:414] "containerMap: removing container" podUID="31153884-0178-422c-b089-b06f4e6b149b" containerName="payload-processing" Nov 02 18:16:14.224244 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.224066 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31153884-0178-422c-b089-b06f4e6b149b" containerName="payload-processing" Nov 02 18:16:14.224662 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.224278 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="183637b2-b2f8-458e-a20a-326c7d5c6333" containerName="payload-processing" Nov 02 18:16:14.224662 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.224309 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="86914c05-ecb6-4c59-b4f9-ed49aa828caf" containerName="payload-processing" Nov 02 18:16:14.224662 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.224319 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="31153884-0178-422c-b089-b06f4e6b149b" containerName="payload-processing" Nov 02 18:16:14.229598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.229089 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.234509 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.234482 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr"] Nov 02 18:16:14.241071 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.240255 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-50b3a6-openshift-default-dockercfg-4xhh4\"" Nov 02 18:16:14.309541 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.309508 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:14.324182 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.324151 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31153884-0178-422c-b089-b06f4e6b149b" path="/var/lib/kubelet/pods/31153884-0178-422c-b089-b06f4e6b149b/volumes" Nov 02 18:16:14.324795 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.324778 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86914c05-ecb6-4c59-b4f9-ed49aa828caf" path="/var/lib/kubelet/pods/86914c05-ecb6-4c59-b4f9-ed49aa828caf/volumes" Nov 02 18:16:14.402976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.373170 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3110f2f-be09-4cbb-9687-e642a0268441-istio-token\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.402976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.373356 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-credential-socket\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.402976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.373395 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-istio-envoy\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.402976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.373422 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3110f2f-be09-4cbb-9687-e642a0268441-istiod-ca-cert\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.402976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.373522 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-workload-socket\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.402976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.373559 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-istio-data\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.402976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.373750 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3110f2f-be09-4cbb-9687-e642a0268441-istio-podinfo\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.402976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.373783 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-workload-certs\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.402976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.373853 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwqlz\" (UniqueName: \"kubernetes.io/projected/b3110f2f-be09-4cbb-9687-e642a0268441-kube-api-access-pwqlz\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.462982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.462834 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:14.483438 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.483325 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-credential-socket\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.484404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.484376 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-istio-envoy\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.484594 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.484579 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3110f2f-be09-4cbb-9687-e642a0268441-istiod-ca-cert\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.484816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.484798 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-workload-socket\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.484972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.484936 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-istio-data\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.485291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.485271 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3110f2f-be09-4cbb-9687-e642a0268441-istio-podinfo\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.485449 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.485426 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-workload-certs\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.485603 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.485561 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-workload-socket\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.485693 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.485595 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwqlz\" (UniqueName: \"kubernetes.io/projected/b3110f2f-be09-4cbb-9687-e642a0268441-kube-api-access-pwqlz\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.485869 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.485850 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3110f2f-be09-4cbb-9687-e642a0268441-istio-token\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.499219 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.486260 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3110f2f-be09-4cbb-9687-e642a0268441-istiod-ca-cert\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.499435 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.484268 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-credential-socket\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.499534 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.486837 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-workload-certs\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.499636 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.487070 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-istio-data\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.517361 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.517326 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3110f2f-be09-4cbb-9687-e642a0268441-istio-podinfo\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.529518 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.529451 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-istio-envoy\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.549975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.532407 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3110f2f-be09-4cbb-9687-e642a0268441-istio-token\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.549975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.539114 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwqlz\" (UniqueName: \"kubernetes.io/projected/b3110f2f-be09-4cbb-9687-e642a0268441-kube-api-access-pwqlz\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.552151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.551888 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:14.711160 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.711094 5549 generic.go:356] "Generic (PLEG): container finished" podID="183637b2-b2f8-458e-a20a-326c7d5c6333" containerID="0a3a9345e93fb71a77534fe215a5c9408c6b260ea12ac3fb6367dc9758b58477" exitCode=0 Nov 02 18:16:14.711160 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.711143 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" Nov 02 18:16:14.711694 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.711173 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" event={"ID":"183637b2-b2f8-458e-a20a-326c7d5c6333","Type":"ContainerDied","Data":"0a3a9345e93fb71a77534fe215a5c9408c6b260ea12ac3fb6367dc9758b58477"} Nov 02 18:16:14.711694 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.711226 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd" event={"ID":"183637b2-b2f8-458e-a20a-326c7d5c6333","Type":"ContainerDied","Data":"ce02e394184d862cceba4bb444ad489cbf82c91ee45125d80ecdd87c097da7db"} Nov 02 18:16:14.711694 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.711259 5549 scope.go:122] "RemoveContainer" containerID="0a3a9345e93fb71a77534fe215a5c9408c6b260ea12ac3fb6367dc9758b58477" Nov 02 18:16:14.730117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.730084 5549 scope.go:122] "RemoveContainer" containerID="0a3a9345e93fb71a77534fe215a5c9408c6b260ea12ac3fb6367dc9758b58477" Nov 02 18:16:14.730497 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:14.730469 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3a9345e93fb71a77534fe215a5c9408c6b260ea12ac3fb6367dc9758b58477\": container with ID starting with 0a3a9345e93fb71a77534fe215a5c9408c6b260ea12ac3fb6367dc9758b58477 not found: ID does not exist" containerID="0a3a9345e93fb71a77534fe215a5c9408c6b260ea12ac3fb6367dc9758b58477" Nov 02 18:16:14.730640 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.730503 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3a9345e93fb71a77534fe215a5c9408c6b260ea12ac3fb6367dc9758b58477"} err="failed to get container status \"0a3a9345e93fb71a77534fe215a5c9408c6b260ea12ac3fb6367dc9758b58477\": rpc error: code = NotFound desc = could not find container \"0a3a9345e93fb71a77534fe215a5c9408c6b260ea12ac3fb6367dc9758b58477\": container with ID starting with 0a3a9345e93fb71a77534fe215a5c9408c6b260ea12ac3fb6367dc9758b58477 not found: ID does not exist" Nov 02 18:16:14.743787 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.743717 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd"] Nov 02 18:16:14.748728 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.748698 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-fc95kbppd"] Nov 02 18:16:14.906711 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:16:14.906677 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3110f2f_be09_4cbb_9687_e642a0268441.slice/crio-ec30ff6c4c2d19253fb6f459f93305a1f4416b8bfaa59d5370eefa837f81edaf WatchSource:0}: Error finding container ec30ff6c4c2d19253fb6f459f93305a1f4416b8bfaa59d5370eefa837f81edaf: Status 404 returned error can't find the container with id ec30ff6c4c2d19253fb6f459f93305a1f4416b8bfaa59d5370eefa837f81edaf Nov 02 18:16:14.909342 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.909306 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr"] Nov 02 18:16:14.910008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.909935 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:16:14.910065 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.910023 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:16:14.910130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:14.910064 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:16:15.053977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:15.050788 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:15.722836 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:15.722795 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" event={"ID":"b3110f2f-be09-4cbb-9687-e642a0268441","Type":"ContainerStarted","Data":"bc33d8602d13b5912358056b5ca160550230851af9a698f74bfa68fb19a90f6a"} Nov 02 18:16:15.723316 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:15.722852 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" event={"ID":"b3110f2f-be09-4cbb-9687-e642a0268441","Type":"ContainerStarted","Data":"ec30ff6c4c2d19253fb6f459f93305a1f4416b8bfaa59d5370eefa837f81edaf"} Nov 02 18:16:15.749136 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:15.749086 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" podStartSLOduration=1.7490715 podStartE2EDuration="1.7490715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:15.747110259 +0000 UTC m=+2547.535615786" watchObservedRunningTime="2026-11-02 18:16:15.7490715 +0000 UTC m=+2547.537577029" Nov 02 18:16:15.946996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:15.940845 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:16.323108 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:16.323072 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="183637b2-b2f8-458e-a20a-326c7d5c6333" path="/var/lib/kubelet/pods/183637b2-b2f8-458e-a20a-326c7d5c6333/volumes" Nov 02 18:16:16.555302 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:16.555267 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:16.561317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:16.561292 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:16.579910 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:16.579849 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:16:16.766857 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:16.766825 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:16.772246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:16.772219 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:16:17.151687 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:17.151646 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:17.591856 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:17.591818 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:16:18.248977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:18.248507 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:18.418975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:18.418285 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:18.884974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:18.882789 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:19.854423 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:19.854379 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:20.059979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:20.043551 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh"] Nov 02 18:16:20.370064 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:20.370025 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:20.454168 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:20.454122 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh" Nov 02 18:16:20.578012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:20.574512 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-50b3a6\"" Nov 02 18:16:20.578012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:20.574709 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-50b3a6-dockercfg-q75fl\"" Nov 02 18:16:20.604251 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:20.603250 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec0b8f3e-b65f-4709-95ad-8dab56153f36-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh\" (UID: \"ec0b8f3e-b65f-4709-95ad-8dab56153f36\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh" Nov 02 18:16:20.604251 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:20.603326 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px9sf\" (UniqueName: \"kubernetes.io/projected/ec0b8f3e-b65f-4709-95ad-8dab56153f36-kube-api-access-px9sf\") pod \"payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh\" (UID: \"ec0b8f3e-b65f-4709-95ad-8dab56153f36\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh" Nov 02 18:16:20.662151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:20.660057 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh"] Nov 02 18:16:20.710369 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:20.710325 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec0b8f3e-b65f-4709-95ad-8dab56153f36-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh\" (UID: \"ec0b8f3e-b65f-4709-95ad-8dab56153f36\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh" Nov 02 18:16:20.710649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:20.710625 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px9sf\" (UniqueName: \"kubernetes.io/projected/ec0b8f3e-b65f-4709-95ad-8dab56153f36-kube-api-access-px9sf\") pod \"payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh\" (UID: \"ec0b8f3e-b65f-4709-95ad-8dab56153f36\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh" Nov 02 18:16:20.712343 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:20.712316 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec0b8f3e-b65f-4709-95ad-8dab56153f36-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh\" (UID: \"ec0b8f3e-b65f-4709-95ad-8dab56153f36\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh" Nov 02 18:16:20.762059 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:20.753699 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px9sf\" (UniqueName: \"kubernetes.io/projected/ec0b8f3e-b65f-4709-95ad-8dab56153f36-kube-api-access-px9sf\") pod \"payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh\" (UID: \"ec0b8f3e-b65f-4709-95ad-8dab56153f36\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh" Nov 02 18:16:20.910038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:20.909223 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:20.981979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:20.974519 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh" Nov 02 18:16:21.962987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:21.962934 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh"] Nov 02 18:16:22.054555 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:16:22.054513 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0b8f3e_b65f_4709_95ad_8dab56153f36.slice/crio-629cf25d53aa16db78dc4e67154f00bf4a9e764304f8c2836b6fe807344242d3 WatchSource:0}: Error finding container 629cf25d53aa16db78dc4e67154f00bf4a9e764304f8c2836b6fe807344242d3: Status 404 returned error can't find the container with id 629cf25d53aa16db78dc4e67154f00bf4a9e764304f8c2836b6fe807344242d3 Nov 02 18:16:22.462940 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:22.461857 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:22.711656 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:22.711611 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh" event={"ID":"ec0b8f3e-b65f-4709-95ad-8dab56153f36","Type":"ContainerStarted","Data":"4a1e6f017e8edddac7d9b290849d4e3af58bb2e0bb239bc33294324a0e2fddfe"} Nov 02 18:16:22.711863 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:22.711691 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh" event={"ID":"ec0b8f3e-b65f-4709-95ad-8dab56153f36","Type":"ContainerStarted","Data":"629cf25d53aa16db78dc4e67154f00bf4a9e764304f8c2836b6fe807344242d3"} Nov 02 18:16:22.711965 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:22.711907 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh" Nov 02 18:16:24.692479 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:24.692442 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:25.919992 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:25.918336 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:26.882573 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:26.878934 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:27.161548 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:27.161451 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:27.574576 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:27.574544 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:27.744236 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:27.744193 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:28.022977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:28.017522 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:28.190978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:28.190669 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:28.819564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:28.819525 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:29.628737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:29.628649 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:30.488616 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:30.488401 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:31.274120 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:31.274082 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:32.022974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:32.020333 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:32.215282 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:32.215251 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:32.359002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:32.358908 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:32.810892 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:32.810843 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:33.479525 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:33.479487 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:33.637824 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:33.637770 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:33.757035 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:33.756729 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh" Nov 02 18:16:33.804535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:33.804497 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:33.832980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:33.814744 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh" podStartSLOduration=14.81472541 podStartE2EDuration="14.81472541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:22.733691592 +0000 UTC m=+2554.522197121" watchObservedRunningTime="2026-11-02 18:16:33.81472541 +0000 UTC m=+2565.603230939" Nov 02 18:16:34.602064 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:34.601159 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:35.574142 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:35.574103 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:36.451027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:36.450977 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:36.594894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:36.594854 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:36.730429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:36.730301 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:37.781581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:37.781534 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:38.565682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:38.565642 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:39.600161 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:39.600110 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:40.375537 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:40.375502 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:41.128584 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:41.128544 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:41.819164 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:41.818897 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:42.472618 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:42.472581 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:43.354257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:43.354211 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:44.154083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:44.151417 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:44.969852 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:44.969814 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:45.696541 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:45.696503 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:46.437503 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:46.437467 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:47.224291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:47.224242 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:47.591597 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:16:47.591510 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:16:48.014075 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:48.014034 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:48.150940 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:48.150898 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:48.286157 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:48.286069 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:48.979540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:48.979501 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:49.641726 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:49.641685 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:50.400688 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:50.400649 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:51.071975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:51.071732 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:51.853291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:51.852989 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:52.478505 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:52.478465 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:53.250049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:53.248934 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:54.033004 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:54.032787 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:54.679880 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:54.679840 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:55.431762 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:55.429402 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:56.078147 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:56.078111 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:56.813945 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:56.813913 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:56.953420 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:56.953363 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:57.099194 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:57.099107 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:57.815028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:57.805449 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:58.563205 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:58.563154 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:59.206795 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:59.206759 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:59.888415 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:16:59.888365 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:00.598567 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:00.598527 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:01.350087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:01.350046 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:02.019814 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:02.019777 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:02.229319 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:02.229270 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:02.724846 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:02.724810 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:03.709086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:03.709026 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:04.471707 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:04.471670 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:04.790045 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:04.789970 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:05.207773 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:05.207734 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:05.916596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:05.916556 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:06.139844 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.139809 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms"] Nov 02 18:17:06.157652 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.148442 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.166872 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.166798 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-312f21\"/\"openshift-service-ca.crt\"" Nov 02 18:17:06.167117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.166866 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-312f21\"/\"default-dockercfg-bq8zl\"" Nov 02 18:17:06.167244 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.166999 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-312f21\"/\"test-model-312f21-kserve-self-signed-certs\"" Nov 02 18:17:06.167339 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.167064 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-312f21\"/\"kube-root-ca.crt\"" Nov 02 18:17:06.167553 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.167531 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms"] Nov 02 18:17:06.214037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.214004 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:06.218181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.218151 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-home\") pod \"test-model-312f21-kserve-dfc5cf8d5-xdzms\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.218357 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.218192 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-tmp-dir\") pod \"test-model-312f21-kserve-dfc5cf8d5-xdzms\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.218357 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.218220 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-dshm\") pod \"test-model-312f21-kserve-dfc5cf8d5-xdzms\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.218474 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.218404 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-model-cache\") pod \"test-model-312f21-kserve-dfc5cf8d5-xdzms\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.218474 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.218427 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtvdc\" (UniqueName: \"kubernetes.io/projected/7990f1b0-ff87-4445-8a17-93052b4c9586-kube-api-access-xtvdc\") pod \"test-model-312f21-kserve-dfc5cf8d5-xdzms\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.218589 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.218528 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7990f1b0-ff87-4445-8a17-93052b4c9586-tls-certs\") pod \"test-model-312f21-kserve-dfc5cf8d5-xdzms\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.325633 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.325589 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-home\") pod \"test-model-312f21-kserve-dfc5cf8d5-xdzms\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.364980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.353809 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-tmp-dir\") pod \"test-model-312f21-kserve-dfc5cf8d5-xdzms\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.364980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.353883 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-dshm\") pod \"test-model-312f21-kserve-dfc5cf8d5-xdzms\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.364980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.354130 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-model-cache\") pod \"test-model-312f21-kserve-dfc5cf8d5-xdzms\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.364980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.354158 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtvdc\" (UniqueName: \"kubernetes.io/projected/7990f1b0-ff87-4445-8a17-93052b4c9586-kube-api-access-xtvdc\") pod \"test-model-312f21-kserve-dfc5cf8d5-xdzms\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.364980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.354267 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7990f1b0-ff87-4445-8a17-93052b4c9586-tls-certs\") pod \"test-model-312f21-kserve-dfc5cf8d5-xdzms\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.364980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.357217 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-home\") pod \"test-model-312f21-kserve-dfc5cf8d5-xdzms\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.364980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.357802 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-tmp-dir\") pod \"test-model-312f21-kserve-dfc5cf8d5-xdzms\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.364980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.359108 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-model-cache\") pod \"test-model-312f21-kserve-dfc5cf8d5-xdzms\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.370979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.368739 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7990f1b0-ff87-4445-8a17-93052b4c9586-tls-certs\") pod \"test-model-312f21-kserve-dfc5cf8d5-xdzms\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.370979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.369114 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-dshm\") pod \"test-model-312f21-kserve-dfc5cf8d5-xdzms\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.409009 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.397990 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtvdc\" (UniqueName: \"kubernetes.io/projected/7990f1b0-ff87-4445-8a17-93052b4c9586-kube-api-access-xtvdc\") pod \"test-model-312f21-kserve-dfc5cf8d5-xdzms\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.482084 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.482047 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:06.785704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.785677 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms"] Nov 02 18:17:06.786065 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:17:06.786044 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7990f1b0_ff87_4445_8a17_93052b4c9586.slice/crio-e551cc083eaa1d1984d7e3133ba954dbfe50dd586106f6cc766e95f35bfbc4c1 WatchSource:0}: Error finding container e551cc083eaa1d1984d7e3133ba954dbfe50dd586106f6cc766e95f35bfbc4c1: Status 404 returned error can't find the container with id e551cc083eaa1d1984d7e3133ba954dbfe50dd586106f6cc766e95f35bfbc4c1 Nov 02 18:17:06.826971 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:06.826925 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:07.411912 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:07.411870 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" event={"ID":"7990f1b0-ff87-4445-8a17-93052b4c9586","Type":"ContainerStarted","Data":"20cd6bed1629142e0b4d0328c535c0334b0875c77979aeabd738eb8a950766ff"} Nov 02 18:17:07.412352 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:07.411921 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" event={"ID":"7990f1b0-ff87-4445-8a17-93052b4c9586","Type":"ContainerStarted","Data":"e551cc083eaa1d1984d7e3133ba954dbfe50dd586106f6cc766e95f35bfbc4c1"} Nov 02 18:17:07.434738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:07.434685 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" podStartSLOduration=1.434671259 podStartE2EDuration="1.434671259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:07.42925035 +0000 UTC m=+2599.217755875" watchObservedRunningTime="2026-11-02 18:17:07.434671259 +0000 UTC m=+2599.223176786" Nov 02 18:17:07.482657 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:07.482630 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:07.492475 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:07.492444 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:08.222716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:08.222662 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:08.939976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:08.938349 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:09.572742 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:09.572707 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:17:09.724063 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:09.724023 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:10.577767 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:10.577658 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:11.459913 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:11.459877 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:12.356976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:12.356260 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:13.161939 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:13.161881 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:14.284990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:14.284573 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:15.140780 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:15.140746 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:16.014131 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.014053 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:16.465966 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.465927 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj"] Nov 02 18:17:16.473294 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.473266 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.482979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.479328 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-tmp-dir\") pod \"test-model-0b0610-kserve-684695468f-kfhfj\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.514989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.512530 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxr69\" (UniqueName: \"kubernetes.io/projected/4a853b47-0443-484f-b3a3-fdd13be51ba7-kube-api-access-fxr69\") pod \"test-model-0b0610-kserve-684695468f-kfhfj\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.514989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.512759 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-model-cache\") pod \"test-model-0b0610-kserve-684695468f-kfhfj\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.514989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.512895 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a853b47-0443-484f-b3a3-fdd13be51ba7-tls-certs\") pod \"test-model-0b0610-kserve-684695468f-kfhfj\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.514989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.513095 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-home\") pod \"test-model-0b0610-kserve-684695468f-kfhfj\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.514989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.513158 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-dshm\") pod \"test-model-0b0610-kserve-684695468f-kfhfj\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.514989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.491113 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:16.514989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.514495 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0b0610\"/\"openshift-service-ca.crt\"" Nov 02 18:17:16.514989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.514597 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0b0610\"/\"kube-root-ca.crt\"" Nov 02 18:17:16.514989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.514733 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0b0610\"/\"default-dockercfg-rbkc8\"" Nov 02 18:17:16.514989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.514844 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0b0610\"/\"test-model-0b0610-kserve-self-signed-certs\"" Nov 02 18:17:16.525798 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.525767 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj"] Nov 02 18:17:16.534087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.534062 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:16.546472 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.546449 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:17:16.560837 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.560809 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:16.615176 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.615129 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-dshm\") pod \"test-model-0b0610-kserve-684695468f-kfhfj\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.615372 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.615226 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-tmp-dir\") pod \"test-model-0b0610-kserve-684695468f-kfhfj\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.615443 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.615407 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxr69\" (UniqueName: \"kubernetes.io/projected/4a853b47-0443-484f-b3a3-fdd13be51ba7-kube-api-access-fxr69\") pod \"test-model-0b0610-kserve-684695468f-kfhfj\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.615510 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.615494 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-model-cache\") pod \"test-model-0b0610-kserve-684695468f-kfhfj\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.615614 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.615593 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a853b47-0443-484f-b3a3-fdd13be51ba7-tls-certs\") pod \"test-model-0b0610-kserve-684695468f-kfhfj\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.615731 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.615717 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-home\") pod \"test-model-0b0610-kserve-684695468f-kfhfj\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.616235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.616208 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-home\") pod \"test-model-0b0610-kserve-684695468f-kfhfj\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.616658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.616634 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-model-cache\") pod \"test-model-0b0610-kserve-684695468f-kfhfj\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.616883 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.616854 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-tmp-dir\") pod \"test-model-0b0610-kserve-684695468f-kfhfj\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.623076 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.623039 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-dshm\") pod \"test-model-0b0610-kserve-684695468f-kfhfj\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.623476 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.623246 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a853b47-0443-484f-b3a3-fdd13be51ba7-tls-certs\") pod \"test-model-0b0610-kserve-684695468f-kfhfj\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.625938 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.625915 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxr69\" (UniqueName: \"kubernetes.io/projected/4a853b47-0443-484f-b3a3-fdd13be51ba7-kube-api-access-fxr69\") pod \"test-model-0b0610-kserve-684695468f-kfhfj\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:16.735698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.735539 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:16.840835 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:16.836087 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:17.265239 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:17:17.265202 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a853b47_0443_484f_b3a3_fdd13be51ba7.slice/crio-b8a02f66f54fb39f38ba53a972ff32c61ef51e2261624d1a197af3476bc41809 WatchSource:0}: Error finding container b8a02f66f54fb39f38ba53a972ff32c61ef51e2261624d1a197af3476bc41809: Status 404 returned error can't find the container with id b8a02f66f54fb39f38ba53a972ff32c61ef51e2261624d1a197af3476bc41809 Nov 02 18:17:17.265661 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:17.265611 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj"] Nov 02 18:17:17.449399 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:17.448326 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:17.542714 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:17.542674 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" event={"ID":"4a853b47-0443-484f-b3a3-fdd13be51ba7","Type":"ContainerStarted","Data":"0ff2564ba5a097315a4700e3c0ab392bbffbd036bfd1dc24393be2bbddb0d955"} Nov 02 18:17:17.542932 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:17.542726 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" event={"ID":"4a853b47-0443-484f-b3a3-fdd13be51ba7","Type":"ContainerStarted","Data":"b8a02f66f54fb39f38ba53a972ff32c61ef51e2261624d1a197af3476bc41809"} Nov 02 18:17:17.563921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:17.563820 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" podStartSLOduration=1.563803188 podStartE2EDuration="1.563803188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:17.560283669 +0000 UTC m=+2609.348789198" watchObservedRunningTime="2026-11-02 18:17:17.563803188 +0000 UTC m=+2609.352308716" Nov 02 18:17:17.614468 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:17:17.614430 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:17:17.837109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:17.837030 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:18.273947 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:18.273878 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:18.984483 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:18.983903 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:19.135903 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:19.135866 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:19.265332 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:19.265226 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:19.571879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:19.571803 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:19.717660 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:19.717627 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:19.995526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:19.995494 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:20.156977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:20.146179 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:20.726993 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:20.726889 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:21.508928 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:21.508828 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:22.307990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:22.304392 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:23.011622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:23.011590 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:23.603884 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:23.603849 5549 generic.go:356] "Generic (PLEG): container finished" podID="b3110f2f-be09-4cbb-9687-e642a0268441" containerID="bc33d8602d13b5912358056b5ca160550230851af9a698f74bfa68fb19a90f6a" exitCode=0 Nov 02 18:17:23.604347 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:23.603921 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" event={"ID":"b3110f2f-be09-4cbb-9687-e642a0268441","Type":"ContainerDied","Data":"bc33d8602d13b5912358056b5ca160550230851af9a698f74bfa68fb19a90f6a"} Nov 02 18:17:23.604469 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:23.604454 5549 scope.go:122] "RemoveContainer" containerID="bc33d8602d13b5912358056b5ca160550230851af9a698f74bfa68fb19a90f6a" Nov 02 18:17:23.606804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:23.606776 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:17:23.606890 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:23.606846 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:17:23.606935 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:23.606894 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:17:24.194916 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:24.194867 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:24.351683 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:24.351525 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:24.500771 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:24.500740 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:24.614829 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:24.614796 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" event={"ID":"b3110f2f-be09-4cbb-9687-e642a0268441","Type":"ContainerStarted","Data":"16d2b804ef90eb3ab1637fe274ef6e14f8ec47a94773305df2fd15997344ae16"} Nov 02 18:17:25.124853 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:25.124819 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:25.556100 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:25.556068 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:17:25.556284 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:25.556259 5549 patch_prober.go:29] interesting pod/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body= Nov 02 18:17:25.556334 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:25.556314 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" podUID="b3110f2f-be09-4cbb-9687-e642a0268441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" Nov 02 18:17:25.565174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:25.565145 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:25.759980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:25.734928 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:26.302908 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:26.302862 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:26.556253 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:26.556165 5549 patch_prober.go:29] interesting pod/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body= Nov 02 18:17:26.556416 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:26.556240 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" podUID="b3110f2f-be09-4cbb-9687-e642a0268441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" Nov 02 18:17:26.837012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:26.836900 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:26.850463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:26.850439 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:27.118269 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:27.118184 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:27.555456 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:27.555424 5549 patch_prober.go:29] interesting pod/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body= Nov 02 18:17:27.555650 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:27.555494 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" podUID="b3110f2f-be09-4cbb-9687-e642a0268441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" Nov 02 18:17:27.654355 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:27.654324 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:17:27.892126 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:27.892032 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:28.150785 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:28.150678 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:28.556522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:28.556485 5549 patch_prober.go:29] interesting pod/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body= Nov 02 18:17:28.556740 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:28.556549 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" podUID="b3110f2f-be09-4cbb-9687-e642a0268441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" Nov 02 18:17:28.678773 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:28.678739 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:29.013978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.013797 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj"] Nov 02 18:17:29.023251 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.023220 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj"] Nov 02 18:17:29.023488 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.023320 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.029607 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.029582 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-22dfc1\"/\"kube-root-ca.crt\"" Nov 02 18:17:29.031515 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.031485 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-22dfc1\"/\"test-model-22dfc1-kserve-self-signed-certs\"" Nov 02 18:17:29.031865 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.031767 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-22dfc1\"/\"default-dockercfg-qp2fr\"" Nov 02 18:17:29.032182 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.032163 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-22dfc1\"/\"openshift-service-ca.crt\"" Nov 02 18:17:29.127802 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.127774 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:29.133370 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.133340 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-dshm\") pod \"test-model-22dfc1-kserve-698cd6fc87-xdwdj\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.133514 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.133471 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63194adc-5a21-49ca-a101-da5038c43036-tls-certs\") pod \"test-model-22dfc1-kserve-698cd6fc87-xdwdj\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.133588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.133566 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-model-cache\") pod \"test-model-22dfc1-kserve-698cd6fc87-xdwdj\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.133832 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.133812 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hrgk\" (UniqueName: \"kubernetes.io/projected/63194adc-5a21-49ca-a101-da5038c43036-kube-api-access-6hrgk\") pod \"test-model-22dfc1-kserve-698cd6fc87-xdwdj\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.134033 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.134014 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-tmp-dir\") pod \"test-model-22dfc1-kserve-698cd6fc87-xdwdj\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.134137 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.134121 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-home\") pod \"test-model-22dfc1-kserve-698cd6fc87-xdwdj\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.238978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.236496 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-home\") pod \"test-model-22dfc1-kserve-698cd6fc87-xdwdj\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.238978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.236667 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-dshm\") pod \"test-model-22dfc1-kserve-698cd6fc87-xdwdj\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.238978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.236777 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63194adc-5a21-49ca-a101-da5038c43036-tls-certs\") pod \"test-model-22dfc1-kserve-698cd6fc87-xdwdj\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.238978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.236860 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-model-cache\") pod \"test-model-22dfc1-kserve-698cd6fc87-xdwdj\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.238978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.236933 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hrgk\" (UniqueName: \"kubernetes.io/projected/63194adc-5a21-49ca-a101-da5038c43036-kube-api-access-6hrgk\") pod \"test-model-22dfc1-kserve-698cd6fc87-xdwdj\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.238978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.237095 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-tmp-dir\") pod \"test-model-22dfc1-kserve-698cd6fc87-xdwdj\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.238978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.237616 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-tmp-dir\") pod \"test-model-22dfc1-kserve-698cd6fc87-xdwdj\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.238978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.237888 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-home\") pod \"test-model-22dfc1-kserve-698cd6fc87-xdwdj\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.253590 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.253554 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-model-cache\") pod \"test-model-22dfc1-kserve-698cd6fc87-xdwdj\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.294295 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.283622 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63194adc-5a21-49ca-a101-da5038c43036-tls-certs\") pod \"test-model-22dfc1-kserve-698cd6fc87-xdwdj\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.294295 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.287282 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-dshm\") pod \"test-model-22dfc1-kserve-698cd6fc87-xdwdj\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.370594 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.370541 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hrgk\" (UniqueName: \"kubernetes.io/projected/63194adc-5a21-49ca-a101-da5038c43036-kube-api-access-6hrgk\") pod \"test-model-22dfc1-kserve-698cd6fc87-xdwdj\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.555632 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.555513 5549 patch_prober.go:29] interesting pod/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body= Nov 02 18:17:29.555632 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.555613 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" podUID="b3110f2f-be09-4cbb-9687-e642a0268441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" Nov 02 18:17:29.668973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.665486 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:29.712836 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:29.710561 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:30.048410 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:17:30.048369 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63194adc_5a21_49ca_a101_da5038c43036.slice/crio-4518963ab92cac3527593a18eb25bf20396544686ea758609cafc3cd7ee10e19 WatchSource:0}: Error finding container 4518963ab92cac3527593a18eb25bf20396544686ea758609cafc3cd7ee10e19: Status 404 returned error can't find the container with id 4518963ab92cac3527593a18eb25bf20396544686ea758609cafc3cd7ee10e19 Nov 02 18:17:30.049181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:30.049087 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj"] Nov 02 18:17:30.497381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:30.496843 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:30.556318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:30.556284 5549 patch_prober.go:29] interesting pod/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body= Nov 02 18:17:30.556539 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:30.556355 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" podUID="b3110f2f-be09-4cbb-9687-e642a0268441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" Nov 02 18:17:30.660252 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:30.660203 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:30.741021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:30.740981 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" event={"ID":"63194adc-5a21-49ca-a101-da5038c43036","Type":"ContainerStarted","Data":"4d6b1ba35e2c08d8a590a49727a56a36a8fd669ad90ddec6fc735a36f8e41bc1"} Nov 02 18:17:30.741196 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:30.741033 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" event={"ID":"63194adc-5a21-49ca-a101-da5038c43036","Type":"ContainerStarted","Data":"4518963ab92cac3527593a18eb25bf20396544686ea758609cafc3cd7ee10e19"} Nov 02 18:17:30.763276 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:30.763181 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" podStartSLOduration=2.763167934 podStartE2EDuration="2.763167934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:30.759136962 +0000 UTC m=+2622.547642481" watchObservedRunningTime="2026-11-02 18:17:30.763167934 +0000 UTC m=+2622.551673461" Nov 02 18:17:31.320535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:31.320502 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:31.560802 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:31.560771 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:17:31.561357 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:31.561336 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:17:31.562152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:31.562134 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:17:31.666495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:31.666422 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:32.013827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:32.013785 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:32.828668 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:32.828628 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:33.514893 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:33.514847 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:34.206784 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:34.206748 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:34.964993 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:34.963488 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:35.720899 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:35.712634 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:36.424221 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:36.424154 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:37.244625 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:37.244581 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:37.788085 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:37.788039 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:37.914235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:37.914203 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:38.046033 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:38.045190 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:38.915973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:38.914825 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:39.666378 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:39.666340 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:39.679227 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:39.679197 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:39.719032 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:39.718996 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:39.833595 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:39.833565 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:17:40.370975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:40.370727 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:40.715275 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:40.715235 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:41.428348 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:41.428309 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:42.144659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:42.144619 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:42.923779 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:42.923699 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:43.641177 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:43.641138 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:44.364997 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:44.362547 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:45.013204 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:45.013167 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:45.991088 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:45.991051 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:46.738499 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:46.738464 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:47.378338 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:47.378304 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:47.590794 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:17:47.590758 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:17:48.036024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:48.035936 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:48.716729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:48.712283 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:49.346372 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:49.346330 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:50.009685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:50.009647 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:50.145042 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:50.145008 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:50.295268 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:50.295172 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:51.014685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:51.014648 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:51.782003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:51.781193 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:52.517421 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:52.517380 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:53.207569 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:53.207531 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:53.937070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:53.931545 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:54.544017 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:54.534113 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:55.299877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:55.299831 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:55.665162 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:55.665084 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:17:56.343102 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:56.338346 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:57.459408 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:57.459371 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:58.976939 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:58.972865 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:59.904694 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:17:59.904641 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:00.779544 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:00.779503 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:03.439824 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:03.439791 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:03.918735 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:03.918693 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:04.560976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:04.557897 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:05.320648 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:05.320610 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:06.004744 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:06.004463 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:06.685978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:06.683119 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:07.332501 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:07.332460 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:07.985976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:07.978796 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:08.669276 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:08.669243 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:09.414593 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:09.414482 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:10.129493 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:10.129450 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:10.950396 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:10.949845 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:11.748770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:11.748692 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:12.532688 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:12.532636 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:13.228325 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:13.228288 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:14.047424 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:14.047305 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:14.761049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:14.760507 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:15.535072 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:15.534995 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:16.252236 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:16.252193 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:16.888187 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:16.888143 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:17.554317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:17.554282 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:17.620507 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:18:17.620469 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:18:18.478228 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:18.478192 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:19.152604 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:19.152561 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:19.824406 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:19.824371 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:20.493493 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:20.493454 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:21.175977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:21.174552 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:21.813099 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:21.813047 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:22.528872 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:22.528839 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:23.436169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:23.436135 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:24.313914 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:24.313855 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:25.059856 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:25.059700 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:26.407861 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:26.407809 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:27.090142 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:27.090096 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:27.941194 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:27.941159 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:28.683883 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:28.683701 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:29.539503 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:29.539461 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:30.421614 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:30.421576 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:31.138360 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:31.138310 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:31.960004 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:31.959964 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:32.611881 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:32.611843 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:33.301855 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:33.299834 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:34.033919 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:34.031730 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:34.710389 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:34.710089 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:35.378993 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:35.373858 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:36.125977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:36.123659 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:36.921937 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:36.921672 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:37.712797 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:37.712738 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:38.550929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:38.550876 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:39.334976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:39.330266 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:40.131084 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:40.131047 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:40.893631 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:40.893527 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:41.627104 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:41.627066 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:42.419975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:42.419552 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:43.231873 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:43.231805 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:43.916254 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:43.916218 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:44.754174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:44.754139 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:45.419948 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:45.419815 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:46.097026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:46.096940 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:46.751777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:46.751708 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:47.434030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:47.433987 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:47.589572 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:18:47.589532 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:18:48.217236 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:48.217187 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:48.912703 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:48.912206 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:49.651975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:49.647696 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:49.848653 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:49.848623 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:18:50.098202 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:50.098172 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:18:50.438039 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:50.437659 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:51.199982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:51.199850 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:51.836553 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:51.836513 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:52.615500 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:52.615463 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:53.264451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:53.264416 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:54.008975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:54.007016 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:54.724976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:54.722318 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:55.435972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:55.435914 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:55.781046 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:55.781008 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:18:56.148213 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:56.148122 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:56.821445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:56.821407 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:57.913904 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:57.913069 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:58.645916 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:58.642803 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:59.324543 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:18:59.324503 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:00.140550 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:00.140511 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:00.284404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:00.284330 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:00.919571 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:00.919532 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:01.627783 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:01.627721 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:02.094603 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:02.094570 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:02.317384 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:02.317348 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:03.023763 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:03.023731 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:03.742975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:03.738189 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:04.434391 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:04.434344 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:05.129088 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:05.129028 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:05.938990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:05.933572 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:06.567295 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:06.563727 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:07.223984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:07.222721 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:07.946589 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:07.946551 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:08.609582 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:08.609546 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:09.375486 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:09.375444 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:10.068613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:10.068566 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:10.818152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:10.818101 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:11.527643 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:11.527594 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:12.248977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:12.246775 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:12.856780 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:12.856745 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:13.511237 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:13.511202 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:14.230370 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:14.230304 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:15.011731 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:15.011695 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:15.653296 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:15.653262 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:16.353694 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:16.348713 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:17.015642 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:17.015587 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:17.589335 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:19:17.589297 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:19:17.651324 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:17.651285 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:18.459193 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:18.459154 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:19.248234 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:19.247556 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:19.933455 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:19.933421 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:20.625646 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:20.625599 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:21.321143 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:21.321067 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:22.136681 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:22.136643 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:22.427468 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:22.427371 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks"] Nov 02 18:19:22.526880 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:22.526845 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:22.682124 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:22.673260 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:22.896661 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:22.896623 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r"] Nov 02 18:19:22.981922 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:22.981883 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:23.117719 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:23.117679 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:23.709123 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:23.709086 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:24.434871 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:24.434832 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:25.094651 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:25.094603 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:25.813943 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:25.813906 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:26.454982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:26.452470 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:27.115789 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:27.115750 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:27.428440 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:27.428356 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" podUID="a6e9e87c-d4b0-4191-990a-56a9531c627e" containerName="payload-processing" containerID="cri-o://cfd6e64dc4d4bbcb68887168cc57c336130b428005b58a3643343d88ebcf865e" gracePeriod=25 Nov 02 18:19:27.764405 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:27.764377 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" Nov 02 18:19:27.832981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:27.831984 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e9e87c-d4b0-4191-990a-56a9531c627e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6e9e87c-d4b0-4191-990a-56a9531c627e-provider-credentials\") pod \"a6e9e87c-d4b0-4191-990a-56a9531c627e\" (UID: \"a6e9e87c-d4b0-4191-990a-56a9531c627e\") " Nov 02 18:19:27.839577 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:27.837024 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e9e87c-d4b0-4191-990a-56a9531c627e-kube-api-access-d6gzf\" (UniqueName: \"kubernetes.io/projected/a6e9e87c-d4b0-4191-990a-56a9531c627e-kube-api-access-d6gzf\") pod \"a6e9e87c-d4b0-4191-990a-56a9531c627e\" (UID: \"a6e9e87c-d4b0-4191-990a-56a9531c627e\") " Nov 02 18:19:27.840712 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:27.840684 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e9e87c-d4b0-4191-990a-56a9531c627e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e9e87c-d4b0-4191-990a-56a9531c627e-plugins-config-volume\") pod \"a6e9e87c-d4b0-4191-990a-56a9531c627e\" (UID: \"a6e9e87c-d4b0-4191-990a-56a9531c627e\") " Nov 02 18:19:27.840932 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:27.840915 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e9e87c-d4b0-4191-990a-56a9531c627e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6e9e87c-d4b0-4191-990a-56a9531c627e-routing-overlay-volume\") pod \"a6e9e87c-d4b0-4191-990a-56a9531c627e\" (UID: \"a6e9e87c-d4b0-4191-990a-56a9531c627e\") " Nov 02 18:19:27.842986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:27.842938 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e9e87c-d4b0-4191-990a-56a9531c627e-routing-overlay-volume" pod "a6e9e87c-d4b0-4191-990a-56a9531c627e" (UID: "a6e9e87c-d4b0-4191-990a-56a9531c627e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:19:27.849017 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:27.848947 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e9e87c-d4b0-4191-990a-56a9531c627e-plugins-config-volume" pod "a6e9e87c-d4b0-4191-990a-56a9531c627e" (UID: "a6e9e87c-d4b0-4191-990a-56a9531c627e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:19:27.849481 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:27.849452 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:27.867187 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:27.866612 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e9e87c-d4b0-4191-990a-56a9531c627e-provider-credentials" pod "a6e9e87c-d4b0-4191-990a-56a9531c627e" (UID: "a6e9e87c-d4b0-4191-990a-56a9531c627e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:19:27.869545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:27.869411 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e9e87c-d4b0-4191-990a-56a9531c627e-kube-api-access-d6gzf" pod "a6e9e87c-d4b0-4191-990a-56a9531c627e" (UID: "a6e9e87c-d4b0-4191-990a-56a9531c627e"). InnerVolumeSpecName "kube-api-access-d6gzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:19:27.898442 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:27.898393 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" podUID="75ee5ca5-ee28-4d41-94c4-aba5b11b28c7" containerName="payload-processing" containerID="cri-o://eb6376ea42db355084eeb9f5f1bde51d49ef3a5ad17e06841b0aff76ce244527" gracePeriod=25 Nov 02 18:19:27.944018 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:27.943983 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6e9e87c-d4b0-4191-990a-56a9531c627e-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:19:27.944018 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:27.944013 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6gzf\" (UniqueName: \"kubernetes.io/projected/a6e9e87c-d4b0-4191-990a-56a9531c627e-kube-api-access-d6gzf\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:19:27.944177 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:27.944026 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e9e87c-d4b0-4191-990a-56a9531c627e-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:19:27.944177 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:27.944036 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6e9e87c-d4b0-4191-990a-56a9531c627e-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:19:28.080755 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.080717 5549 generic.go:356] "Generic (PLEG): container finished" podID="75ee5ca5-ee28-4d41-94c4-aba5b11b28c7" containerID="eb6376ea42db355084eeb9f5f1bde51d49ef3a5ad17e06841b0aff76ce244527" exitCode=0 Nov 02 18:19:28.080921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.080888 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" event={"ID":"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7","Type":"ContainerDied","Data":"eb6376ea42db355084eeb9f5f1bde51d49ef3a5ad17e06841b0aff76ce244527"} Nov 02 18:19:28.082803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.082775 5549 generic.go:356] "Generic (PLEG): container finished" podID="a6e9e87c-d4b0-4191-990a-56a9531c627e" containerID="cfd6e64dc4d4bbcb68887168cc57c336130b428005b58a3643343d88ebcf865e" exitCode=0 Nov 02 18:19:28.083046 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.082849 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" event={"ID":"a6e9e87c-d4b0-4191-990a-56a9531c627e","Type":"ContainerDied","Data":"cfd6e64dc4d4bbcb68887168cc57c336130b428005b58a3643343d88ebcf865e"} Nov 02 18:19:28.083046 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.082912 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" event={"ID":"a6e9e87c-d4b0-4191-990a-56a9531c627e","Type":"ContainerDied","Data":"9e63ed70c3404fcd547efc2921f91f12c37353daabcbe2e3cfb278ac8f4e2047"} Nov 02 18:19:28.083046 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.082944 5549 scope.go:122] "RemoveContainer" containerID="cfd6e64dc4d4bbcb68887168cc57c336130b428005b58a3643343d88ebcf865e" Nov 02 18:19:28.083244 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.083121 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks" Nov 02 18:19:28.102780 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.099265 5549 scope.go:122] "RemoveContainer" containerID="cfd6e64dc4d4bbcb68887168cc57c336130b428005b58a3643343d88ebcf865e" Nov 02 18:19:28.102780 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:19:28.099679 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd6e64dc4d4bbcb68887168cc57c336130b428005b58a3643343d88ebcf865e\": container with ID starting with cfd6e64dc4d4bbcb68887168cc57c336130b428005b58a3643343d88ebcf865e not found: ID does not exist" containerID="cfd6e64dc4d4bbcb68887168cc57c336130b428005b58a3643343d88ebcf865e" Nov 02 18:19:28.102780 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.099721 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd6e64dc4d4bbcb68887168cc57c336130b428005b58a3643343d88ebcf865e"} err="failed to get container status \"cfd6e64dc4d4bbcb68887168cc57c336130b428005b58a3643343d88ebcf865e\": rpc error: code = NotFound desc = could not find container \"cfd6e64dc4d4bbcb68887168cc57c336130b428005b58a3643343d88ebcf865e\": container with ID starting with cfd6e64dc4d4bbcb68887168cc57c336130b428005b58a3643343d88ebcf865e not found: ID does not exist" Nov 02 18:19:28.127979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.123145 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks"] Nov 02 18:19:28.128402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.128379 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-42d786/payload-processing-external-model-e2e-extmod-a-42d786-695fxvvks"] Nov 02 18:19:28.168687 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.168660 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" Nov 02 18:19:28.201188 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.201155 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt"] Nov 02 18:19:28.202269 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.202251 5549 cpu_manager.go:414] "containerMap: removing container" podUID="a6e9e87c-d4b0-4191-990a-56a9531c627e" containerName="payload-processing" Nov 02 18:19:28.202269 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.202270 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e9e87c-d4b0-4191-990a-56a9531c627e" containerName="payload-processing" Nov 02 18:19:28.202390 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.202287 5549 cpu_manager.go:414] "containerMap: removing container" podUID="75ee5ca5-ee28-4d41-94c4-aba5b11b28c7" containerName="payload-processing" Nov 02 18:19:28.202390 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.202292 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ee5ca5-ee28-4d41-94c4-aba5b11b28c7" containerName="payload-processing" Nov 02 18:19:28.202467 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.202459 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e9e87c-d4b0-4191-990a-56a9531c627e" containerName="payload-processing" Nov 02 18:19:28.202513 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.202471 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ee5ca5-ee28-4d41-94c4-aba5b11b28c7" containerName="payload-processing" Nov 02 18:19:28.206821 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.206803 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" Nov 02 18:19:28.210832 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.210811 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-06644b-serving-cert\"" Nov 02 18:19:28.216078 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.216052 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt"] Nov 02 18:19:28.249510 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.249485 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-routing-overlay-volume\") pod \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\" (UID: \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\") " Nov 02 18:19:28.249684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.249597 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-provider-credentials\") pod \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\" (UID: \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\") " Nov 02 18:19:28.249749 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.249720 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-kube-api-access-2c4wn\" (UniqueName: \"kubernetes.io/projected/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-kube-api-access-2c4wn\") pod \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\" (UID: \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\") " Nov 02 18:19:28.249860 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.249846 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-plugins-config-volume\") pod \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\" (UID: \"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7\") " Nov 02 18:19:28.250333 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.249881 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-routing-overlay-volume" pod "75ee5ca5-ee28-4d41-94c4-aba5b11b28c7" (UID: "75ee5ca5-ee28-4d41-94c4-aba5b11b28c7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:19:28.250445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.250292 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-plugins-config-volume" pod "75ee5ca5-ee28-4d41-94c4-aba5b11b28c7" (UID: "75ee5ca5-ee28-4d41-94c4-aba5b11b28c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:19:28.250754 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.250727 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:19:28.250870 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.250757 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:19:28.252289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.252265 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-provider-credentials" pod "75ee5ca5-ee28-4d41-94c4-aba5b11b28c7" (UID: "75ee5ca5-ee28-4d41-94c4-aba5b11b28c7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:19:28.252399 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.252356 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-kube-api-access-2c4wn" pod "75ee5ca5-ee28-4d41-94c4-aba5b11b28c7" (UID: "75ee5ca5-ee28-4d41-94c4-aba5b11b28c7"). InnerVolumeSpecName "kube-api-access-2c4wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:19:28.326090 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.326018 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e9e87c-d4b0-4191-990a-56a9531c627e" path="/var/lib/kubelet/pods/a6e9e87c-d4b0-4191-990a-56a9531c627e/volumes" Nov 02 18:19:28.352636 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.352607 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85a86f2-b60d-4dc6-aeb6-be4872586f40-maas-api-tls\") pod \"maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt\" (UID: \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" Nov 02 18:19:28.352789 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.352642 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85a86f2-b60d-4dc6-aeb6-be4872586f40-metrics-certs\") pod \"maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt\" (UID: \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" Nov 02 18:19:28.352863 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.352816 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85a86f2-b60d-4dc6-aeb6-be4872586f40-openshift-service-ca\") pod \"maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt\" (UID: \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" Nov 02 18:19:28.352921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.352876 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ztl\" (UniqueName: \"kubernetes.io/projected/b85a86f2-b60d-4dc6-aeb6-be4872586f40-kube-api-access-s4ztl\") pod \"maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt\" (UID: \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" Nov 02 18:19:28.353187 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.353165 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:19:28.353232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.353192 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c4wn\" (UniqueName: \"kubernetes.io/projected/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7-kube-api-access-2c4wn\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:19:28.454411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.454374 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85a86f2-b60d-4dc6-aeb6-be4872586f40-maas-api-tls\") pod \"maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt\" (UID: \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" Nov 02 18:19:28.454681 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.454416 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85a86f2-b60d-4dc6-aeb6-be4872586f40-metrics-certs\") pod \"maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt\" (UID: \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" Nov 02 18:19:28.454681 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.454508 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85a86f2-b60d-4dc6-aeb6-be4872586f40-openshift-service-ca\") pod \"maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt\" (UID: \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" Nov 02 18:19:28.454681 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.454541 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ztl\" (UniqueName: \"kubernetes.io/projected/b85a86f2-b60d-4dc6-aeb6-be4872586f40-kube-api-access-s4ztl\") pod \"maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt\" (UID: \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" Nov 02 18:19:28.455317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.455287 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85a86f2-b60d-4dc6-aeb6-be4872586f40-openshift-service-ca\") pod \"maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt\" (UID: \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" Nov 02 18:19:28.457105 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.457080 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85a86f2-b60d-4dc6-aeb6-be4872586f40-maas-api-tls\") pod \"maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt\" (UID: \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" Nov 02 18:19:28.457105 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.457108 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85a86f2-b60d-4dc6-aeb6-be4872586f40-metrics-certs\") pod \"maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt\" (UID: \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" Nov 02 18:19:28.505972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.500102 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ztl\" (UniqueName: \"kubernetes.io/projected/b85a86f2-b60d-4dc6-aeb6-be4872586f40-kube-api-access-s4ztl\") pod \"maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt\" (UID: \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" Nov 02 18:19:28.505972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.505679 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:28.528591 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.528407 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" Nov 02 18:19:28.845733 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:19:28.845703 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85a86f2_b60d_4dc6_aeb6_be4872586f40.slice/crio-541ac9dacbbccf2fbb6e081a5712dd76f9b9255a8fab1ba9c567808ba0cd97cc WatchSource:0}: Error finding container 541ac9dacbbccf2fbb6e081a5712dd76f9b9255a8fab1ba9c567808ba0cd97cc: Status 404 returned error can't find the container with id 541ac9dacbbccf2fbb6e081a5712dd76f9b9255a8fab1ba9c567808ba0cd97cc Nov 02 18:19:28.846188 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.846103 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt"] Nov 02 18:19:28.847852 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:28.847828 5549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:19:29.093106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:29.093070 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" Nov 02 18:19:29.093322 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:29.093097 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r" event={"ID":"75ee5ca5-ee28-4d41-94c4-aba5b11b28c7","Type":"ContainerDied","Data":"035cc30a67ca92460a864ece70b7ab2988c7514523610a31de312efa6f8f53be"} Nov 02 18:19:29.093322 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:29.093200 5549 scope.go:122] "RemoveContainer" containerID="eb6376ea42db355084eeb9f5f1bde51d49ef3a5ad17e06841b0aff76ce244527" Nov 02 18:19:29.096428 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:29.096390 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" event={"ID":"b85a86f2-b60d-4dc6-aeb6-be4872586f40","Type":"ContainerStarted","Data":"541ac9dacbbccf2fbb6e081a5712dd76f9b9255a8fab1ba9c567808ba0cd97cc"} Nov 02 18:19:29.116788 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:29.116759 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r"] Nov 02 18:19:29.123599 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:29.123574 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-bdb9lcq8r"] Nov 02 18:19:29.265979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:29.264594 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:29.948987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:29.943527 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:30.084020 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:30.083945 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:30.115889 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:30.115852 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" event={"ID":"b85a86f2-b60d-4dc6-aeb6-be4872586f40","Type":"ContainerStarted","Data":"d6c7935d3ee434b7ce1866f97043a4d0a3551a83365bd3a1e3168e45de75161e"} Nov 02 18:19:30.116106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:30.115938 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" Nov 02 18:19:30.147729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:30.147678 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" podStartSLOduration=100.615145099 podStartE2EDuration="1m41.147665624s" totalImagesPullingTime="532.520525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:19:30.145149934 +0000 UTC m=+2741.933655476" watchObservedRunningTime="2026-11-02 18:19:30.147665624 +0000 UTC m=+2741.936171152" Nov 02 18:19:30.326976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:30.326879 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ee5ca5-ee28-4d41-94c4-aba5b11b28c7" path="/var/lib/kubelet/pods/75ee5ca5-ee28-4d41-94c4-aba5b11b28c7/volumes" Nov 02 18:19:30.507909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:30.507872 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:30.765468 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:30.765391 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:30.915690 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:30.915656 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:31.491907 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:31.491871 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:31.821078 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:31.820857 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:32.471575 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:32.469807 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:33.146024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:33.138932 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:33.830815 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:33.830775 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:34.522093 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:34.522060 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:35.251814 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:35.251763 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:36.007978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:36.007725 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:36.173851 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:36.173816 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" Nov 02 18:19:36.799293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:36.799262 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:38.043051 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:38.042930 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:38.845984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:38.841775 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:39.530977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:39.529942 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:40.273653 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:40.273613 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:40.918291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:40.918228 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:41.739087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:41.739051 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:42.430975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:42.429729 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:43.064976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:43.063199 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:43.959011 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:43.952752 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:44.574083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:44.574041 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:45.249273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:45.248871 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:45.969385 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:45.969348 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:46.633983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:46.633930 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:47.358796 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:47.358745 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:47.590877 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:19:47.590841 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:19:48.021799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:48.019463 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:48.645681 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:48.645646 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:49.345058 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:49.340713 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:49.987753 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:49.987717 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:50.693232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:50.693197 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:51.326819 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:51.326780 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:52.045540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:52.045501 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:52.837982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:52.836153 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:53.499030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:53.498985 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:54.224908 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:54.224872 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:54.940025 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:54.937989 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:55.709572 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:55.709519 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:56.447768 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:56.447709 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:57.118976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:57.115385 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:57.760525 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:57.760368 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:58.684687 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:58.683636 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:59.451080 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:19:59.451023 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:00.246310 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:00.246271 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:01.037813 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:01.037774 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:01.842176 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:01.841670 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:01.952280 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:01.952246 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:20:02.167700 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:02.167592 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:02.712396 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:02.712361 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:02.881335 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:02.881288 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:04.044083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:04.044042 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:04.656256 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:04.656221 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:05.220472 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:05.220428 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:05.994665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:05.994627 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:06.780794 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:06.780746 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:07.592060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:07.592013 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:08.412945 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:08.412905 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:09.250238 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:09.250070 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:10.022003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:10.021926 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:10.776510 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:10.776454 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:11.545640 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:11.542648 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:12.359152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:12.358157 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:13.014567 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:13.014524 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:13.759063 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:13.759017 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:14.628652 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:14.628614 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:15.400770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.400734 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:15.543161 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.543128 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh"] Nov 02 18:20:15.551473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.551431 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.559377 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.559351 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh"] Nov 02 18:20:15.559723 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.559698 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-06644b\"/\"kube-root-ca.crt\"" Nov 02 18:20:15.560330 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.560308 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-06644b\"/\"openshift-service-ca.crt\"" Nov 02 18:20:15.562826 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.562804 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-06644b\"/\"default-dockercfg-h4lbx\"" Nov 02 18:20:15.563134 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.563096 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-06644b\"/\"test-model-06644b-kserve-self-signed-certs\"" Nov 02 18:20:15.630533 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.630492 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:15.660236 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.660157 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-home\") pod \"test-model-06644b-kserve-66c55b658c-hjnvh\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.660402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.660255 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96fcaedb-4f59-4385-92c4-5c04bbef5d20-tls-certs\") pod \"test-model-06644b-kserve-66c55b658c-hjnvh\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.660580 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.660561 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-tmp-dir\") pod \"test-model-06644b-kserve-66c55b658c-hjnvh\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.660641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.660608 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-model-cache\") pod \"test-model-06644b-kserve-66c55b658c-hjnvh\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.660734 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.660709 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqr7q\" (UniqueName: \"kubernetes.io/projected/96fcaedb-4f59-4385-92c4-5c04bbef5d20-kube-api-access-xqr7q\") pod \"test-model-06644b-kserve-66c55b658c-hjnvh\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.660784 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.660772 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-dshm\") pod \"test-model-06644b-kserve-66c55b658c-hjnvh\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.762974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.762350 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-dshm\") pod \"test-model-06644b-kserve-66c55b658c-hjnvh\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.762974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.762566 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-home\") pod \"test-model-06644b-kserve-66c55b658c-hjnvh\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.762974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.762646 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96fcaedb-4f59-4385-92c4-5c04bbef5d20-tls-certs\") pod \"test-model-06644b-kserve-66c55b658c-hjnvh\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.762974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.762792 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-tmp-dir\") pod \"test-model-06644b-kserve-66c55b658c-hjnvh\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.762974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.762879 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-model-cache\") pod \"test-model-06644b-kserve-66c55b658c-hjnvh\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.763443 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.762990 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqr7q\" (UniqueName: \"kubernetes.io/projected/96fcaedb-4f59-4385-92c4-5c04bbef5d20-kube-api-access-xqr7q\") pod \"test-model-06644b-kserve-66c55b658c-hjnvh\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.782976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.763680 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-home\") pod \"test-model-06644b-kserve-66c55b658c-hjnvh\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.782976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.781079 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-tmp-dir\") pod \"test-model-06644b-kserve-66c55b658c-hjnvh\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.782976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.781431 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-model-cache\") pod \"test-model-06644b-kserve-66c55b658c-hjnvh\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.787977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.783589 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96fcaedb-4f59-4385-92c4-5c04bbef5d20-tls-certs\") pod \"test-model-06644b-kserve-66c55b658c-hjnvh\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.847372 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.839662 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-dshm\") pod \"test-model-06644b-kserve-66c55b658c-hjnvh\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.847372 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.846237 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqr7q\" (UniqueName: \"kubernetes.io/projected/96fcaedb-4f59-4385-92c4-5c04bbef5d20-kube-api-access-xqr7q\") pod \"test-model-06644b-kserve-66c55b658c-hjnvh\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:15.881766 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:15.881727 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:16.187789 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:20:16.187757 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96fcaedb_4f59_4385_92c4_5c04bbef5d20.slice/crio-aa6f179742932efd8590a26bc627c486191b88f10c22b3e8d2ce8d99f3937604 WatchSource:0}: Error finding container aa6f179742932efd8590a26bc627c486191b88f10c22b3e8d2ce8d99f3937604: Status 404 returned error can't find the container with id aa6f179742932efd8590a26bc627c486191b88f10c22b3e8d2ce8d99f3937604 Nov 02 18:20:16.188490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:16.188468 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh"] Nov 02 18:20:16.429009 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:16.427275 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:16.755462 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:16.755427 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" event={"ID":"96fcaedb-4f59-4385-92c4-5c04bbef5d20","Type":"ContainerStarted","Data":"593e7b0e6766d0be75c34dd949aefe7176ff90dcf900a2ffddb0b96061906934"} Nov 02 18:20:16.755932 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:16.755470 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" event={"ID":"96fcaedb-4f59-4385-92c4-5c04bbef5d20","Type":"ContainerStarted","Data":"aa6f179742932efd8590a26bc627c486191b88f10c22b3e8d2ce8d99f3937604"} Nov 02 18:20:16.781729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:16.781676 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" podStartSLOduration=1.781663339 podStartE2EDuration="1.781663339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:20:16.776174797 +0000 UTC m=+2788.564680325" watchObservedRunningTime="2026-11-02 18:20:16.781663339 +0000 UTC m=+2788.570168866" Nov 02 18:20:16.882182 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:16.882147 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:17.074684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:17.074561 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:17.589525 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:20:17.589489 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:20:17.915103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:17.915023 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:18.028181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:18.028141 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm"] Nov 02 18:20:18.032889 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:18.032862 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480b0c62-wlrpm"] Nov 02 18:20:18.323425 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:18.323392 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90437a6d-ed10-429a-a0bd-2ec2d891579e" path="/var/lib/kubelet/pods/90437a6d-ed10-429a-a0bd-2ec2d891579e/volumes" Nov 02 18:20:18.844923 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:18.844886 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:18.853981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:18.850835 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7"] Nov 02 18:20:18.998461 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:18.994858 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:19.741669 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:19.741635 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:20.516934 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:20.516861 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:21.286523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:21.286478 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:22.033024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:22.026484 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:22.819281 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:22.819242 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:23.684375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:23.684323 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:24.338765 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:24.338726 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:25.224966 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:25.222881 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:25.882896 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:25.882862 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:25.896881 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:25.896853 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:25.922987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:25.909792 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:26.566052 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:26.564550 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:26.879278 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:26.879199 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:20:27.380925 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:27.380887 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:28.235970 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:28.235925 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:28.961920 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:28.961878 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:29.809300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:29.809217 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:29.937280 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:29.937244 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm"] Nov 02 18:20:30.067044 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:30.052869 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:30.409607 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:30.409504 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:30.610169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:30.610138 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:31.270734 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:31.270627 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:31.931042 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:31.930906 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:32.603832 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:32.601345 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:33.368484 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:33.368443 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:33.882555 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:33.882495 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" podUID="190289c2-9f3a-4201-9cf6-7b38e5edfa86" containerName="main" containerID="cri-o://ddb5028765cde3a42e1aa34d6e5ed19d961f19572550fffdb588febff10b73af" gracePeriod=45 Nov 02 18:20:34.055774 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.055725 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:34.228523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.228494 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:20:34.405909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.405877 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-dshm\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-dshm\") pod \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " Nov 02 18:20:34.406124 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.405964 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-tmp-dir\") pod \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " Nov 02 18:20:34.406257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.406242 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-home\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-home\") pod \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " Nov 02 18:20:34.406324 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.406270 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/190289c2-9f3a-4201-9cf6-7b38e5edfa86-tls-certs\" (UniqueName: \"kubernetes.io/secret/190289c2-9f3a-4201-9cf6-7b38e5edfa86-tls-certs\") pod \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " Nov 02 18:20:34.406324 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.406292 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/190289c2-9f3a-4201-9cf6-7b38e5edfa86-kube-api-access-bsbsp\" (UniqueName: \"kubernetes.io/projected/190289c2-9f3a-4201-9cf6-7b38e5edfa86-kube-api-access-bsbsp\") pod \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " Nov 02 18:20:34.406324 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.406302 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-tmp-dir" pod "190289c2-9f3a-4201-9cf6-7b38e5edfa86" (UID: "190289c2-9f3a-4201-9cf6-7b38e5edfa86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:20:34.406494 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.406342 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-model-cache\") pod \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\" (UID: \"190289c2-9f3a-4201-9cf6-7b38e5edfa86\") " Nov 02 18:20:34.406558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.406531 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-home" pod "190289c2-9f3a-4201-9cf6-7b38e5edfa86" (UID: "190289c2-9f3a-4201-9cf6-7b38e5edfa86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:20:34.406774 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.406743 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-model-cache" pod "190289c2-9f3a-4201-9cf6-7b38e5edfa86" (UID: "190289c2-9f3a-4201-9cf6-7b38e5edfa86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:20:34.407179 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.407160 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:20:34.407265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.407183 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:20:34.407265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.407197 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:20:34.408393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.408365 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-dshm" pod "190289c2-9f3a-4201-9cf6-7b38e5edfa86" (UID: "190289c2-9f3a-4201-9cf6-7b38e5edfa86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:20:34.408393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.408373 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190289c2-9f3a-4201-9cf6-7b38e5edfa86-tls-certs" pod "190289c2-9f3a-4201-9cf6-7b38e5edfa86" (UID: "190289c2-9f3a-4201-9cf6-7b38e5edfa86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:20:34.408534 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.408424 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190289c2-9f3a-4201-9cf6-7b38e5edfa86-kube-api-access-bsbsp" pod "190289c2-9f3a-4201-9cf6-7b38e5edfa86" (UID: "190289c2-9f3a-4201-9cf6-7b38e5edfa86"). InnerVolumeSpecName "kube-api-access-bsbsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:20:34.508746 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.508710 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190289c2-9f3a-4201-9cf6-7b38e5edfa86-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:20:34.508746 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.508742 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsbsp\" (UniqueName: \"kubernetes.io/projected/190289c2-9f3a-4201-9cf6-7b38e5edfa86-kube-api-access-bsbsp\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:20:34.508746 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.508752 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190289c2-9f3a-4201-9cf6-7b38e5edfa86-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:20:34.840034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.838861 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:34.949338 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.949303 5549 generic.go:356] "Generic (PLEG): container finished" podID="190289c2-9f3a-4201-9cf6-7b38e5edfa86" containerID="ddb5028765cde3a42e1aa34d6e5ed19d961f19572550fffdb588febff10b73af" exitCode=0 Nov 02 18:20:34.949745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.949349 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" Nov 02 18:20:34.949745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.949388 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" event={"ID":"190289c2-9f3a-4201-9cf6-7b38e5edfa86","Type":"ContainerDied","Data":"ddb5028765cde3a42e1aa34d6e5ed19d961f19572550fffdb588febff10b73af"} Nov 02 18:20:34.949745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.949428 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7" event={"ID":"190289c2-9f3a-4201-9cf6-7b38e5edfa86","Type":"ContainerDied","Data":"7493ee72c8339ef9b8afbb4d6ee8df22312fadfe1ab5bcf3d2e62eed4d173dbd"} Nov 02 18:20:34.949745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.949447 5549 scope.go:122] "RemoveContainer" containerID="ddb5028765cde3a42e1aa34d6e5ed19d961f19572550fffdb588febff10b73af" Nov 02 18:20:34.962155 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.962130 5549 scope.go:122] "RemoveContainer" containerID="ddb5028765cde3a42e1aa34d6e5ed19d961f19572550fffdb588febff10b73af" Nov 02 18:20:34.962438 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:20:34.962416 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb5028765cde3a42e1aa34d6e5ed19d961f19572550fffdb588febff10b73af\": container with ID starting with ddb5028765cde3a42e1aa34d6e5ed19d961f19572550fffdb588febff10b73af not found: ID does not exist" containerID="ddb5028765cde3a42e1aa34d6e5ed19d961f19572550fffdb588febff10b73af" Nov 02 18:20:34.962516 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.962451 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb5028765cde3a42e1aa34d6e5ed19d961f19572550fffdb588febff10b73af"} err="failed to get container status \"ddb5028765cde3a42e1aa34d6e5ed19d961f19572550fffdb588febff10b73af\": rpc error: code = NotFound desc = could not find container \"ddb5028765cde3a42e1aa34d6e5ed19d961f19572550fffdb588febff10b73af\": container with ID starting with ddb5028765cde3a42e1aa34d6e5ed19d961f19572550fffdb588febff10b73af not found: ID does not exist" Nov 02 18:20:34.991526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.991494 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7"] Nov 02 18:20:34.995860 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:34.995839 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-78383f/test-model-78383f-kserve-845bf778df-8n6x7"] Nov 02 18:20:35.514440 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:35.511451 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:36.327678 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:36.327636 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="190289c2-9f3a-4201-9cf6-7b38e5edfa86" path="/var/lib/kubelet/pods/190289c2-9f3a-4201-9cf6-7b38e5edfa86/volumes" Nov 02 18:20:36.407785 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:36.404653 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:36.628693 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:36.628609 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx"] Nov 02 18:20:36.641481 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:36.641446 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75"] Nov 02 18:20:36.732493 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:36.732453 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:36.914030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:36.904355 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:37.187259 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:37.187168 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:37.549820 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:37.549780 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:38.329471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:38.329436 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:39.205480 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:39.205437 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:39.967710 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:39.967668 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:40.733982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:40.733889 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:41.431976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:41.431620 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:41.628984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:41.628899 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" podUID="ec19d984-fac8-4ec5-9666-d3c9d3505b11" containerName="payload-processing" containerID="cri-o://cc6d3423bc09a5b4d73f8462a7adbd4aa45509e4715662717d5734a93a664e4e" gracePeriod=25 Nov 02 18:20:41.642320 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:41.642284 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" podUID="d9cbadb5-683a-46a2-964a-7c144a467125" containerName="payload-pre-processing" containerID="cri-o://042e9b81a513769b0d5c3da6f9a4ce20ad79243762b70cb2ed9069cc381179d4" gracePeriod=25 Nov 02 18:20:42.123882 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:42.123845 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:42.289801 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:42.289756 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" podUID="ec19d984-fac8-4ec5-9666-d3c9d3505b11" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:20:42.290001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:42.289774 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" podUID="d9cbadb5-683a-46a2-964a-7c144a467125" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:20:42.834919 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:42.834873 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:43.226139 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:43.226107 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:43.554465 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:43.554389 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:43.702311 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:43.702278 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:44.248700 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:44.248661 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:44.954977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:44.951683 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:44.971713 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:44.971635 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" podUID="3c2b7cf4-52fc-4044-ad57-34fb04d241c3" containerName="main" containerID="cri-o://322ff85d759e9dce77e6afa6a005485a7f82b301897387940c3a1e0a7f3f8a2a" gracePeriod=45 Nov 02 18:20:45.110111 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.110072 5549 generic.go:356] "Generic (PLEG): container finished" podID="3c2b7cf4-52fc-4044-ad57-34fb04d241c3" containerID="322ff85d759e9dce77e6afa6a005485a7f82b301897387940c3a1e0a7f3f8a2a" exitCode=0 Nov 02 18:20:45.110254 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.110141 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" event={"ID":"3c2b7cf4-52fc-4044-ad57-34fb04d241c3","Type":"ContainerDied","Data":"322ff85d759e9dce77e6afa6a005485a7f82b301897387940c3a1e0a7f3f8a2a"} Nov 02 18:20:45.279984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.279924 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:20:45.357753 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.357718 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-tls-certs\") pod \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " Nov 02 18:20:45.357930 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.357774 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-home\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-home\") pod \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " Nov 02 18:20:45.357930 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.357823 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-dshm\") pod \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " Nov 02 18:20:45.357930 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.357849 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-model-cache\") pod \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " Nov 02 18:20:45.358173 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.357964 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-tmp-dir\") pod \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " Nov 02 18:20:45.358173 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.358042 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-kube-api-access-szz9n\" (UniqueName: \"kubernetes.io/projected/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-kube-api-access-szz9n\") pod \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\" (UID: \"3c2b7cf4-52fc-4044-ad57-34fb04d241c3\") " Nov 02 18:20:45.358320 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.358168 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-home" pod "3c2b7cf4-52fc-4044-ad57-34fb04d241c3" (UID: "3c2b7cf4-52fc-4044-ad57-34fb04d241c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:20:45.358320 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.358210 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-model-cache" pod "3c2b7cf4-52fc-4044-ad57-34fb04d241c3" (UID: "3c2b7cf4-52fc-4044-ad57-34fb04d241c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:20:45.358320 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.358302 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-tmp-dir" pod "3c2b7cf4-52fc-4044-ad57-34fb04d241c3" (UID: "3c2b7cf4-52fc-4044-ad57-34fb04d241c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:20:45.359079 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.359052 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:20:45.359079 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.359080 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:20:45.359295 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.359094 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:20:45.360389 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.360365 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-tls-certs" pod "3c2b7cf4-52fc-4044-ad57-34fb04d241c3" (UID: "3c2b7cf4-52fc-4044-ad57-34fb04d241c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:20:45.360558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.360541 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-dshm" pod "3c2b7cf4-52fc-4044-ad57-34fb04d241c3" (UID: "3c2b7cf4-52fc-4044-ad57-34fb04d241c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:20:45.360630 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.360549 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-kube-api-access-szz9n" pod "3c2b7cf4-52fc-4044-ad57-34fb04d241c3" (UID: "3c2b7cf4-52fc-4044-ad57-34fb04d241c3"). InnerVolumeSpecName "kube-api-access-szz9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:20:45.460833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.460798 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:20:45.460833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.460829 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:20:45.461054 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.460839 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szz9n\" (UniqueName: \"kubernetes.io/projected/3c2b7cf4-52fc-4044-ad57-34fb04d241c3-kube-api-access-szz9n\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:20:45.707946 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:45.707894 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:46.119762 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:46.119669 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" Nov 02 18:20:46.119762 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:46.119692 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" event={"ID":"3c2b7cf4-52fc-4044-ad57-34fb04d241c3","Type":"ContainerDied","Data":"034d1ecf7590c3e77611a0a84ae3545cea4b5c6f5c011ac4042828417e62d692"} Nov 02 18:20:46.119762 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:46.119748 5549 scope.go:122] "RemoveContainer" containerID="322ff85d759e9dce77e6afa6a005485a7f82b301897387940c3a1e0a7f3f8a2a" Nov 02 18:20:46.150072 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:46.150046 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm"] Nov 02 18:20:46.154381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:46.154355 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm"] Nov 02 18:20:46.215371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:46.215330 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-658b8c/test-model-658b8c-kserve-5d9f6d89cf-df8fm" podUID="3c2b7cf4-52fc-4044-ad57-34fb04d241c3" containerName="main" probeResult="failure" output="Get \"https://10.133.1.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:20:46.326536 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:46.326502 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c2b7cf4-52fc-4044-ad57-34fb04d241c3" path="/var/lib/kubelet/pods/3c2b7cf4-52fc-4044-ad57-34fb04d241c3/volumes" Nov 02 18:20:46.462266 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:46.461904 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:47.164981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:47.159036 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:47.589219 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:20:47.589183 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:20:47.848686 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:47.847735 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:48.533415 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:48.533376 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:49.231421 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:49.231379 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:49.946693 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:49.946658 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:50.358406 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:50.349242 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:52.005612 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:52.005567 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:52.289053 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:52.288962 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" podUID="d9cbadb5-683a-46a2-964a-7c144a467125" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:20:52.289053 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:52.289021 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" podUID="ec19d984-fac8-4ec5-9666-d3c9d3505b11" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:20:52.736189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:52.735490 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:53.557438 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:53.557401 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:54.433798 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:54.433761 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:55.135928 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:55.132144 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:55.826263 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:55.826228 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:56.551028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:56.543555 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:57.228226 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:57.227208 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:57.905241 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:57.905199 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:58.651971 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:58.649069 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:59.275925 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:20:59.275810 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:00.061434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:00.061393 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:00.205838 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:00.205802 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:00.567320 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:00.567286 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:00.837845 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:00.837751 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8"] Nov 02 18:21:00.838088 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:00.838020 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" containerID="cri-o://a90f923d31d7b3b16c3121acba8214aa159d7185be731660e85b7738ab6e3478" gracePeriod=30 Nov 02 18:21:00.919183 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:00.919145 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:01.085771 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:01.085716 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:01.659535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:01.659489 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:01.955202 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:01.955127 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:02.018018 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.017985 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" Nov 02 18:21:02.081612 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.081584 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" Nov 02 18:21:02.157098 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.157045 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec19d984-fac8-4ec5-9666-d3c9d3505b11-kube-api-access-9blzk\" (UniqueName: \"kubernetes.io/projected/ec19d984-fac8-4ec5-9666-d3c9d3505b11-kube-api-access-9blzk\") pod \"ec19d984-fac8-4ec5-9666-d3c9d3505b11\" (UID: \"ec19d984-fac8-4ec5-9666-d3c9d3505b11\") " Nov 02 18:21:02.157515 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.157492 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec19d984-fac8-4ec5-9666-d3c9d3505b11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec19d984-fac8-4ec5-9666-d3c9d3505b11-plugins-config-volume\") pod \"ec19d984-fac8-4ec5-9666-d3c9d3505b11\" (UID: \"ec19d984-fac8-4ec5-9666-d3c9d3505b11\") " Nov 02 18:21:02.157965 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.157929 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec19d984-fac8-4ec5-9666-d3c9d3505b11-plugins-config-volume" pod "ec19d984-fac8-4ec5-9666-d3c9d3505b11" (UID: "ec19d984-fac8-4ec5-9666-d3c9d3505b11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:02.158173 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.158154 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec19d984-fac8-4ec5-9666-d3c9d3505b11-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:02.159403 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.159379 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec19d984-fac8-4ec5-9666-d3c9d3505b11-kube-api-access-9blzk" pod "ec19d984-fac8-4ec5-9666-d3c9d3505b11" (UID: "ec19d984-fac8-4ec5-9666-d3c9d3505b11"). InnerVolumeSpecName "kube-api-access-9blzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:02.213400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.213364 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:02.259223 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.259186 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9cbadb5-683a-46a2-964a-7c144a467125-kube-api-access-rqlxv\" (UniqueName: \"kubernetes.io/projected/d9cbadb5-683a-46a2-964a-7c144a467125-kube-api-access-rqlxv\") pod \"d9cbadb5-683a-46a2-964a-7c144a467125\" (UID: \"d9cbadb5-683a-46a2-964a-7c144a467125\") " Nov 02 18:21:02.259470 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.259453 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9cbadb5-683a-46a2-964a-7c144a467125-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9cbadb5-683a-46a2-964a-7c144a467125-plugins-config-volume\") pod \"d9cbadb5-683a-46a2-964a-7c144a467125\" (UID: \"d9cbadb5-683a-46a2-964a-7c144a467125\") " Nov 02 18:21:02.259987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.259923 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9cbadb5-683a-46a2-964a-7c144a467125-plugins-config-volume" pod "d9cbadb5-683a-46a2-964a-7c144a467125" (UID: "d9cbadb5-683a-46a2-964a-7c144a467125"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:02.260302 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.260278 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9blzk\" (UniqueName: \"kubernetes.io/projected/ec19d984-fac8-4ec5-9666-d3c9d3505b11-kube-api-access-9blzk\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:02.260400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.260308 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9cbadb5-683a-46a2-964a-7c144a467125-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:02.261592 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.261564 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9cbadb5-683a-46a2-964a-7c144a467125-kube-api-access-rqlxv" pod "d9cbadb5-683a-46a2-964a-7c144a467125" (UID: "d9cbadb5-683a-46a2-964a-7c144a467125"). InnerVolumeSpecName "kube-api-access-rqlxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:02.362012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.361976 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqlxv\" (UniqueName: \"kubernetes.io/projected/d9cbadb5-683a-46a2-964a-7c144a467125-kube-api-access-rqlxv\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:02.430522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.430485 5549 generic.go:356] "Generic (PLEG): container finished" podID="d9cbadb5-683a-46a2-964a-7c144a467125" containerID="042e9b81a513769b0d5c3da6f9a4ce20ad79243762b70cb2ed9069cc381179d4" exitCode=0 Nov 02 18:21:02.430713 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.430536 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" Nov 02 18:21:02.430713 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.430569 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" event={"ID":"d9cbadb5-683a-46a2-964a-7c144a467125","Type":"ContainerDied","Data":"042e9b81a513769b0d5c3da6f9a4ce20ad79243762b70cb2ed9069cc381179d4"} Nov 02 18:21:02.430713 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.430635 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75" event={"ID":"d9cbadb5-683a-46a2-964a-7c144a467125","Type":"ContainerDied","Data":"02061ae7934592aa6f52394e92352d4582707a12990dfe9025aad29ac4a0af79"} Nov 02 18:21:02.430713 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.430663 5549 scope.go:122] "RemoveContainer" containerID="042e9b81a513769b0d5c3da6f9a4ce20ad79243762b70cb2ed9069cc381179d4" Nov 02 18:21:02.432270 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.432242 5549 generic.go:356] "Generic (PLEG): container finished" podID="ec19d984-fac8-4ec5-9666-d3c9d3505b11" containerID="cc6d3423bc09a5b4d73f8462a7adbd4aa45509e4715662717d5734a93a664e4e" exitCode=0 Nov 02 18:21:02.432355 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.432304 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" Nov 02 18:21:02.432355 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.432336 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" event={"ID":"ec19d984-fac8-4ec5-9666-d3c9d3505b11","Type":"ContainerDied","Data":"cc6d3423bc09a5b4d73f8462a7adbd4aa45509e4715662717d5734a93a664e4e"} Nov 02 18:21:02.432517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.432375 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx" event={"ID":"ec19d984-fac8-4ec5-9666-d3c9d3505b11","Type":"ContainerDied","Data":"4c902760f1d49b2121b8ead81626e122d78ef9c558dda4f40e8272810cb08c3d"} Nov 02 18:21:02.444189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.444169 5549 scope.go:122] "RemoveContainer" containerID="042e9b81a513769b0d5c3da6f9a4ce20ad79243762b70cb2ed9069cc381179d4" Nov 02 18:21:02.444473 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:21:02.444454 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042e9b81a513769b0d5c3da6f9a4ce20ad79243762b70cb2ed9069cc381179d4\": container with ID starting with 042e9b81a513769b0d5c3da6f9a4ce20ad79243762b70cb2ed9069cc381179d4 not found: ID does not exist" containerID="042e9b81a513769b0d5c3da6f9a4ce20ad79243762b70cb2ed9069cc381179d4" Nov 02 18:21:02.444534 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.444481 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042e9b81a513769b0d5c3da6f9a4ce20ad79243762b70cb2ed9069cc381179d4"} err="failed to get container status \"042e9b81a513769b0d5c3da6f9a4ce20ad79243762b70cb2ed9069cc381179d4\": rpc error: code = NotFound desc = could not find container \"042e9b81a513769b0d5c3da6f9a4ce20ad79243762b70cb2ed9069cc381179d4\": container with ID starting with 042e9b81a513769b0d5c3da6f9a4ce20ad79243762b70cb2ed9069cc381179d4 not found: ID does not exist" Nov 02 18:21:02.444534 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.444498 5549 scope.go:122] "RemoveContainer" containerID="cc6d3423bc09a5b4d73f8462a7adbd4aa45509e4715662717d5734a93a664e4e" Nov 02 18:21:02.456913 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.456890 5549 scope.go:122] "RemoveContainer" containerID="cc6d3423bc09a5b4d73f8462a7adbd4aa45509e4715662717d5734a93a664e4e" Nov 02 18:21:02.457239 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:21:02.457214 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc6d3423bc09a5b4d73f8462a7adbd4aa45509e4715662717d5734a93a664e4e\": container with ID starting with cc6d3423bc09a5b4d73f8462a7adbd4aa45509e4715662717d5734a93a664e4e not found: ID does not exist" containerID="cc6d3423bc09a5b4d73f8462a7adbd4aa45509e4715662717d5734a93a664e4e" Nov 02 18:21:02.457323 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.457245 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc6d3423bc09a5b4d73f8462a7adbd4aa45509e4715662717d5734a93a664e4e"} err="failed to get container status \"cc6d3423bc09a5b4d73f8462a7adbd4aa45509e4715662717d5734a93a664e4e\": rpc error: code = NotFound desc = could not find container \"cc6d3423bc09a5b4d73f8462a7adbd4aa45509e4715662717d5734a93a664e4e\": container with ID starting with cc6d3423bc09a5b4d73f8462a7adbd4aa45509e4715662717d5734a93a664e4e not found: ID does not exist" Nov 02 18:21:02.457569 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.457557 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75"] Nov 02 18:21:02.462230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.462204 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-78383f-56d4ff85c9-g5k75"] Nov 02 18:21:02.471825 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.471749 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx"] Nov 02 18:21:02.477140 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.477114 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-78383f-8559948df9-nj4tx"] Nov 02 18:21:02.819808 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:02.819711 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:03.492855 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:03.492817 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:04.255508 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:04.255476 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:04.328942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:04.328906 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9cbadb5-683a-46a2-964a-7c144a467125" path="/var/lib/kubelet/pods/d9cbadb5-683a-46a2-964a-7c144a467125/volumes" Nov 02 18:21:04.329599 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:04.329580 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec19d984-fac8-4ec5-9666-d3c9d3505b11" path="/var/lib/kubelet/pods/ec19d984-fac8-4ec5-9666-d3c9d3505b11/volumes" Nov 02 18:21:04.948324 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:04.948289 5549 patch_prober.go:29] interesting pod/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:21:04.948495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:04.948353 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:21:05.019251 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:05.019204 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:05.728391 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:05.728359 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:06.141593 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.141563 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:21:06.176933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.176898 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8"] Nov 02 18:21:06.177878 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.177862 5549 cpu_manager.go:414] "containerMap: removing container" podUID="190289c2-9f3a-4201-9cf6-7b38e5edfa86" containerName="main" Nov 02 18:21:06.177878 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.177879 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190289c2-9f3a-4201-9cf6-7b38e5edfa86" containerName="main" Nov 02 18:21:06.178024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.177898 5549 cpu_manager.go:414] "containerMap: removing container" podUID="3c2b7cf4-52fc-4044-ad57-34fb04d241c3" containerName="main" Nov 02 18:21:06.178024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.177904 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c2b7cf4-52fc-4044-ad57-34fb04d241c3" containerName="main" Nov 02 18:21:06.178024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.177914 5549 cpu_manager.go:414] "containerMap: removing container" podUID="ec19d984-fac8-4ec5-9666-d3c9d3505b11" containerName="payload-processing" Nov 02 18:21:06.178024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.177920 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec19d984-fac8-4ec5-9666-d3c9d3505b11" containerName="payload-processing" Nov 02 18:21:06.178024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.177930 5549 cpu_manager.go:414] "containerMap: removing container" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" Nov 02 18:21:06.178024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.177935 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" Nov 02 18:21:06.178024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.177947 5549 cpu_manager.go:414] "containerMap: removing container" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" Nov 02 18:21:06.178024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.177966 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" Nov 02 18:21:06.178024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.178025 5549 cpu_manager.go:414] "containerMap: removing container" podUID="d9cbadb5-683a-46a2-964a-7c144a467125" containerName="payload-pre-processing" Nov 02 18:21:06.178321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.178031 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9cbadb5-683a-46a2-964a-7c144a467125" containerName="payload-pre-processing" Nov 02 18:21:06.178321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.178180 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c2b7cf4-52fc-4044-ad57-34fb04d241c3" containerName="main" Nov 02 18:21:06.178321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.178189 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec19d984-fac8-4ec5-9666-d3c9d3505b11" containerName="payload-processing" Nov 02 18:21:06.178321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.178198 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" Nov 02 18:21:06.178321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.178206 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerName="istio-proxy" Nov 02 18:21:06.178321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.178219 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9cbadb5-683a-46a2-964a-7c144a467125" containerName="payload-pre-processing" Nov 02 18:21:06.178321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.178228 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="190289c2-9f3a-4201-9cf6-7b38e5edfa86" containerName="main" Nov 02 18:21:06.182336 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.182318 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.193683 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.193657 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-63be7a-openshift-default-dockercfg-5274g\"" Nov 02 18:21:06.200298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.200244 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8"] Nov 02 18:21:06.230031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.229999 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-workload-socket\") pod \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " Nov 02 18:21:06.230176 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.230037 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-data\") pod \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " Nov 02 18:21:06.230241 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.230209 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-workload-certs\") pod \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " Nov 02 18:21:06.230291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.230256 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-envoy\") pod \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " Nov 02 18:21:06.230328 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.230304 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-workload-socket" pod "8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" (UID: "8c8e221e-3c26-48d8-9a17-3fd88e6c99ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:06.230365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.230330 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istiod-ca-cert\") pod \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " Nov 02 18:21:06.230418 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.230391 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-data" pod "8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" (UID: "8c8e221e-3c26-48d8-9a17-3fd88e6c99ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:06.230418 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.230408 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-credential-socket\") pod \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " Nov 02 18:21:06.230495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.230450 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-token\" (UniqueName: \"kubernetes.io/projected/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-token\") pod \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " Nov 02 18:21:06.230559 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.230542 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-workload-certs" pod "8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" (UID: "8c8e221e-3c26-48d8-9a17-3fd88e6c99ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:06.230595 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.230548 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-kube-api-access-lfzxb\" (UniqueName: \"kubernetes.io/projected/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-kube-api-access-lfzxb\") pod \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " Nov 02 18:21:06.230635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.230592 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-podinfo\") pod \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\" (UID: \"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef\") " Nov 02 18:21:06.230927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.230893 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-credential-socket" pod "8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" (UID: "8c8e221e-3c26-48d8-9a17-3fd88e6c99ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:06.231125 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.231038 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.231289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.231268 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.231552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.231527 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istiod-ca-cert" pod "8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" (UID: "8c8e221e-3c26-48d8-9a17-3fd88e6c99ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:06.231720 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.231701 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.232012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.231944 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.232192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.232175 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.232426 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.232405 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx9m5\" (UniqueName: \"kubernetes.io/projected/5a3d31ae-cabe-438d-91c5-d5852c562ab3-kube-api-access-lx9m5\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.232859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.232836 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.232990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.232882 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.233172 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.233150 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-envoy" pod "8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" (UID: "8c8e221e-3c26-48d8-9a17-3fd88e6c99ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:06.233172 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.233076 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.233358 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.233338 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:06.233432 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.233397 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:06.233432 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.233416 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:06.233540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.233430 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:06.233540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.233444 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:06.233540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.233461 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:06.233540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.233506 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-kube-api-access-lfzxb" pod "8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" (UID: "8c8e221e-3c26-48d8-9a17-3fd88e6c99ef"). InnerVolumeSpecName "kube-api-access-lfzxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:06.234100 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.234071 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-token" pod "8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" (UID: "8c8e221e-3c26-48d8-9a17-3fd88e6c99ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:06.234304 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.234285 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-podinfo" pod "8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" (UID: "8c8e221e-3c26-48d8-9a17-3fd88e6c99ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:21:06.335213 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.335182 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.335347 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.335232 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx9m5\" (UniqueName: \"kubernetes.io/projected/5a3d31ae-cabe-438d-91c5-d5852c562ab3-kube-api-access-lx9m5\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.335427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.335365 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.335427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.335385 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.335541 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.335488 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.335541 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.335515 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.335659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.335540 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.335659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.335588 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.335659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.335604 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.335659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.335639 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.335848 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.335694 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.335848 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.335746 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:06.335848 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.335766 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfzxb\" (UniqueName: \"kubernetes.io/projected/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-kube-api-access-lfzxb\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:06.335848 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.335781 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:06.336053 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.335926 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.336053 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.336027 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.336209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.336189 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.337835 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.337817 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.337998 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.337979 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.344866 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.344840 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.345064 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.345045 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx9m5\" (UniqueName: \"kubernetes.io/projected/5a3d31ae-cabe-438d-91c5-d5852c562ab3-kube-api-access-lx9m5\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.482439 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.482094 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:06.503263 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.503218 5549 generic.go:356] "Generic (PLEG): container finished" podID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" containerID="a90f923d31d7b3b16c3121acba8214aa159d7185be731660e85b7738ab6e3478" exitCode=0 Nov 02 18:21:06.503493 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.503358 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" event={"ID":"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef","Type":"ContainerDied","Data":"a90f923d31d7b3b16c3121acba8214aa159d7185be731660e85b7738ab6e3478"} Nov 02 18:21:06.503493 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.503408 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" event={"ID":"8c8e221e-3c26-48d8-9a17-3fd88e6c99ef","Type":"ContainerDied","Data":"704ce3c10bbee656b84d71314d0f5edbd12d79308042ebd03bb3cf0a7757397f"} Nov 02 18:21:06.503493 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.503477 5549 scope.go:122] "RemoveContainer" containerID="a90f923d31d7b3b16c3121acba8214aa159d7185be731660e85b7738ab6e3478" Nov 02 18:21:06.503698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.503646 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8" Nov 02 18:21:06.505787 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.505761 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:06.529075 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.528936 5549 scope.go:122] "RemoveContainer" containerID="aac067ee553514a799a2413e2799512f06b8fd446137d39748b46c57248a089b" Nov 02 18:21:06.537910 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.537849 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8"] Nov 02 18:21:06.550787 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.550756 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-78383f-openshift-default-5797f74797-kxhf8"] Nov 02 18:21:06.551249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.551227 5549 scope.go:122] "RemoveContainer" containerID="a90f923d31d7b3b16c3121acba8214aa159d7185be731660e85b7738ab6e3478" Nov 02 18:21:06.551633 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:21:06.551611 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90f923d31d7b3b16c3121acba8214aa159d7185be731660e85b7738ab6e3478\": container with ID starting with a90f923d31d7b3b16c3121acba8214aa159d7185be731660e85b7738ab6e3478 not found: ID does not exist" containerID="a90f923d31d7b3b16c3121acba8214aa159d7185be731660e85b7738ab6e3478" Nov 02 18:21:06.551747 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.551644 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90f923d31d7b3b16c3121acba8214aa159d7185be731660e85b7738ab6e3478"} err="failed to get container status \"a90f923d31d7b3b16c3121acba8214aa159d7185be731660e85b7738ab6e3478\": rpc error: code = NotFound desc = could not find container \"a90f923d31d7b3b16c3121acba8214aa159d7185be731660e85b7738ab6e3478\": container with ID starting with a90f923d31d7b3b16c3121acba8214aa159d7185be731660e85b7738ab6e3478 not found: ID does not exist" Nov 02 18:21:06.551747 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.551670 5549 scope.go:122] "RemoveContainer" containerID="aac067ee553514a799a2413e2799512f06b8fd446137d39748b46c57248a089b" Nov 02 18:21:06.552095 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:21:06.552064 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aac067ee553514a799a2413e2799512f06b8fd446137d39748b46c57248a089b\": container with ID starting with aac067ee553514a799a2413e2799512f06b8fd446137d39748b46c57248a089b not found: ID does not exist" containerID="aac067ee553514a799a2413e2799512f06b8fd446137d39748b46c57248a089b" Nov 02 18:21:06.552225 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.552104 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac067ee553514a799a2413e2799512f06b8fd446137d39748b46c57248a089b"} err="failed to get container status \"aac067ee553514a799a2413e2799512f06b8fd446137d39748b46c57248a089b\": rpc error: code = NotFound desc = could not find container \"aac067ee553514a799a2413e2799512f06b8fd446137d39748b46c57248a089b\": container with ID starting with aac067ee553514a799a2413e2799512f06b8fd446137d39748b46c57248a089b not found: ID does not exist" Nov 02 18:21:06.830390 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:21:06.830359 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3d31ae_cabe_438d_91c5_d5852c562ab3.slice/crio-fd729bd9a876352988315e6bdd04e548ab0408cf7aba1c49042a26960a1e93dc WatchSource:0}: Error finding container fd729bd9a876352988315e6bdd04e548ab0408cf7aba1c49042a26960a1e93dc: Status 404 returned error can't find the container with id fd729bd9a876352988315e6bdd04e548ab0408cf7aba1c49042a26960a1e93dc Nov 02 18:21:06.830738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.830639 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8"] Nov 02 18:21:06.834289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.834255 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:06.834386 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.834323 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:06.834386 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:06.834357 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:07.227099 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:07.227058 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:07.514121 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:07.514022 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" event={"ID":"5a3d31ae-cabe-438d-91c5-d5852c562ab3","Type":"ContainerStarted","Data":"3f8e373ecaf16eedfa395c1fc82cb66d3fc7e4e5d2a737d2ab85b902c7be7412"} Nov 02 18:21:07.514121 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:07.514085 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" event={"ID":"5a3d31ae-cabe-438d-91c5-d5852c562ab3","Type":"ContainerStarted","Data":"fd729bd9a876352988315e6bdd04e548ab0408cf7aba1c49042a26960a1e93dc"} Nov 02 18:21:07.539819 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:07.539768 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" podStartSLOduration=144.539749559 podStartE2EDuration="2m24.539749559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:07.535648509 +0000 UTC m=+2839.324154037" watchObservedRunningTime="2026-11-02 18:21:07.539749559 +0000 UTC m=+2839.328255088" Nov 02 18:21:07.922036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:07.920938 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:08.325641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:08.325608 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c8e221e-3c26-48d8-9a17-3fd88e6c99ef" path="/var/lib/kubelet/pods/8c8e221e-3c26-48d8-9a17-3fd88e6c99ef/volumes" Nov 02 18:21:08.507242 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:08.507192 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:08.508325 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:08.508285 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:08.509432 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:08.509406 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Nov 02 18:21:08.509534 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:08.509451 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" podUID="5a3d31ae-cabe-438d-91c5-d5852c562ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Nov 02 18:21:08.745194 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:08.745154 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr"] Nov 02 18:21:08.759742 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:08.759703 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr"] Nov 02 18:21:08.873317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:08.862135 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:09.007819 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:09.007704 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:09.506271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:09.506225 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Nov 02 18:21:09.506505 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:09.506283 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" podUID="5a3d31ae-cabe-438d-91c5-d5852c562ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Nov 02 18:21:09.730688 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:09.729975 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:10.436520 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:10.436372 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:10.510462 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:10.510418 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Nov 02 18:21:10.510679 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:10.510486 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" podUID="5a3d31ae-cabe-438d-91c5-d5852c562ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Nov 02 18:21:11.118976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:11.117142 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:11.506664 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:11.506619 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Nov 02 18:21:11.507163 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:11.506690 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" podUID="5a3d31ae-cabe-438d-91c5-d5852c562ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Nov 02 18:21:11.822093 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:11.821582 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:12.447979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:12.442813 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:12.523976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:12.523668 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Nov 02 18:21:12.523976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:12.523735 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" podUID="5a3d31ae-cabe-438d-91c5-d5852c562ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Nov 02 18:21:13.295544 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:13.295460 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:13.506490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:13.506441 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Nov 02 18:21:13.506706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:13.506516 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" podUID="5a3d31ae-cabe-438d-91c5-d5852c562ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Nov 02 18:21:13.530828 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:13.530796 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:13.746240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:13.746194 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" podUID="e0cea7d9-9796-4a08-bf44-93038137646d" containerName="payload-processing" containerID="cri-o://ab7704d95d63b9ed548eef8044db6b7e08f2ba3e8660f944c290beb6f987b6e6" gracePeriod=25 Nov 02 18:21:13.761064 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:13.761022 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" podUID="afb8728d-ba4e-4787-ac83-e9bbd591ef54" containerName="payload-pre-processing" containerID="cri-o://90de73083bb22c6de76f897bb3ddd0fba72d90d0e72b3fb96df7e4b908cebd75" gracePeriod=25 Nov 02 18:21:14.124312 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:14.124201 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:14.512114 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:14.512086 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:14.512589 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:14.512569 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:14.513232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:14.513214 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:21:14.821255 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:14.821164 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:14.972966 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:14.972924 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:15.521875 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:15.521833 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:16.142243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:16.142206 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:16.182093 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:16.182052 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" podUID="e0cea7d9-9796-4a08-bf44-93038137646d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:21:16.816395 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:16.816312 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:17.190306 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:17.190212 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" podUID="afb8728d-ba4e-4787-ac83-e9bbd591ef54" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:21:17.527863 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:17.527612 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:17.600842 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:21:17.600795 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:21:18.220259 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:18.220221 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:18.947313 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:18.947213 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:19.710964 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:19.710912 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:20.444075 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:20.444035 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:20.930027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:20.929988 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv"] Nov 02 18:21:20.930831 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:20.930793 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" podUID="f2e000bf-d401-45c8-8450-a0a6fbf2043b" containerName="maas-api" containerID="cri-o://5c18389b452e090484bc23b8c15c285ccbfcef2837f77b1427e18c1605968d43" gracePeriod=30 Nov 02 18:21:21.012599 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.012564 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:21.490213 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.490186 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" Nov 02 18:21:21.524810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.524776 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw"] Nov 02 18:21:21.525853 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.525794 5549 cpu_manager.go:414] "containerMap: removing container" podUID="f2e000bf-d401-45c8-8450-a0a6fbf2043b" containerName="maas-api" Nov 02 18:21:21.525853 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.525813 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2e000bf-d401-45c8-8450-a0a6fbf2043b" containerName="maas-api" Nov 02 18:21:21.526101 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.526007 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2e000bf-d401-45c8-8450-a0a6fbf2043b" containerName="maas-api" Nov 02 18:21:21.530093 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.530072 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.539005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.538981 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw"] Nov 02 18:21:21.620574 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.620541 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2e000bf-d401-45c8-8450-a0a6fbf2043b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e000bf-d401-45c8-8450-a0a6fbf2043b-openshift-service-ca\") pod \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\" (UID: \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\") " Nov 02 18:21:21.620757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.620673 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2e000bf-d401-45c8-8450-a0a6fbf2043b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2e000bf-d401-45c8-8450-a0a6fbf2043b-metrics-certs\") pod \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\" (UID: \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\") " Nov 02 18:21:21.620848 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.620781 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2e000bf-d401-45c8-8450-a0a6fbf2043b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2e000bf-d401-45c8-8450-a0a6fbf2043b-maas-api-tls\") pod \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\" (UID: \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\") " Nov 02 18:21:21.621000 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.620970 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e000bf-d401-45c8-8450-a0a6fbf2043b-openshift-service-ca" pod "f2e000bf-d401-45c8-8450-a0a6fbf2043b" (UID: "f2e000bf-d401-45c8-8450-a0a6fbf2043b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:21.621172 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.620981 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2e000bf-d401-45c8-8450-a0a6fbf2043b-kube-api-access-p98nw\" (UniqueName: \"kubernetes.io/projected/f2e000bf-d401-45c8-8450-a0a6fbf2043b-kube-api-access-p98nw\") pod \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\" (UID: \"f2e000bf-d401-45c8-8450-a0a6fbf2043b\") " Nov 02 18:21:21.622221 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.622195 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e000bf-d401-45c8-8450-a0a6fbf2043b-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:21.623166 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.623144 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e000bf-d401-45c8-8450-a0a6fbf2043b-maas-api-tls" pod "f2e000bf-d401-45c8-8450-a0a6fbf2043b" (UID: "f2e000bf-d401-45c8-8450-a0a6fbf2043b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:21:21.623353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.623330 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e000bf-d401-45c8-8450-a0a6fbf2043b-metrics-certs" pod "f2e000bf-d401-45c8-8450-a0a6fbf2043b" (UID: "f2e000bf-d401-45c8-8450-a0a6fbf2043b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:21:21.623437 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.623407 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e000bf-d401-45c8-8450-a0a6fbf2043b-kube-api-access-p98nw" pod "f2e000bf-d401-45c8-8450-a0a6fbf2043b" (UID: "f2e000bf-d401-45c8-8450-a0a6fbf2043b"). InnerVolumeSpecName "kube-api-access-p98nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:21.665196 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.665163 5549 generic.go:356] "Generic (PLEG): container finished" podID="f2e000bf-d401-45c8-8450-a0a6fbf2043b" containerID="5c18389b452e090484bc23b8c15c285ccbfcef2837f77b1427e18c1605968d43" exitCode=0 Nov 02 18:21:21.665385 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.665205 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" event={"ID":"f2e000bf-d401-45c8-8450-a0a6fbf2043b","Type":"ContainerDied","Data":"5c18389b452e090484bc23b8c15c285ccbfcef2837f77b1427e18c1605968d43"} Nov 02 18:21:21.665385 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.665215 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" Nov 02 18:21:21.665385 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.665247 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv" event={"ID":"f2e000bf-d401-45c8-8450-a0a6fbf2043b","Type":"ContainerDied","Data":"bc17a1104f5f9b3272e22d5e0098fb8950f5fcd8b032710bcbb8ae1ed00fc529"} Nov 02 18:21:21.665385 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.665270 5549 scope.go:122] "RemoveContainer" containerID="5c18389b452e090484bc23b8c15c285ccbfcef2837f77b1427e18c1605968d43" Nov 02 18:21:21.677609 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.677588 5549 scope.go:122] "RemoveContainer" containerID="5c18389b452e090484bc23b8c15c285ccbfcef2837f77b1427e18c1605968d43" Nov 02 18:21:21.677864 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:21:21.677844 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c18389b452e090484bc23b8c15c285ccbfcef2837f77b1427e18c1605968d43\": container with ID starting with 5c18389b452e090484bc23b8c15c285ccbfcef2837f77b1427e18c1605968d43 not found: ID does not exist" containerID="5c18389b452e090484bc23b8c15c285ccbfcef2837f77b1427e18c1605968d43" Nov 02 18:21:21.677920 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.677873 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c18389b452e090484bc23b8c15c285ccbfcef2837f77b1427e18c1605968d43"} err="failed to get container status \"5c18389b452e090484bc23b8c15c285ccbfcef2837f77b1427e18c1605968d43\": rpc error: code = NotFound desc = could not find container \"5c18389b452e090484bc23b8c15c285ccbfcef2837f77b1427e18c1605968d43\": container with ID starting with 5c18389b452e090484bc23b8c15c285ccbfcef2837f77b1427e18c1605968d43 not found: ID does not exist" Nov 02 18:21:21.696603 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.696572 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv"] Nov 02 18:21:21.701180 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.701122 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-658b8c-644849974c-6xnwv"] Nov 02 18:21:21.744675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.736815 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:21.744675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.738599 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-data\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.744675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.738695 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-podinfo\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.744675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.738732 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-envoy\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.744675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.738780 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istiod-ca-cert\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.744675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.738999 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-credential-socket\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.744675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.739046 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-token\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.744675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.739093 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr6h8\" (UniqueName: \"kubernetes.io/projected/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-kube-api-access-zr6h8\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.744675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.739152 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-workload-socket\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.744675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.739207 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-workload-certs\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.744675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.739335 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2e000bf-d401-45c8-8450-a0a6fbf2043b-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:21.744675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.739360 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2e000bf-d401-45c8-8450-a0a6fbf2043b-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:21.744675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.739378 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p98nw\" (UniqueName: \"kubernetes.io/projected/f2e000bf-d401-45c8-8450-a0a6fbf2043b-kube-api-access-p98nw\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:21.840612 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.840581 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-credential-socket\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.840814 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.840629 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-token\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.840814 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.840662 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr6h8\" (UniqueName: \"kubernetes.io/projected/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-kube-api-access-zr6h8\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.840814 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.840698 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-workload-socket\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.840814 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.840735 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-workload-certs\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.841065 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.840895 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-data\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.841065 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.841041 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-podinfo\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.841185 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.841056 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-credential-socket\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.841185 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.841076 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-envoy\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.841185 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.841120 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-workload-certs\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.841185 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.841154 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istiod-ca-cert\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.841412 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.841210 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-workload-socket\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.841412 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.841237 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-data\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.841732 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.841712 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istiod-ca-cert\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.843194 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.843172 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-envoy\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.843568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.843547 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-podinfo\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.851659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.851633 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-token\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.851876 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.851859 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr6h8\" (UniqueName: \"kubernetes.io/projected/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-kube-api-access-zr6h8\") pod \"e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:21.858732 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:21.858712 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:22.157436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:22.157403 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw"] Nov 02 18:21:22.157758 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:21:22.157413 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408a8909_9fa2_43ab_8bd9_1dbc52cefb95.slice/crio-f7150f585c713e7be1ea46d5f198a427f467cfe6a2771dcb37661275576efa1c WatchSource:0}: Error finding container f7150f585c713e7be1ea46d5f198a427f467cfe6a2771dcb37661275576efa1c: Status 404 returned error can't find the container with id f7150f585c713e7be1ea46d5f198a427f467cfe6a2771dcb37661275576efa1c Nov 02 18:21:22.160673 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:22.160632 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:22.160791 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:22.160722 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:22.160791 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:22.160771 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:22.327123 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:22.327090 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2e000bf-d401-45c8-8450-a0a6fbf2043b" path="/var/lib/kubelet/pods/f2e000bf-d401-45c8-8450-a0a6fbf2043b/volumes" Nov 02 18:21:22.416616 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:22.416524 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:22.677446 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:22.677342 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" event={"ID":"408a8909-9fa2-43ab-8bd9-1dbc52cefb95","Type":"ContainerStarted","Data":"3c68bbf2eed35a8261c795b30ce8cf65c0b863d0658d56ec7fc914c40cdbfe01"} Nov 02 18:21:22.677446 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:22.677412 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" event={"ID":"408a8909-9fa2-43ab-8bd9-1dbc52cefb95","Type":"ContainerStarted","Data":"f7150f585c713e7be1ea46d5f198a427f467cfe6a2771dcb37661275576efa1c"} Nov 02 18:21:22.717130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:22.717061 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" podStartSLOduration=8.717032425 podStartE2EDuration="8.717032425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:22.71239378 +0000 UTC m=+2854.500899308" watchObservedRunningTime="2026-11-02 18:21:22.717032425 +0000 UTC m=+2854.505537952" Nov 02 18:21:22.858880 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:22.858849 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:23.172881 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:23.168550 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:23.814456 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:23.813855 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:23.861974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:23.861049 5549 patch_prober.go:29] interesting pod/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 02 18:21:23.861974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:23.861113 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" podUID="408a8909-9fa2-43ab-8bd9-1dbc52cefb95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 02 18:21:24.574618 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:24.574508 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:24.859728 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:24.859643 5549 patch_prober.go:29] interesting pod/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 02 18:21:24.859728 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:24.859706 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" podUID="408a8909-9fa2-43ab-8bd9-1dbc52cefb95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 02 18:21:25.434981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:25.420580 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw"] Nov 02 18:21:25.435607 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:25.435561 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" podUID="75712abb-4a4d-4b51-9b9d-620589c1ffb9" containerName="istio-proxy" containerID="cri-o://75461ba9df4b2743da54a3a84f65e48980a8ebcdac81ec891379c173b685531a" gracePeriod=30 Nov 02 18:21:25.447729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:25.447693 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:25.859739 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:25.859703 5549 patch_prober.go:29] interesting pod/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 02 18:21:25.860174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:25.859757 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" podUID="408a8909-9fa2-43ab-8bd9-1dbc52cefb95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 02 18:21:26.074854 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:26.074816 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5"] Nov 02 18:21:26.075088 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:26.075065 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" podUID="8664bcf8-87ef-4033-a3f8-22db481e826c" containerName="istio-proxy" containerID="cri-o://954283cc3e3e214760277e957f41171668dd0aa0fb98a7b8de87c79b5d15c860" gracePeriod=30 Nov 02 18:21:26.234976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:26.225458 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:26.242342 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:26.242299 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" podUID="e0cea7d9-9796-4a08-bf44-93038137646d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:21:26.859395 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:26.859360 5549 patch_prober.go:29] interesting pod/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 02 18:21:26.859623 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:26.859433 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" podUID="408a8909-9fa2-43ab-8bd9-1dbc52cefb95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 02 18:21:27.018231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:27.016546 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:27.230631 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:27.230592 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms"] Nov 02 18:21:27.239811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:27.238064 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" podUID="afb8728d-ba4e-4787-ac83-e9bbd591ef54" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:21:27.269181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:27.269151 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:27.859983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:27.859934 5549 patch_prober.go:29] interesting pod/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 02 18:21:27.860189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:27.860005 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" podUID="408a8909-9fa2-43ab-8bd9-1dbc52cefb95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 02 18:21:28.062210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:28.062174 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:28.796976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:28.796308 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:28.859168 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:28.859133 5549 patch_prober.go:29] interesting pod/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 02 18:21:28.859347 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:28.859195 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" podUID="408a8909-9fa2-43ab-8bd9-1dbc52cefb95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 02 18:21:29.490976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:29.487854 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:29.865243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:29.865170 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:29.865630 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:29.865600 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:29.866489 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:29.866468 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:21:30.295946 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.295907 5549 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:21:30.296195 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.296020 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" podUID="75712abb-4a4d-4b51-9b9d-620589c1ffb9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:21:30.298971 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.298929 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:30.740431 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.740404 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 18:21:30.772861 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.772827 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9"] Nov 02 18:21:30.773910 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.773891 5549 cpu_manager.go:414] "containerMap: removing container" podUID="75712abb-4a4d-4b51-9b9d-620589c1ffb9" containerName="istio-proxy" Nov 02 18:21:30.773983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.773913 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75712abb-4a4d-4b51-9b9d-620589c1ffb9" containerName="istio-proxy" Nov 02 18:21:30.774143 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.774129 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="75712abb-4a4d-4b51-9b9d-620589c1ffb9" containerName="istio-proxy" Nov 02 18:21:30.778150 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.778132 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.809727 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.807210 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9"] Nov 02 18:21:30.819425 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.819394 5549 generic.go:356] "Generic (PLEG): container finished" podID="75712abb-4a4d-4b51-9b9d-620589c1ffb9" containerID="75461ba9df4b2743da54a3a84f65e48980a8ebcdac81ec891379c173b685531a" exitCode=0 Nov 02 18:21:30.819611 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.819461 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" Nov 02 18:21:30.819611 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.819482 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" event={"ID":"75712abb-4a4d-4b51-9b9d-620589c1ffb9","Type":"ContainerDied","Data":"75461ba9df4b2743da54a3a84f65e48980a8ebcdac81ec891379c173b685531a"} Nov 02 18:21:30.819611 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.819543 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw" event={"ID":"75712abb-4a4d-4b51-9b9d-620589c1ffb9","Type":"ContainerDied","Data":"b1f1621fb811d7b6e151f4000c132125532c81f10625c289dcb47f93ee09ac2b"} Nov 02 18:21:30.819611 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.819576 5549 scope.go:122] "RemoveContainer" containerID="75461ba9df4b2743da54a3a84f65e48980a8ebcdac81ec891379c173b685531a" Nov 02 18:21:30.833103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.833077 5549 scope.go:122] "RemoveContainer" containerID="75461ba9df4b2743da54a3a84f65e48980a8ebcdac81ec891379c173b685531a" Nov 02 18:21:30.833613 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:21:30.833579 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75461ba9df4b2743da54a3a84f65e48980a8ebcdac81ec891379c173b685531a\": container with ID starting with 75461ba9df4b2743da54a3a84f65e48980a8ebcdac81ec891379c173b685531a not found: ID does not exist" containerID="75461ba9df4b2743da54a3a84f65e48980a8ebcdac81ec891379c173b685531a" Nov 02 18:21:30.833729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.833627 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75461ba9df4b2743da54a3a84f65e48980a8ebcdac81ec891379c173b685531a"} err="failed to get container status \"75461ba9df4b2743da54a3a84f65e48980a8ebcdac81ec891379c173b685531a\": rpc error: code = NotFound desc = could not find container \"75461ba9df4b2743da54a3a84f65e48980a8ebcdac81ec891379c173b685531a\": container with ID starting with 75461ba9df4b2743da54a3a84f65e48980a8ebcdac81ec891379c173b685531a not found: ID does not exist" Nov 02 18:21:30.866852 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.866815 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-podinfo\") pod \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " Nov 02 18:21:30.867139 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.867115 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istiod-ca-cert\") pod \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " Nov 02 18:21:30.867242 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.867169 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-data\") pod \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " Nov 02 18:21:30.867307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.867266 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-credential-socket\") pod \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " Nov 02 18:21:30.867406 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.867314 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-token\" (UniqueName: \"kubernetes.io/projected/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-token\") pod \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " Nov 02 18:21:30.867406 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.867362 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-workload-certs\") pod \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " Nov 02 18:21:30.867406 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.867394 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75712abb-4a4d-4b51-9b9d-620589c1ffb9-kube-api-access-ncwx5\" (UniqueName: \"kubernetes.io/projected/75712abb-4a4d-4b51-9b9d-620589c1ffb9-kube-api-access-ncwx5\") pod \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " Nov 02 18:21:30.867593 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.867509 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-workload-socket\") pod \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " Nov 02 18:21:30.867593 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.867552 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-data" pod "75712abb-4a4d-4b51-9b9d-620589c1ffb9" (UID: "75712abb-4a4d-4b51-9b9d-620589c1ffb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:30.867711 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.867606 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-envoy\") pod \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\" (UID: \"75712abb-4a4d-4b51-9b9d-620589c1ffb9\") " Nov 02 18:21:30.867711 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.867634 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istiod-ca-cert" pod "75712abb-4a4d-4b51-9b9d-620589c1ffb9" (UID: "75712abb-4a4d-4b51-9b9d-620589c1ffb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:30.868501 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.868176 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.868501 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.868231 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.868501 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.868318 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.868501 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.868371 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.868501 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.868461 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.868871 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.868693 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.868871 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.868735 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c5644b-6e3c-45b3-83da-f8c89a081c19-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.868871 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.868788 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq29c\" (UniqueName: \"kubernetes.io/projected/41c5644b-6e3c-45b3-83da-f8c89a081c19-kube-api-access-gq29c\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.868871 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.868864 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.869138 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.869090 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:30.869138 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.869112 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:30.873977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.869370 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-workload-socket" pod "75712abb-4a4d-4b51-9b9d-620589c1ffb9" (UID: "75712abb-4a4d-4b51-9b9d-620589c1ffb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:30.873977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.869636 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-credential-socket" pod "75712abb-4a4d-4b51-9b9d-620589c1ffb9" (UID: "75712abb-4a4d-4b51-9b9d-620589c1ffb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:30.873977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.870542 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-podinfo" pod "75712abb-4a4d-4b51-9b9d-620589c1ffb9" (UID: "75712abb-4a4d-4b51-9b9d-620589c1ffb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:21:30.873977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.872888 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75712abb-4a4d-4b51-9b9d-620589c1ffb9-kube-api-access-ncwx5" pod "75712abb-4a4d-4b51-9b9d-620589c1ffb9" (UID: "75712abb-4a4d-4b51-9b9d-620589c1ffb9"). InnerVolumeSpecName "kube-api-access-ncwx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:30.873977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.873063 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-workload-certs" pod "75712abb-4a4d-4b51-9b9d-620589c1ffb9" (UID: "75712abb-4a4d-4b51-9b9d-620589c1ffb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:30.874327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.874232 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-envoy" pod "75712abb-4a4d-4b51-9b9d-620589c1ffb9" (UID: "75712abb-4a4d-4b51-9b9d-620589c1ffb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:30.878477 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.878439 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-token" pod "75712abb-4a4d-4b51-9b9d-620589c1ffb9" (UID: "75712abb-4a4d-4b51-9b9d-620589c1ffb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:30.918810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.918780 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:30.971127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.971090 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.971127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.971134 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c5644b-6e3c-45b3-83da-f8c89a081c19-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.971442 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.971165 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq29c\" (UniqueName: \"kubernetes.io/projected/41c5644b-6e3c-45b3-83da-f8c89a081c19-kube-api-access-gq29c\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.971442 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.971235 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.971442 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.971408 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.971635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.971451 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.971635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.971493 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.971635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.971534 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.971635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.971608 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.971879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.971644 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.971879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.971730 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.971879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.971785 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.972127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.971946 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c5644b-6e3c-45b3-83da-f8c89a081c19-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.972127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.972008 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:30.972127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.972029 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:30.972127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.972045 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:30.972127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.972060 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncwx5\" (UniqueName: \"kubernetes.io/projected/75712abb-4a4d-4b51-9b9d-620589c1ffb9-kube-api-access-ncwx5\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:30.972127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.972076 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:30.972127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.972094 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:30.972127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.972108 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75712abb-4a4d-4b51-9b9d-620589c1ffb9-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:30.972427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.972198 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.973946 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.973928 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.974053 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.974036 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.979649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.979625 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:30.979873 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:30.979852 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq29c\" (UniqueName: \"kubernetes.io/projected/41c5644b-6e3c-45b3-83da-f8c89a081c19-kube-api-access-gq29c\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:31.115714 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.115678 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:31.159081 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.159045 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw"] Nov 02 18:21:31.164430 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.164395 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-hkjcw"] Nov 02 18:21:31.220850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.220808 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:31.372040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.371861 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:31.405366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.405339 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 18:21:31.413870 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:21:31.413834 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c5644b_6e3c_45b3_83da_f8c89a081c19.slice/crio-125fb1f413e41e3eb9d7e7c264a916164a693f674b4207ab066daa1b678362ea WatchSource:0}: Error finding container 125fb1f413e41e3eb9d7e7c264a916164a693f674b4207ab066daa1b678362ea: Status 404 returned error can't find the container with id 125fb1f413e41e3eb9d7e7c264a916164a693f674b4207ab066daa1b678362ea Nov 02 18:21:31.415123 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.415094 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9"] Nov 02 18:21:31.420149 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.419645 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:31.420149 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.419735 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:31.420149 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.419781 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:31.450337 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.450301 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5"] Nov 02 18:21:31.452047 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.452002 5549 cpu_manager.go:414] "containerMap: removing container" podUID="8664bcf8-87ef-4033-a3f8-22db481e826c" containerName="istio-proxy" Nov 02 18:21:31.452047 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.452028 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8664bcf8-87ef-4033-a3f8-22db481e826c" containerName="istio-proxy" Nov 02 18:21:31.453335 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.453311 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8664bcf8-87ef-4033-a3f8-22db481e826c" containerName="istio-proxy" Nov 02 18:21:31.471205 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.471174 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.480348 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.480309 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5"] Nov 02 18:21:31.482558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.482522 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8664bcf8-87ef-4033-a3f8-22db481e826c-kube-api-access-6425q\" (UniqueName: \"kubernetes.io/projected/8664bcf8-87ef-4033-a3f8-22db481e826c-kube-api-access-6425q\") pod \"8664bcf8-87ef-4033-a3f8-22db481e826c\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " Nov 02 18:21:31.482687 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.482600 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-credential-socket\") pod \"8664bcf8-87ef-4033-a3f8-22db481e826c\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " Nov 02 18:21:31.482750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.482726 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-workload-certs\") pod \"8664bcf8-87ef-4033-a3f8-22db481e826c\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " Nov 02 18:21:31.482823 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.482752 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-envoy\") pod \"8664bcf8-87ef-4033-a3f8-22db481e826c\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " Nov 02 18:21:31.482888 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.482822 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-data\") pod \"8664bcf8-87ef-4033-a3f8-22db481e826c\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " Nov 02 18:21:31.482969 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.482932 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-workload-socket\") pod \"8664bcf8-87ef-4033-a3f8-22db481e826c\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " Nov 02 18:21:31.483033 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.482985 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8664bcf8-87ef-4033-a3f8-22db481e826c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8664bcf8-87ef-4033-a3f8-22db481e826c-istiod-ca-cert\") pod \"8664bcf8-87ef-4033-a3f8-22db481e826c\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " Nov 02 18:21:31.483033 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.483022 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-podinfo\") pod \"8664bcf8-87ef-4033-a3f8-22db481e826c\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " Nov 02 18:21:31.483156 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.483103 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-token\" (UniqueName: \"kubernetes.io/projected/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-token\") pod \"8664bcf8-87ef-4033-a3f8-22db481e826c\" (UID: \"8664bcf8-87ef-4033-a3f8-22db481e826c\") " Nov 02 18:21:31.483628 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.483596 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-workload-socket" pod "8664bcf8-87ef-4033-a3f8-22db481e826c" (UID: "8664bcf8-87ef-4033-a3f8-22db481e826c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:31.484812 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.484782 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-credential-socket" pod "8664bcf8-87ef-4033-a3f8-22db481e826c" (UID: "8664bcf8-87ef-4033-a3f8-22db481e826c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:31.485350 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.485322 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-data" pod "8664bcf8-87ef-4033-a3f8-22db481e826c" (UID: "8664bcf8-87ef-4033-a3f8-22db481e826c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:31.485457 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.485375 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:31.485457 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.485401 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:31.485828 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.485770 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8664bcf8-87ef-4033-a3f8-22db481e826c-istiod-ca-cert" pod "8664bcf8-87ef-4033-a3f8-22db481e826c" (UID: "8664bcf8-87ef-4033-a3f8-22db481e826c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:31.486075 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.485987 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-workload-certs" pod "8664bcf8-87ef-4033-a3f8-22db481e826c" (UID: "8664bcf8-87ef-4033-a3f8-22db481e826c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:31.489307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.489275 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8664bcf8-87ef-4033-a3f8-22db481e826c-kube-api-access-6425q" pod "8664bcf8-87ef-4033-a3f8-22db481e826c" (UID: "8664bcf8-87ef-4033-a3f8-22db481e826c"). InnerVolumeSpecName "kube-api-access-6425q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:31.492356 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.492329 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-podinfo" pod "8664bcf8-87ef-4033-a3f8-22db481e826c" (UID: "8664bcf8-87ef-4033-a3f8-22db481e826c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:21:31.494118 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.494091 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-envoy" pod "8664bcf8-87ef-4033-a3f8-22db481e826c" (UID: "8664bcf8-87ef-4033-a3f8-22db481e826c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:31.498454 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.498396 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-token" pod "8664bcf8-87ef-4033-a3f8-22db481e826c" (UID: "8664bcf8-87ef-4033-a3f8-22db481e826c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:31.587722 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.587676 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.587818 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.587763 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.587894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.587874 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752af71e-0e0d-48b9-9555-9ce325e913ce-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.588007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.587923 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.588099 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.588009 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.588099 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.588027 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7prdx\" (UniqueName: \"kubernetes.io/projected/752af71e-0e0d-48b9-9555-9ce325e913ce-kube-api-access-7prdx\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.588099 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.588076 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.588274 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.588129 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.588274 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.588192 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.588382 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.588349 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6425q\" (UniqueName: \"kubernetes.io/projected/8664bcf8-87ef-4033-a3f8-22db481e826c-kube-api-access-6425q\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:31.588382 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.588360 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:31.588382 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.588370 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:31.588382 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.588378 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:31.588587 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.588386 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8664bcf8-87ef-4033-a3f8-22db481e826c-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:31.588587 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.588394 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:31.588587 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.588402 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8664bcf8-87ef-4033-a3f8-22db481e826c-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:31.689836 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.689748 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.689836 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.689802 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.690120 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.689909 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752af71e-0e0d-48b9-9555-9ce325e913ce-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.690120 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.689975 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.690120 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.690079 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.690120 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.690104 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7prdx\" (UniqueName: \"kubernetes.io/projected/752af71e-0e0d-48b9-9555-9ce325e913ce-kube-api-access-7prdx\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.690353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.690165 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.690353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.690232 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.690353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.690239 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.690621 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.690590 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.690733 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.690691 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.690799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.690755 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752af71e-0e0d-48b9-9555-9ce325e913ce-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.690861 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.690795 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.691211 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.691173 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.692823 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.692785 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.693009 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.692987 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.700080 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.700056 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.704202 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.703981 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7prdx\" (UniqueName: \"kubernetes.io/projected/752af71e-0e0d-48b9-9555-9ce325e913ce-kube-api-access-7prdx\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.832789 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.832748 5549 generic.go:356] "Generic (PLEG): container finished" podID="8664bcf8-87ef-4033-a3f8-22db481e826c" containerID="954283cc3e3e214760277e957f41171668dd0aa0fb98a7b8de87c79b5d15c860" exitCode=0 Nov 02 18:21:31.832789 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.832798 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" Nov 02 18:21:31.833404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.832828 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" event={"ID":"8664bcf8-87ef-4033-a3f8-22db481e826c","Type":"ContainerDied","Data":"954283cc3e3e214760277e957f41171668dd0aa0fb98a7b8de87c79b5d15c860"} Nov 02 18:21:31.833404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.832906 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5" event={"ID":"8664bcf8-87ef-4033-a3f8-22db481e826c","Type":"ContainerDied","Data":"9675e3690c9bb27d32fffb5f2e1747dd63a75bf592b8de00075b398226040378"} Nov 02 18:21:31.833404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.832935 5549 scope.go:122] "RemoveContainer" containerID="954283cc3e3e214760277e957f41171668dd0aa0fb98a7b8de87c79b5d15c860" Nov 02 18:21:31.839725 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.839695 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" event={"ID":"41c5644b-6e3c-45b3-83da-f8c89a081c19","Type":"ContainerStarted","Data":"9ca961c54ee96766071f62b9bf92d1d53ecbc9596bdc5563145b66f04d2cc8c2"} Nov 02 18:21:31.839868 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.839754 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" event={"ID":"41c5644b-6e3c-45b3-83da-f8c89a081c19","Type":"ContainerStarted","Data":"125fb1f413e41e3eb9d7e7c264a916164a693f674b4207ab066daa1b678362ea"} Nov 02 18:21:31.851065 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.851036 5549 scope.go:122] "RemoveContainer" containerID="954283cc3e3e214760277e957f41171668dd0aa0fb98a7b8de87c79b5d15c860" Nov 02 18:21:31.851408 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:21:31.851376 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"954283cc3e3e214760277e957f41171668dd0aa0fb98a7b8de87c79b5d15c860\": container with ID starting with 954283cc3e3e214760277e957f41171668dd0aa0fb98a7b8de87c79b5d15c860 not found: ID does not exist" containerID="954283cc3e3e214760277e957f41171668dd0aa0fb98a7b8de87c79b5d15c860" Nov 02 18:21:31.851475 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.851420 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"954283cc3e3e214760277e957f41171668dd0aa0fb98a7b8de87c79b5d15c860"} err="failed to get container status \"954283cc3e3e214760277e957f41171668dd0aa0fb98a7b8de87c79b5d15c860\": rpc error: code = NotFound desc = could not find container \"954283cc3e3e214760277e957f41171668dd0aa0fb98a7b8de87c79b5d15c860\": container with ID starting with 954283cc3e3e214760277e957f41171668dd0aa0fb98a7b8de87c79b5d15c860 not found: ID does not exist" Nov 02 18:21:31.871500 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.871447 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" podStartSLOduration=168.87143416 podStartE2EDuration="2m48.87143416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:31.870383277 +0000 UTC m=+2863.658888798" watchObservedRunningTime="2026-11-02 18:21:31.87143416 +0000 UTC m=+2863.659939678" Nov 02 18:21:31.897043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.897011 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5"] Nov 02 18:21:31.897197 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.897181 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:31.899703 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.899679 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-h8bt5"] Nov 02 18:21:31.948171 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:31.948134 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:32.099093 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:32.099059 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:32.115846 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:32.115805 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:32.117268 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:32.117238 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body= Nov 02 18:21:32.117412 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:32.117291 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" podUID="41c5644b-6e3c-45b3-83da-f8c89a081c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" Nov 02 18:21:32.255083 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:21:32.255043 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752af71e_0e0d_48b9_9555_9ce325e913ce.slice/crio-23abc1ec3b1566bc768ee466709ef801950e4c9020550a327ff4c850d9b2632d WatchSource:0}: Error finding container 23abc1ec3b1566bc768ee466709ef801950e4c9020550a327ff4c850d9b2632d: Status 404 returned error can't find the container with id 23abc1ec3b1566bc768ee466709ef801950e4c9020550a327ff4c850d9b2632d Nov 02 18:21:32.256209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:32.256181 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5"] Nov 02 18:21:32.258513 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:32.258475 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:32.258628 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:32.258594 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:32.258695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:32.258670 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:32.280204 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:32.280182 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:32.699141 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:32.699104 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75712abb-4a4d-4b51-9b9d-620589c1ffb9" path="/var/lib/kubelet/pods/75712abb-4a4d-4b51-9b9d-620589c1ffb9/volumes" Nov 02 18:21:32.701333 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:32.701303 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8664bcf8-87ef-4033-a3f8-22db481e826c" path="/var/lib/kubelet/pods/8664bcf8-87ef-4033-a3f8-22db481e826c/volumes" Nov 02 18:21:32.853289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:32.853196 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" event={"ID":"752af71e-0e0d-48b9-9555-9ce325e913ce","Type":"ContainerStarted","Data":"563f0c0517aeeeb1bfc22c321770cf482d3c5c899aa76b90faf19e8f049ee3fa"} Nov 02 18:21:32.853289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:32.853264 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" event={"ID":"752af71e-0e0d-48b9-9555-9ce325e913ce","Type":"ContainerStarted","Data":"23abc1ec3b1566bc768ee466709ef801950e4c9020550a327ff4c850d9b2632d"} Nov 02 18:21:32.898972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:32.890463 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" podStartSLOduration=109.890440061 podStartE2EDuration="1m49.890440061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:32.876465147 +0000 UTC m=+2864.664970674" watchObservedRunningTime="2026-11-02 18:21:32.890440061 +0000 UTC m=+2864.678945588" Nov 02 18:21:32.898972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:32.898064 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:32.966806 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:32.966764 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:33.118508 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:33.117825 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body= Nov 02 18:21:33.118508 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:33.117897 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" podUID="41c5644b-6e3c-45b3-83da-f8c89a081c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" Nov 02 18:21:33.155190 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:33.152513 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:33.302976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:33.295379 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:33.662242 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:33.659755 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg"] Nov 02 18:21:33.662242 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:33.660046 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" podUID="0c3b2112-be30-40ae-bc4b-84d75341a926" containerName="istio-proxy" containerID="cri-o://7875f01f9f135719a6dba7c099fdce17c752085ffa2ee3359c95775fa69edae7" gracePeriod=30 Nov 02 18:21:33.907209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:33.900698 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:33.910480 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:33.910446 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Nov 02 18:21:33.922032 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:33.919271 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" podUID="752af71e-0e0d-48b9-9555-9ce325e913ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Nov 02 18:21:34.137974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.136432 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body= Nov 02 18:21:34.137974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.136493 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" podUID="41c5644b-6e3c-45b3-83da-f8c89a081c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" Nov 02 18:21:34.365278 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.365243 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" Nov 02 18:21:34.375715 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.375678 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0cea7d9-9796-4a08-bf44-93038137646d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0cea7d9-9796-4a08-bf44-93038137646d-plugins-config-volume\") pod \"e0cea7d9-9796-4a08-bf44-93038137646d\" (UID: \"e0cea7d9-9796-4a08-bf44-93038137646d\") " Nov 02 18:21:34.375997 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.375971 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0cea7d9-9796-4a08-bf44-93038137646d-kube-api-access-mnddj\" (UniqueName: \"kubernetes.io/projected/e0cea7d9-9796-4a08-bf44-93038137646d-kube-api-access-mnddj\") pod \"e0cea7d9-9796-4a08-bf44-93038137646d\" (UID: \"e0cea7d9-9796-4a08-bf44-93038137646d\") " Nov 02 18:21:34.377312 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.377281 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0cea7d9-9796-4a08-bf44-93038137646d-plugins-config-volume" pod "e0cea7d9-9796-4a08-bf44-93038137646d" (UID: "e0cea7d9-9796-4a08-bf44-93038137646d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:34.377612 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.377589 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0cea7d9-9796-4a08-bf44-93038137646d-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:34.546978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.546854 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0cea7d9-9796-4a08-bf44-93038137646d-kube-api-access-mnddj" pod "e0cea7d9-9796-4a08-bf44-93038137646d" (UID: "e0cea7d9-9796-4a08-bf44-93038137646d"). InnerVolumeSpecName "kube-api-access-mnddj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:34.663170 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.661296 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnddj\" (UniqueName: \"kubernetes.io/projected/e0cea7d9-9796-4a08-bf44-93038137646d-kube-api-access-mnddj\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:34.673597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.668683 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx"] Nov 02 18:21:34.673597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.670259 5549 cpu_manager.go:414] "containerMap: removing container" podUID="e0cea7d9-9796-4a08-bf44-93038137646d" containerName="payload-processing" Nov 02 18:21:34.673597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.670280 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0cea7d9-9796-4a08-bf44-93038137646d" containerName="payload-processing" Nov 02 18:21:34.673597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.670495 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0cea7d9-9796-4a08-bf44-93038137646d" containerName="payload-processing" Nov 02 18:21:34.673597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.673019 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" Nov 02 18:21:34.677517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.677490 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx"] Nov 02 18:21:34.677667 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.677604 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.763761 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.763725 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb8728d-ba4e-4787-ac83-e9bbd591ef54-kube-api-access-zstss\" (UniqueName: \"kubernetes.io/projected/afb8728d-ba4e-4787-ac83-e9bbd591ef54-kube-api-access-zstss\") pod \"afb8728d-ba4e-4787-ac83-e9bbd591ef54\" (UID: \"afb8728d-ba4e-4787-ac83-e9bbd591ef54\") " Nov 02 18:21:34.763930 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.763800 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afb8728d-ba4e-4787-ac83-e9bbd591ef54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afb8728d-ba4e-4787-ac83-e9bbd591ef54-plugins-config-volume\") pod \"afb8728d-ba4e-4787-ac83-e9bbd591ef54\" (UID: \"afb8728d-ba4e-4787-ac83-e9bbd591ef54\") " Nov 02 18:21:34.764529 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.764502 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb8728d-ba4e-4787-ac83-e9bbd591ef54-plugins-config-volume" pod "afb8728d-ba4e-4787-ac83-e9bbd591ef54" (UID: "afb8728d-ba4e-4787-ac83-e9bbd591ef54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:34.764696 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.764658 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.764794 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.764716 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8998b9b1-c956-4ffb-9274-9eb5443894f9-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.764794 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.764754 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.764909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.764858 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.765001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.764979 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.765117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.765102 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.765307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.765287 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6j5f\" (UniqueName: \"kubernetes.io/projected/8998b9b1-c956-4ffb-9274-9eb5443894f9-kube-api-access-k6j5f\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.765520 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.765490 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.765583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.765543 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.765817 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.765798 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afb8728d-ba4e-4787-ac83-e9bbd591ef54-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:34.766962 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.766923 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb8728d-ba4e-4787-ac83-e9bbd591ef54-kube-api-access-zstss" pod "afb8728d-ba4e-4787-ac83-e9bbd591ef54" (UID: "afb8728d-ba4e-4787-ac83-e9bbd591ef54"). InnerVolumeSpecName "kube-api-access-zstss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:34.887030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.868495 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.941366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.921103 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8998b9b1-c956-4ffb-9274-9eb5443894f9-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.941366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.921175 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.941366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.921299 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.941366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.921413 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.941366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.921529 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.941366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.921574 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6j5f\" (UniqueName: \"kubernetes.io/projected/8998b9b1-c956-4ffb-9274-9eb5443894f9-kube-api-access-k6j5f\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.941366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.921665 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.941366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.921704 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.941366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.921928 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zstss\" (UniqueName: \"kubernetes.io/projected/afb8728d-ba4e-4787-ac83-e9bbd591ef54-kube-api-access-zstss\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:34.941366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.924745 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.941366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.926083 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8998b9b1-c956-4ffb-9274-9eb5443894f9-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.941366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.926265 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.941366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.927154 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.941366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.927923 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.941366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.929383 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Nov 02 18:21:34.941366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.929426 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" podUID="752af71e-0e0d-48b9-9555-9ce325e913ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Nov 02 18:21:34.941366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.930428 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:34.941366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.939590 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.964090 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.945633 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.964090 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.950063 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:34.970406 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:34.964999 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6j5f\" (UniqueName: \"kubernetes.io/projected/8998b9b1-c956-4ffb-9274-9eb5443894f9-kube-api-access-k6j5f\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:35.041985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.041921 5549 generic.go:356] "Generic (PLEG): container finished" podID="afb8728d-ba4e-4787-ac83-e9bbd591ef54" containerID="90de73083bb22c6de76f897bb3ddd0fba72d90d0e72b3fb96df7e4b908cebd75" exitCode=0 Nov 02 18:21:35.042188 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.042113 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" event={"ID":"afb8728d-ba4e-4787-ac83-e9bbd591ef54","Type":"ContainerDied","Data":"90de73083bb22c6de76f897bb3ddd0fba72d90d0e72b3fb96df7e4b908cebd75"} Nov 02 18:21:35.042188 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.042154 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" event={"ID":"afb8728d-ba4e-4787-ac83-e9bbd591ef54","Type":"ContainerDied","Data":"9e30c4e648f42f0af367c96a0ded4f3a3f8f166ea9db26c794506110a68ad6b5"} Nov 02 18:21:35.042188 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.042174 5549 scope.go:122] "RemoveContainer" containerID="90de73083bb22c6de76f897bb3ddd0fba72d90d0e72b3fb96df7e4b908cebd75" Nov 02 18:21:35.042343 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.042278 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr" Nov 02 18:21:35.048847 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.047796 5549 generic.go:356] "Generic (PLEG): container finished" podID="e0cea7d9-9796-4a08-bf44-93038137646d" containerID="ab7704d95d63b9ed548eef8044db6b7e08f2ba3e8660f944c290beb6f987b6e6" exitCode=0 Nov 02 18:21:35.048847 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.047845 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" Nov 02 18:21:35.048847 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.047885 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" event={"ID":"e0cea7d9-9796-4a08-bf44-93038137646d","Type":"ContainerDied","Data":"ab7704d95d63b9ed548eef8044db6b7e08f2ba3e8660f944c290beb6f987b6e6"} Nov 02 18:21:35.048847 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.047963 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr" event={"ID":"e0cea7d9-9796-4a08-bf44-93038137646d","Type":"ContainerDied","Data":"cd75d4e1cef3feb47b350ab01095d3eec57070de667ce8275cf724b61f9a3d77"} Nov 02 18:21:35.053846 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.053820 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:35.063060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.063041 5549 scope.go:122] "RemoveContainer" containerID="90de73083bb22c6de76f897bb3ddd0fba72d90d0e72b3fb96df7e4b908cebd75" Nov 02 18:21:35.063352 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:21:35.063332 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90de73083bb22c6de76f897bb3ddd0fba72d90d0e72b3fb96df7e4b908cebd75\": container with ID starting with 90de73083bb22c6de76f897bb3ddd0fba72d90d0e72b3fb96df7e4b908cebd75 not found: ID does not exist" containerID="90de73083bb22c6de76f897bb3ddd0fba72d90d0e72b3fb96df7e4b908cebd75" Nov 02 18:21:35.063421 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.063358 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90de73083bb22c6de76f897bb3ddd0fba72d90d0e72b3fb96df7e4b908cebd75"} err="failed to get container status \"90de73083bb22c6de76f897bb3ddd0fba72d90d0e72b3fb96df7e4b908cebd75\": rpc error: code = NotFound desc = could not find container \"90de73083bb22c6de76f897bb3ddd0fba72d90d0e72b3fb96df7e4b908cebd75\": container with ID starting with 90de73083bb22c6de76f897bb3ddd0fba72d90d0e72b3fb96df7e4b908cebd75 not found: ID does not exist" Nov 02 18:21:35.063421 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.063378 5549 scope.go:122] "RemoveContainer" containerID="ab7704d95d63b9ed548eef8044db6b7e08f2ba3e8660f944c290beb6f987b6e6" Nov 02 18:21:35.074013 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.073984 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr"] Nov 02 18:21:35.106005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.103487 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-658b8c-6c7cdc8cf5-lbdsr"] Nov 02 18:21:35.130976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.130463 5549 scope.go:122] "RemoveContainer" containerID="ab7704d95d63b9ed548eef8044db6b7e08f2ba3e8660f944c290beb6f987b6e6" Nov 02 18:21:35.131201 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.131084 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body= Nov 02 18:21:35.131201 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.131133 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" podUID="41c5644b-6e3c-45b3-83da-f8c89a081c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" Nov 02 18:21:35.135466 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:21:35.135421 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7704d95d63b9ed548eef8044db6b7e08f2ba3e8660f944c290beb6f987b6e6\": container with ID starting with ab7704d95d63b9ed548eef8044db6b7e08f2ba3e8660f944c290beb6f987b6e6 not found: ID does not exist" containerID="ab7704d95d63b9ed548eef8044db6b7e08f2ba3e8660f944c290beb6f987b6e6" Nov 02 18:21:35.135648 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.135467 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7704d95d63b9ed548eef8044db6b7e08f2ba3e8660f944c290beb6f987b6e6"} err="failed to get container status \"ab7704d95d63b9ed548eef8044db6b7e08f2ba3e8660f944c290beb6f987b6e6\": rpc error: code = NotFound desc = could not find container \"ab7704d95d63b9ed548eef8044db6b7e08f2ba3e8660f944c290beb6f987b6e6\": container with ID starting with ab7704d95d63b9ed548eef8044db6b7e08f2ba3e8660f944c290beb6f987b6e6 not found: ID does not exist" Nov 02 18:21:35.142241 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.142165 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr"] Nov 02 18:21:35.148787 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.148736 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-658b8c-8f4c6c479-tjqpr"] Nov 02 18:21:35.435332 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.435249 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx"] Nov 02 18:21:35.435332 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:21:35.435286 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8998b9b1_c956_4ffb_9274_9eb5443894f9.slice/crio-efe3bb009d6604629419e814b7c58ec51896ca01ae7e7a9a1c0c0d9120177c3a WatchSource:0}: Error finding container efe3bb009d6604629419e814b7c58ec51896ca01ae7e7a9a1c0c0d9120177c3a: Status 404 returned error can't find the container with id efe3bb009d6604629419e814b7c58ec51896ca01ae7e7a9a1c0c0d9120177c3a Nov 02 18:21:35.438536 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.438503 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:35.438659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.438571 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:35.438659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.438603 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:35.647155 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.642605 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:35.802061 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.799569 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:35.919976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.919877 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Nov 02 18:21:35.919976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:35.919937 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" podUID="752af71e-0e0d-48b9-9555-9ce325e913ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Nov 02 18:21:36.129035 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:36.127980 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body= Nov 02 18:21:36.147010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:36.138045 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" podUID="41c5644b-6e3c-45b3-83da-f8c89a081c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" Nov 02 18:21:36.156153 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:36.156116 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" event={"ID":"8998b9b1-c956-4ffb-9274-9eb5443894f9","Type":"ContainerStarted","Data":"4eb20ff7a517e4db37b943aa9e483440758a7c6d7e2e848145a3a5f52f73f22b"} Nov 02 18:21:36.156310 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:36.156181 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" event={"ID":"8998b9b1-c956-4ffb-9274-9eb5443894f9","Type":"ContainerStarted","Data":"efe3bb009d6604629419e814b7c58ec51896ca01ae7e7a9a1c0c0d9120177c3a"} Nov 02 18:21:36.218005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:36.201640 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" podStartSLOduration=113.201599612 podStartE2EDuration="1m53.201599612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:36.177328108 +0000 UTC m=+2867.965833636" watchObservedRunningTime="2026-11-02 18:21:36.201599612 +0000 UTC m=+2867.990105143" Nov 02 18:21:36.365089 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:36.365046 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb8728d-ba4e-4787-ac83-e9bbd591ef54" path="/var/lib/kubelet/pods/afb8728d-ba4e-4787-ac83-e9bbd591ef54/volumes" Nov 02 18:21:36.365612 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:36.365594 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0cea7d9-9796-4a08-bf44-93038137646d" path="/var/lib/kubelet/pods/e0cea7d9-9796-4a08-bf44-93038137646d/volumes" Nov 02 18:21:36.701977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:36.700239 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:36.960491 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:36.949330 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:36.960491 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:36.950683 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Nov 02 18:21:36.960491 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:36.956140 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" podUID="752af71e-0e0d-48b9-9555-9ce325e913ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Nov 02 18:21:36.973799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:36.973762 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd"] Nov 02 18:21:36.974870 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:36.974844 5549 cpu_manager.go:414] "containerMap: removing container" podUID="afb8728d-ba4e-4787-ac83-e9bbd591ef54" containerName="payload-pre-processing" Nov 02 18:21:36.974870 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:36.974869 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb8728d-ba4e-4787-ac83-e9bbd591ef54" containerName="payload-pre-processing" Nov 02 18:21:36.975151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:36.975139 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb8728d-ba4e-4787-ac83-e9bbd591ef54" containerName="payload-pre-processing" Nov 02 18:21:37.055211 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.041660 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" Nov 02 18:21:37.055211 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.046935 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd"] Nov 02 18:21:37.070055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.057143 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:37.070055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.057586 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body= Nov 02 18:21:37.070055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.057649 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" podUID="8998b9b1-c956-4ffb-9274-9eb5443894f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" Nov 02 18:21:37.070055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.061377 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b6e563-a46e-42e2-bc91-5a0277272b56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd\" (UID: \"f2b6e563-a46e-42e2-bc91-5a0277272b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" Nov 02 18:21:37.070055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.061782 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6flmd\" (UniqueName: \"kubernetes.io/projected/f2b6e563-a46e-42e2-bc91-5a0277272b56-kube-api-access-6flmd\") pod \"payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd\" (UID: \"f2b6e563-a46e-42e2-bc91-5a0277272b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" Nov 02 18:21:37.188243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.185277 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b6e563-a46e-42e2-bc91-5a0277272b56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd\" (UID: \"f2b6e563-a46e-42e2-bc91-5a0277272b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" Nov 02 18:21:37.188243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.185600 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6flmd\" (UniqueName: \"kubernetes.io/projected/f2b6e563-a46e-42e2-bc91-5a0277272b56-kube-api-access-6flmd\") pod \"payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd\" (UID: \"f2b6e563-a46e-42e2-bc91-5a0277272b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" Nov 02 18:21:37.214031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.207368 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body= Nov 02 18:21:37.214031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.207436 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" podUID="41c5644b-6e3c-45b3-83da-f8c89a081c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" Nov 02 18:21:37.214031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.208054 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2966e7\"" Nov 02 18:21:37.214031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.208219 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2966e7-dockercfg-x5spn\"" Nov 02 18:21:37.235982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.215927 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b6e563-a46e-42e2-bc91-5a0277272b56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd\" (UID: \"f2b6e563-a46e-42e2-bc91-5a0277272b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" Nov 02 18:21:37.320977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.318924 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6flmd\" (UniqueName: \"kubernetes.io/projected/f2b6e563-a46e-42e2-bc91-5a0277272b56-kube-api-access-6flmd\") pod \"payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd\" (UID: \"f2b6e563-a46e-42e2-bc91-5a0277272b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" Nov 02 18:21:37.424977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.424059 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:37.424977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.424798 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" Nov 02 18:21:37.679860 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.671020 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx"] Nov 02 18:21:37.679860 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.671551 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" podUID="b20603e9-d7ea-4bac-bf54-c1c59d343fa8" containerName="istio-proxy" containerID="cri-o://56414aed1f8107254cf65658482003cf8d5d70f056894ec6379e343631faa47f" gracePeriod=30 Nov 02 18:21:37.700284 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.691778 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8"] Nov 02 18:21:37.700284 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.692094 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" podUID="468e1807-5ca2-4bef-8674-0879f2b3a77e" containerName="istio-proxy" containerID="cri-o://48eeaee54e05500ad1e69249059ca6c5f245d875d0a0165926f2f4f87332e7f8" gracePeriod=30 Nov 02 18:21:37.901027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.897680 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Nov 02 18:21:37.901027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.897743 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" podUID="752af71e-0e0d-48b9-9555-9ce325e913ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Nov 02 18:21:38.023978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:37.992851 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:38.078171 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:38.067579 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body= Nov 02 18:21:38.078171 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:38.067646 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" podUID="8998b9b1-c956-4ffb-9274-9eb5443894f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" Nov 02 18:21:38.268511 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:38.268476 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body= Nov 02 18:21:38.270151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:38.270117 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" podUID="41c5644b-6e3c-45b3-83da-f8c89a081c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" Nov 02 18:21:38.573975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:38.573571 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj"] Nov 02 18:21:38.587976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:38.584084 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 18:21:38.696540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:38.696501 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:38.752285 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:38.729716 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd"] Nov 02 18:21:38.838801 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:38.838715 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:39.038083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:39.028295 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:39.038083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:39.028441 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Nov 02 18:21:39.038083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:39.028502 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" podUID="752af71e-0e0d-48b9-9555-9ce325e913ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Nov 02 18:21:39.071097 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:39.061238 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body= Nov 02 18:21:39.071097 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:39.061295 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" podUID="8998b9b1-c956-4ffb-9274-9eb5443894f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" Nov 02 18:21:39.127819 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:39.120335 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body= Nov 02 18:21:39.127819 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:39.120392 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" podUID="41c5644b-6e3c-45b3-83da-f8c89a081c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" Nov 02 18:21:39.456978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:39.456405 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 02 18:21:39.456978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:39.456470 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" podUID="0c3b2112-be30-40ae-bc4b-84d75341a926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 02 18:21:39.708040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:39.696811 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:40.062643 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.062489 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body= Nov 02 18:21:40.062899 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.062866 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" podUID="8998b9b1-c956-4ffb-9274-9eb5443894f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" Nov 02 18:21:40.075156 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.075122 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Nov 02 18:21:40.075415 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.075387 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" podUID="752af71e-0e0d-48b9-9555-9ce325e913ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Nov 02 18:21:40.174617 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.174218 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" event={"ID":"f2b6e563-a46e-42e2-bc91-5a0277272b56","Type":"ContainerStarted","Data":"8d86ac4a6797893de697669125fbd7b745dd11e963b40cefff833832b52f4d14"} Nov 02 18:21:40.174617 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.174422 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" event={"ID":"f2b6e563-a46e-42e2-bc91-5a0277272b56","Type":"ContainerStarted","Data":"39730ba6e585f13b26eba42e3e419111389329161d2852de7d0893b823a78c1a"} Nov 02 18:21:40.175531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.175478 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" Nov 02 18:21:40.177683 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.177659 5549 generic.go:356] "Generic (PLEG): container finished" podID="0c3b2112-be30-40ae-bc4b-84d75341a926" containerID="7875f01f9f135719a6dba7c099fdce17c752085ffa2ee3359c95775fa69edae7" exitCode=0 Nov 02 18:21:40.177832 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.177819 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" event={"ID":"0c3b2112-be30-40ae-bc4b-84d75341a926","Type":"ContainerDied","Data":"7875f01f9f135719a6dba7c099fdce17c752085ffa2ee3359c95775fa69edae7"} Nov 02 18:21:40.208001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.198559 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:40.238974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.236113 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:40.246973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.246188 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:21:40.299520 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.299455 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" podStartSLOduration=4.299432843 podStartE2EDuration="4.299432843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:40.265186589 +0000 UTC m=+2872.053692118" watchObservedRunningTime="2026-11-02 18:21:40.299432843 +0000 UTC m=+2872.087938373" Nov 02 18:21:40.339271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.338912 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 18:21:40.441109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.441073 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-podinfo\") pod \"0c3b2112-be30-40ae-bc4b-84d75341a926\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " Nov 02 18:21:40.441786 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.441760 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-token\" (UniqueName: \"kubernetes.io/projected/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-token\") pod \"0c3b2112-be30-40ae-bc4b-84d75341a926\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " Nov 02 18:21:40.442383 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.442362 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c3b2112-be30-40ae-bc4b-84d75341a926-kube-api-access-j589r\" (UniqueName: \"kubernetes.io/projected/0c3b2112-be30-40ae-bc4b-84d75341a926-kube-api-access-j589r\") pod \"0c3b2112-be30-40ae-bc4b-84d75341a926\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " Nov 02 18:21:40.442816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.442796 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-data\") pod \"0c3b2112-be30-40ae-bc4b-84d75341a926\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " Nov 02 18:21:40.443023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.443006 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-workload-socket\") pod \"0c3b2112-be30-40ae-bc4b-84d75341a926\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " Nov 02 18:21:40.443184 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.443167 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-workload-certs\") pod \"0c3b2112-be30-40ae-bc4b-84d75341a926\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " Nov 02 18:21:40.443572 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.443554 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-envoy\") pod \"0c3b2112-be30-40ae-bc4b-84d75341a926\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " Nov 02 18:21:40.443944 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.443925 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c3b2112-be30-40ae-bc4b-84d75341a926-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3b2112-be30-40ae-bc4b-84d75341a926-istiod-ca-cert\") pod \"0c3b2112-be30-40ae-bc4b-84d75341a926\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " Nov 02 18:21:40.444180 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.444162 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-credential-socket\") pod \"0c3b2112-be30-40ae-bc4b-84d75341a926\" (UID: \"0c3b2112-be30-40ae-bc4b-84d75341a926\") " Nov 02 18:21:40.445761 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.445738 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-data" pod "0c3b2112-be30-40ae-bc4b-84d75341a926" (UID: "0c3b2112-be30-40ae-bc4b-84d75341a926"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:40.446089 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.446068 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-workload-socket" pod "0c3b2112-be30-40ae-bc4b-84d75341a926" (UID: "0c3b2112-be30-40ae-bc4b-84d75341a926"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:40.449340 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.449312 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-workload-certs" pod "0c3b2112-be30-40ae-bc4b-84d75341a926" (UID: "0c3b2112-be30-40ae-bc4b-84d75341a926"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:40.450454 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.450438 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c3b2112-be30-40ae-bc4b-84d75341a926-istiod-ca-cert" pod "0c3b2112-be30-40ae-bc4b-84d75341a926" (UID: "0c3b2112-be30-40ae-bc4b-84d75341a926"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:40.450579 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.450565 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z"] Nov 02 18:21:40.453116 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.453099 5549 cpu_manager.go:414] "containerMap: removing container" podUID="0c3b2112-be30-40ae-bc4b-84d75341a926" containerName="istio-proxy" Nov 02 18:21:40.456389 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.456370 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c3b2112-be30-40ae-bc4b-84d75341a926" containerName="istio-proxy" Nov 02 18:21:40.456845 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.456827 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c3b2112-be30-40ae-bc4b-84d75341a926" containerName="istio-proxy" Nov 02 18:21:40.459743 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.453336 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-credential-socket" pod "0c3b2112-be30-40ae-bc4b-84d75341a926" (UID: "0c3b2112-be30-40ae-bc4b-84d75341a926"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:40.460136 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.457586 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-envoy" pod "0c3b2112-be30-40ae-bc4b-84d75341a926" (UID: "0c3b2112-be30-40ae-bc4b-84d75341a926"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:40.518344 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.518300 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-podinfo" pod "0c3b2112-be30-40ae-bc4b-84d75341a926" (UID: "0c3b2112-be30-40ae-bc4b-84d75341a926"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:21:40.518618 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.518585 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3b2112-be30-40ae-bc4b-84d75341a926-kube-api-access-j589r" pod "0c3b2112-be30-40ae-bc4b-84d75341a926" (UID: "0c3b2112-be30-40ae-bc4b-84d75341a926"). InnerVolumeSpecName "kube-api-access-j589r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:40.523072 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.523042 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z"] Nov 02 18:21:40.523221 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.523162 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.525491 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.525464 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:40.526468 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.526445 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd"] Nov 02 18:21:40.526793 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.526766 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" podUID="781f545f-9659-4d21-9e53-85d4b82a7214" containerName="istio-proxy" containerID="cri-o://78366fe1c80b10886716cfa09181443404c28b41f9d40e19e0a91d9ff0a26d3d" gracePeriod=30 Nov 02 18:21:40.537716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.537637 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-token" pod "0c3b2112-be30-40ae-bc4b-84d75341a926" (UID: "0c3b2112-be30-40ae-bc4b-84d75341a926"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:40.552575 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.552543 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3b2112-be30-40ae-bc4b-84d75341a926-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:40.552702 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.552582 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:40.552702 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.552598 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:40.552702 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.552613 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:40.552702 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.552629 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j589r\" (UniqueName: \"kubernetes.io/projected/0c3b2112-be30-40ae-bc4b-84d75341a926-kube-api-access-j589r\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:40.552702 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.552643 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:40.552702 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.552658 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:40.553083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.552672 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:40.553083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.552928 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2112-be30-40ae-bc4b-84d75341a926-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:40.634141 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.634050 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc"] Nov 02 18:21:40.634596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.634564 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" podUID="5639111b-8dcd-4b4a-a845-f21825bda0d9" containerName="istio-proxy" containerID="cri-o://7901a099298e95a847238d46d4ffe24d63475a2d9809b3a8fcee3557307afc66" gracePeriod=30 Nov 02 18:21:40.658517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.658460 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t"] Nov 02 18:21:40.658980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.658935 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" podUID="ca0043f1-fb84-4f38-b637-2315256e0c46" containerName="istio-proxy" containerID="cri-o://52aa504a43c50ad1cd12420fa399ab6a81cdd39423c82e42c208f1297a4456ce" gracePeriod=30 Nov 02 18:21:40.662092 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.662044 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.662278 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.662258 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.662492 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.662471 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.662653 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.662635 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55fbc482-0099-414a-a9d0-87d3d0256de7-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.662805 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.662789 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.662934 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.662916 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd2jk\" (UniqueName: \"kubernetes.io/projected/55fbc482-0099-414a-a9d0-87d3d0256de7-kube-api-access-gd2jk\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.663185 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.663164 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.663368 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.663351 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.663598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.663574 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.766502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.766455 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.766684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.766521 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.766684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.766603 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.766810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.766683 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.766810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.766705 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.766810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.766770 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.766810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.766804 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55fbc482-0099-414a-a9d0-87d3d0256de7-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.767041 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.766836 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.767041 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.766854 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2jk\" (UniqueName: \"kubernetes.io/projected/55fbc482-0099-414a-a9d0-87d3d0256de7-kube-api-access-gd2jk\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.767652 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.767633 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.767851 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.767836 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.768535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.768302 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.768932 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.768887 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55fbc482-0099-414a-a9d0-87d3d0256de7-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.769036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.768973 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.771221 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.770509 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.771221 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.771096 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.775849 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.775823 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd2jk\" (UniqueName: \"kubernetes.io/projected/55fbc482-0099-414a-a9d0-87d3d0256de7-kube-api-access-gd2jk\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.821272 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.821230 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:40.918387 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:40.918298 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:41.050595 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:41.050475 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:41.054160 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:41.052639 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:41.055005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:41.054454 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:21:41.055005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:41.054587 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body= Nov 02 18:21:41.055005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:41.054635 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" podUID="8998b9b1-c956-4ffb-9274-9eb5443894f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" Nov 02 18:21:41.244114 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:41.244078 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" event={"ID":"0c3b2112-be30-40ae-bc4b-84d75341a926","Type":"ContainerDied","Data":"c08a66c46bfc89ecf4064819779f3392c07938cf156382f2710dfa1ba3893ef5"} Nov 02 18:21:41.244291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:41.244147 5549 scope.go:122] "RemoveContainer" containerID="7875f01f9f135719a6dba7c099fdce17c752085ffa2ee3359c95775fa69edae7" Nov 02 18:21:41.244908 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:41.244883 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg" Nov 02 18:21:41.262807 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:41.262741 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z"] Nov 02 18:21:41.263864 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:21:41.263830 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55fbc482_0099_414a_a9d0_87d3d0256de7.slice/crio-533d39784fd440fdbc987777dc0ee3e1d11ed552e81ad9969de94cf8f458f3c7 WatchSource:0}: Error finding container 533d39784fd440fdbc987777dc0ee3e1d11ed552e81ad9969de94cf8f458f3c7: Status 404 returned error can't find the container with id 533d39784fd440fdbc987777dc0ee3e1d11ed552e81ad9969de94cf8f458f3c7 Nov 02 18:21:41.267354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:41.267316 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:41.267473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:41.267385 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:41.267473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:41.267417 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:41.333721 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:41.333689 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg"] Nov 02 18:21:41.333875 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:41.333738 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-pk7vg"] Nov 02 18:21:41.422486 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:41.422434 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:42.055158 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:42.055123 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body= Nov 02 18:21:42.055644 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:42.055183 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" podUID="8998b9b1-c956-4ffb-9274-9eb5443894f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" Nov 02 18:21:42.239907 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:42.239865 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:42.255047 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:42.255004 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" event={"ID":"55fbc482-0099-414a-a9d0-87d3d0256de7","Type":"ContainerStarted","Data":"03d4fdd935f368db0c6b4219e79bf62c9244cbdf09de3e0d14b9d440714d4b82"} Nov 02 18:21:42.255224 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:42.255071 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" event={"ID":"55fbc482-0099-414a-a9d0-87d3d0256de7","Type":"ContainerStarted","Data":"533d39784fd440fdbc987777dc0ee3e1d11ed552e81ad9969de94cf8f458f3c7"} Nov 02 18:21:42.260888 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:42.260848 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" podUID="7990f1b0-ff87-4445-8a17-93052b4c9586" containerName="main" containerID="cri-o://20cd6bed1629142e0b4d0328c535c0334b0875c77979aeabd738eb8a950766ff" gracePeriod=45 Nov 02 18:21:42.317972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:42.316741 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" podStartSLOduration=104.316719556 podStartE2EDuration="1m44.316719556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:42.278693696 +0000 UTC m=+2874.067199224" watchObservedRunningTime="2026-11-02 18:21:42.316719556 +0000 UTC m=+2874.105225085" Nov 02 18:21:42.342972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:42.335836 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c3b2112-be30-40ae-bc4b-84d75341a926" path="/var/lib/kubelet/pods/0c3b2112-be30-40ae-bc4b-84d75341a926/volumes" Nov 02 18:21:42.946601 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:42.941638 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:42.946601 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:42.942523 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 02 18:21:42.946601 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:42.942576 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" podUID="55fbc482-0099-414a-a9d0-87d3d0256de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 02 18:21:42.954709 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:42.954678 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:21:43.169705 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.169663 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body= Nov 02 18:21:43.170192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.169727 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" podUID="8998b9b1-c956-4ffb-9274-9eb5443894f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" Nov 02 18:21:43.171984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.171932 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-tmp-dir\") pod \"7990f1b0-ff87-4445-8a17-93052b4c9586\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " Nov 02 18:21:43.172105 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.172077 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-dshm\") pod \"7990f1b0-ff87-4445-8a17-93052b4c9586\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " Nov 02 18:21:43.172158 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.172139 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7990f1b0-ff87-4445-8a17-93052b4c9586-tls-certs\" (UniqueName: \"kubernetes.io/secret/7990f1b0-ff87-4445-8a17-93052b4c9586-tls-certs\") pod \"7990f1b0-ff87-4445-8a17-93052b4c9586\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " Nov 02 18:21:43.172211 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.172195 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-home\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-home\") pod \"7990f1b0-ff87-4445-8a17-93052b4c9586\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " Nov 02 18:21:43.172279 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.172263 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-model-cache\") pod \"7990f1b0-ff87-4445-8a17-93052b4c9586\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " Nov 02 18:21:43.172556 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.172538 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7990f1b0-ff87-4445-8a17-93052b4c9586-kube-api-access-xtvdc\" (UniqueName: \"kubernetes.io/projected/7990f1b0-ff87-4445-8a17-93052b4c9586-kube-api-access-xtvdc\") pod \"7990f1b0-ff87-4445-8a17-93052b4c9586\" (UID: \"7990f1b0-ff87-4445-8a17-93052b4c9586\") " Nov 02 18:21:43.174275 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.174240 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-tmp-dir" pod "7990f1b0-ff87-4445-8a17-93052b4c9586" (UID: "7990f1b0-ff87-4445-8a17-93052b4c9586"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:43.174471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.174446 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-home" pod "7990f1b0-ff87-4445-8a17-93052b4c9586" (UID: "7990f1b0-ff87-4445-8a17-93052b4c9586"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:43.174646 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.174626 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.174705 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.174651 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.174885 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.174858 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-model-cache" pod "7990f1b0-ff87-4445-8a17-93052b4c9586" (UID: "7990f1b0-ff87-4445-8a17-93052b4c9586"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:43.175747 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.175718 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:43.177717 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.177687 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-dshm" pod "7990f1b0-ff87-4445-8a17-93052b4c9586" (UID: "7990f1b0-ff87-4445-8a17-93052b4c9586"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:43.178317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.178103 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7990f1b0-ff87-4445-8a17-93052b4c9586-kube-api-access-xtvdc" pod "7990f1b0-ff87-4445-8a17-93052b4c9586" (UID: "7990f1b0-ff87-4445-8a17-93052b4c9586"). InnerVolumeSpecName "kube-api-access-xtvdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:43.247009 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.246495 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7990f1b0-ff87-4445-8a17-93052b4c9586-tls-certs" pod "7990f1b0-ff87-4445-8a17-93052b4c9586" (UID: "7990f1b0-ff87-4445-8a17-93052b4c9586"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:21:43.302976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.278732 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.302976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.278768 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7990f1b0-ff87-4445-8a17-93052b4c9586-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.302976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.278784 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7990f1b0-ff87-4445-8a17-93052b4c9586-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.302976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.278816 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtvdc\" (UniqueName: \"kubernetes.io/projected/7990f1b0-ff87-4445-8a17-93052b4c9586-kube-api-access-xtvdc\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.337023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.330660 5549 generic.go:356] "Generic (PLEG): container finished" podID="b20603e9-d7ea-4bac-bf54-c1c59d343fa8" containerID="56414aed1f8107254cf65658482003cf8d5d70f056894ec6379e343631faa47f" exitCode=0 Nov 02 18:21:43.337023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.330922 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" event={"ID":"b20603e9-d7ea-4bac-bf54-c1c59d343fa8","Type":"ContainerDied","Data":"56414aed1f8107254cf65658482003cf8d5d70f056894ec6379e343631faa47f"} Nov 02 18:21:43.337023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.333707 5549 generic.go:356] "Generic (PLEG): container finished" podID="468e1807-5ca2-4bef-8674-0879f2b3a77e" containerID="48eeaee54e05500ad1e69249059ca6c5f245d875d0a0165926f2f4f87332e7f8" exitCode=0 Nov 02 18:21:43.337023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.333938 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" event={"ID":"468e1807-5ca2-4bef-8674-0879f2b3a77e","Type":"ContainerDied","Data":"48eeaee54e05500ad1e69249059ca6c5f245d875d0a0165926f2f4f87332e7f8"} Nov 02 18:21:43.342537 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.341147 5549 generic.go:356] "Generic (PLEG): container finished" podID="7990f1b0-ff87-4445-8a17-93052b4c9586" containerID="20cd6bed1629142e0b4d0328c535c0334b0875c77979aeabd738eb8a950766ff" exitCode=0 Nov 02 18:21:43.344440 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.344362 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" Nov 02 18:21:43.346089 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.344733 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" event={"ID":"7990f1b0-ff87-4445-8a17-93052b4c9586","Type":"ContainerDied","Data":"20cd6bed1629142e0b4d0328c535c0334b0875c77979aeabd738eb8a950766ff"} Nov 02 18:21:43.347008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.346839 5549 scope.go:122] "RemoveContainer" containerID="20cd6bed1629142e0b4d0328c535c0334b0875c77979aeabd738eb8a950766ff" Nov 02 18:21:43.349062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.346811 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" event={"ID":"7990f1b0-ff87-4445-8a17-93052b4c9586","Type":"ContainerDied","Data":"e551cc083eaa1d1984d7e3133ba954dbfe50dd586106f6cc766e95f35bfbc4c1"} Nov 02 18:21:43.363844 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.363826 5549 scope.go:122] "RemoveContainer" containerID="20cd6bed1629142e0b4d0328c535c0334b0875c77979aeabd738eb8a950766ff" Nov 02 18:21:43.364151 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:21:43.364126 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20cd6bed1629142e0b4d0328c535c0334b0875c77979aeabd738eb8a950766ff\": container with ID starting with 20cd6bed1629142e0b4d0328c535c0334b0875c77979aeabd738eb8a950766ff not found: ID does not exist" containerID="20cd6bed1629142e0b4d0328c535c0334b0875c77979aeabd738eb8a950766ff" Nov 02 18:21:43.364316 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.364264 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20cd6bed1629142e0b4d0328c535c0334b0875c77979aeabd738eb8a950766ff"} err="failed to get container status \"20cd6bed1629142e0b4d0328c535c0334b0875c77979aeabd738eb8a950766ff\": rpc error: code = NotFound desc = could not find container \"20cd6bed1629142e0b4d0328c535c0334b0875c77979aeabd738eb8a950766ff\": container with ID starting with 20cd6bed1629142e0b4d0328c535c0334b0875c77979aeabd738eb8a950766ff not found: ID does not exist" Nov 02 18:21:43.378521 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.378488 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms"] Nov 02 18:21:43.444693 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.444646 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms"] Nov 02 18:21:43.519120 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.519083 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:21:43.545000 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.544758 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-312f21/test-model-312f21-kserve-dfc5cf8d5-xdzms" podUID="7990f1b0-ff87-4445-8a17-93052b4c9586" containerName="main" probeResult="failure" output="Get \"https://10.133.1.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:21:43.550774 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.550748 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:21:43.554344 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.554319 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:21:43.634631 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.634549 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-data\") pod \"468e1807-5ca2-4bef-8674-0879f2b3a77e\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " Nov 02 18:21:43.634774 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.634654 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468e1807-5ca2-4bef-8674-0879f2b3a77e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468e1807-5ca2-4bef-8674-0879f2b3a77e-istiod-ca-cert\") pod \"468e1807-5ca2-4bef-8674-0879f2b3a77e\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " Nov 02 18:21:43.634774 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.634681 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468e1807-5ca2-4bef-8674-0879f2b3a77e-kube-api-access-svrpj\" (UniqueName: \"kubernetes.io/projected/468e1807-5ca2-4bef-8674-0879f2b3a77e-kube-api-access-svrpj\") pod \"468e1807-5ca2-4bef-8674-0879f2b3a77e\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " Nov 02 18:21:43.634858 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.634806 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-credential-socket\") pod \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " Nov 02 18:21:43.634897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.634858 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istiod-ca-cert\") pod \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " Nov 02 18:21:43.635016 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.634937 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-credential-socket\") pod \"468e1807-5ca2-4bef-8674-0879f2b3a77e\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " Nov 02 18:21:43.635207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.635149 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-envoy\") pod \"468e1807-5ca2-4bef-8674-0879f2b3a77e\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " Nov 02 18:21:43.635298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.635210 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-workload-certs\") pod \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " Nov 02 18:21:43.635298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.635293 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-data\") pod \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " Nov 02 18:21:43.635409 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.635355 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-workload-certs\") pod \"468e1807-5ca2-4bef-8674-0879f2b3a77e\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " Nov 02 18:21:43.635409 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.635382 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-envoy\") pod \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " Nov 02 18:21:43.635526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.635418 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-kube-api-access-lwzcv\" (UniqueName: \"kubernetes.io/projected/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-kube-api-access-lwzcv\") pod \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " Nov 02 18:21:43.635526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.635444 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-workload-socket\") pod \"468e1807-5ca2-4bef-8674-0879f2b3a77e\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " Nov 02 18:21:43.635526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.635452 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468e1807-5ca2-4bef-8674-0879f2b3a77e-istiod-ca-cert" pod "468e1807-5ca2-4bef-8674-0879f2b3a77e" (UID: "468e1807-5ca2-4bef-8674-0879f2b3a77e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:43.635526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.635485 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-token\" (UniqueName: \"kubernetes.io/projected/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-token\") pod \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " Nov 02 18:21:43.635526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.635517 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-podinfo\") pod \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " Nov 02 18:21:43.635737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.635546 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-workload-socket\") pod \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\" (UID: \"b20603e9-d7ea-4bac-bf54-c1c59d343fa8\") " Nov 02 18:21:43.635737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.635611 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-token\" (UniqueName: \"kubernetes.io/projected/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-token\") pod \"468e1807-5ca2-4bef-8674-0879f2b3a77e\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " Nov 02 18:21:43.635737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.635666 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-podinfo\") pod \"468e1807-5ca2-4bef-8674-0879f2b3a77e\" (UID: \"468e1807-5ca2-4bef-8674-0879f2b3a77e\") " Nov 02 18:21:43.636438 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.636208 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-data" pod "468e1807-5ca2-4bef-8674-0879f2b3a77e" (UID: "468e1807-5ca2-4bef-8674-0879f2b3a77e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:43.636534 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.636485 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-workload-socket" pod "b20603e9-d7ea-4bac-bf54-c1c59d343fa8" (UID: "b20603e9-d7ea-4bac-bf54-c1c59d343fa8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:43.637344 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.636877 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.637344 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.636907 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.637344 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.636923 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468e1807-5ca2-4bef-8674-0879f2b3a77e-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.639781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.639755 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-workload-socket" pod "468e1807-5ca2-4bef-8674-0879f2b3a77e" (UID: "468e1807-5ca2-4bef-8674-0879f2b3a77e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:43.640235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.640208 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-workload-certs" pod "468e1807-5ca2-4bef-8674-0879f2b3a77e" (UID: "468e1807-5ca2-4bef-8674-0879f2b3a77e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:43.640630 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.640609 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-credential-socket" pod "468e1807-5ca2-4bef-8674-0879f2b3a77e" (UID: "468e1807-5ca2-4bef-8674-0879f2b3a77e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:43.641055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.641035 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-credential-socket" pod "b20603e9-d7ea-4bac-bf54-c1c59d343fa8" (UID: "b20603e9-d7ea-4bac-bf54-c1c59d343fa8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:43.641465 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.641445 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istiod-ca-cert" pod "b20603e9-d7ea-4bac-bf54-c1c59d343fa8" (UID: "b20603e9-d7ea-4bac-bf54-c1c59d343fa8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:43.644384 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.644351 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-workload-certs" pod "b20603e9-d7ea-4bac-bf54-c1c59d343fa8" (UID: "b20603e9-d7ea-4bac-bf54-c1c59d343fa8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:43.645274 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.644506 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468e1807-5ca2-4bef-8674-0879f2b3a77e-kube-api-access-svrpj" pod "468e1807-5ca2-4bef-8674-0879f2b3a77e" (UID: "468e1807-5ca2-4bef-8674-0879f2b3a77e"). InnerVolumeSpecName "kube-api-access-svrpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:43.645274 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.644724 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-data" pod "b20603e9-d7ea-4bac-bf54-c1c59d343fa8" (UID: "b20603e9-d7ea-4bac-bf54-c1c59d343fa8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:43.645274 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.644828 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-token" pod "468e1807-5ca2-4bef-8674-0879f2b3a77e" (UID: "468e1807-5ca2-4bef-8674-0879f2b3a77e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:43.647037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.646921 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-envoy" pod "468e1807-5ca2-4bef-8674-0879f2b3a77e" (UID: "468e1807-5ca2-4bef-8674-0879f2b3a77e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:43.647231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.647210 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-podinfo" pod "468e1807-5ca2-4bef-8674-0879f2b3a77e" (UID: "468e1807-5ca2-4bef-8674-0879f2b3a77e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:21:43.665576 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.665502 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-kube-api-access-lwzcv" pod "b20603e9-d7ea-4bac-bf54-c1c59d343fa8" (UID: "b20603e9-d7ea-4bac-bf54-c1c59d343fa8"). InnerVolumeSpecName "kube-api-access-lwzcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:43.665576 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.665552 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-token" pod "b20603e9-d7ea-4bac-bf54-c1c59d343fa8" (UID: "b20603e9-d7ea-4bac-bf54-c1c59d343fa8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:43.665935 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.665890 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-envoy" pod "b20603e9-d7ea-4bac-bf54-c1c59d343fa8" (UID: "b20603e9-d7ea-4bac-bf54-c1c59d343fa8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:43.669189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.669147 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-podinfo" pod "b20603e9-d7ea-4bac-bf54-c1c59d343fa8" (UID: "b20603e9-d7ea-4bac-bf54-c1c59d343fa8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:21:43.738970 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.738924 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.738970 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.738975 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svrpj\" (UniqueName: \"kubernetes.io/projected/468e1807-5ca2-4bef-8674-0879f2b3a77e-kube-api-access-svrpj\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.739210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.738990 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.739210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.739000 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.739210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.739009 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.739210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.739019 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.739210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.739028 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.739210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.739038 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.739210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.739046 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.739210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.739183 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.739210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.739194 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwzcv\" (UniqueName: \"kubernetes.io/projected/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-kube-api-access-lwzcv\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.739210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.739204 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468e1807-5ca2-4bef-8674-0879f2b3a77e-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.739771 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.739213 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.739771 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.739238 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20603e9-d7ea-4bac-bf54-c1c59d343fa8-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.739771 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.739250 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468e1807-5ca2-4bef-8674-0879f2b3a77e-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:43.919415 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.919309 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 02 18:21:43.919415 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.919393 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" podUID="55fbc482-0099-414a-a9d0-87d3d0256de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 02 18:21:43.985864 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:43.985827 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:44.229504 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:44.228344 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:44.229504 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:44.229473 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:44.231067 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:44.231017 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:21:44.357795 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:44.357755 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7990f1b0-ff87-4445-8a17-93052b4c9586" path="/var/lib/kubelet/pods/7990f1b0-ff87-4445-8a17-93052b4c9586/volumes" Nov 02 18:21:44.368142 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:44.367985 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" event={"ID":"468e1807-5ca2-4bef-8674-0879f2b3a77e","Type":"ContainerDied","Data":"eef08e048e9fb49e3fc36555ac68a09db02c9d904254664117f884908dc62551"} Nov 02 18:21:44.368142 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:44.368058 5549 scope.go:122] "RemoveContainer" containerID="48eeaee54e05500ad1e69249059ca6c5f245d875d0a0165926f2f4f87332e7f8" Nov 02 18:21:44.369932 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:44.369341 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8" Nov 02 18:21:44.372579 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:44.372489 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" event={"ID":"b20603e9-d7ea-4bac-bf54-c1c59d343fa8","Type":"ContainerDied","Data":"d0a09a5b1e71d134bd5e4d4a0a753f754858db3e69bb8a5bbdc384ff512f7e35"} Nov 02 18:21:44.374094 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:44.372851 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx" Nov 02 18:21:44.429291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:44.429254 5549 scope.go:122] "RemoveContainer" containerID="56414aed1f8107254cf65658482003cf8d5d70f056894ec6379e343631faa47f" Nov 02 18:21:44.439275 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:44.439243 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx"] Nov 02 18:21:44.446899 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:44.446875 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-n27fx"] Nov 02 18:21:44.460852 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:44.460817 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8"] Nov 02 18:21:44.465227 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:44.465204 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-xdzp8"] Nov 02 18:21:44.918934 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:44.918883 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 02 18:21:44.919176 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:44.918993 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" podUID="55fbc482-0099-414a-a9d0-87d3d0256de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 02 18:21:44.944360 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:44.944017 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:45.072874 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:45.072829 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:45.672974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:45.672601 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:45.751702 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:45.751648 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5lphd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 18:21:45.751911 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:45.751764 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" podUID="781f545f-9659-4d21-9e53-85d4b82a7214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 18:21:46.085267 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.036489 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 02 18:21:46.085267 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.036551 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" podUID="55fbc482-0099-414a-a9d0-87d3d0256de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 02 18:21:46.085267 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.047170 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:46.752973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.746255 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:46.774975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.759703 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 18:21:46.805594 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.805558 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="468e1807-5ca2-4bef-8674-0879f2b3a77e" path="/var/lib/kubelet/pods/468e1807-5ca2-4bef-8674-0879f2b3a77e/volumes" Nov 02 18:21:46.806300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.806276 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b20603e9-d7ea-4bac-bf54-c1c59d343fa8" path="/var/lib/kubelet/pods/b20603e9-d7ea-4bac-bf54-c1c59d343fa8/volumes" Nov 02 18:21:46.823737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.823693 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" podUID="63194adc-5a21-49ca-a101-da5038c43036" containerName="main" probeResult="failure" output="Get \"https://10.133.1.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:21:46.864050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.862220 5549 generic.go:356] "Generic (PLEG): container finished" podID="5639111b-8dcd-4b4a-a845-f21825bda0d9" containerID="7901a099298e95a847238d46d4ffe24d63475a2d9809b3a8fcee3557307afc66" exitCode=0 Nov 02 18:21:46.864050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.862416 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" event={"ID":"5639111b-8dcd-4b4a-a845-f21825bda0d9","Type":"ContainerDied","Data":"7901a099298e95a847238d46d4ffe24d63475a2d9809b3a8fcee3557307afc66"} Nov 02 18:21:46.871622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.871593 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/781f545f-9659-4d21-9e53-85d4b82a7214-kube-api-access-tz4xs\" (UniqueName: \"kubernetes.io/projected/781f545f-9659-4d21-9e53-85d4b82a7214-kube-api-access-tz4xs\") pod \"781f545f-9659-4d21-9e53-85d4b82a7214\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " Nov 02 18:21:46.872047 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.872021 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/781f545f-9659-4d21-9e53-85d4b82a7214-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781f545f-9659-4d21-9e53-85d4b82a7214-istio-podinfo\") pod \"781f545f-9659-4d21-9e53-85d4b82a7214\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " Nov 02 18:21:46.872674 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.872655 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/781f545f-9659-4d21-9e53-85d4b82a7214-istio-token\" (UniqueName: \"kubernetes.io/projected/781f545f-9659-4d21-9e53-85d4b82a7214-istio-token\") pod \"781f545f-9659-4d21-9e53-85d4b82a7214\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " Nov 02 18:21:46.872896 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.872881 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-workload-socket\") pod \"781f545f-9659-4d21-9e53-85d4b82a7214\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " Nov 02 18:21:46.873116 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.873098 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/781f545f-9659-4d21-9e53-85d4b82a7214-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781f545f-9659-4d21-9e53-85d4b82a7214-istiod-ca-cert\") pod \"781f545f-9659-4d21-9e53-85d4b82a7214\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " Nov 02 18:21:46.873333 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.873269 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-workload-certs\") pod \"781f545f-9659-4d21-9e53-85d4b82a7214\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " Nov 02 18:21:46.873998 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.873978 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-credential-socket\") pod \"781f545f-9659-4d21-9e53-85d4b82a7214\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " Nov 02 18:21:46.874175 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.874153 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-istio-envoy\") pod \"781f545f-9659-4d21-9e53-85d4b82a7214\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " Nov 02 18:21:46.874327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.874312 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-istio-data\") pod \"781f545f-9659-4d21-9e53-85d4b82a7214\" (UID: \"781f545f-9659-4d21-9e53-85d4b82a7214\") " Nov 02 18:21:46.874666 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.874642 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781f545f-9659-4d21-9e53-85d4b82a7214-kube-api-access-tz4xs" pod "781f545f-9659-4d21-9e53-85d4b82a7214" (UID: "781f545f-9659-4d21-9e53-85d4b82a7214"). InnerVolumeSpecName "kube-api-access-tz4xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:46.874820 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.874800 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-workload-socket" pod "781f545f-9659-4d21-9e53-85d4b82a7214" (UID: "781f545f-9659-4d21-9e53-85d4b82a7214"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:46.874935 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.874852 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-workload-certs" pod "781f545f-9659-4d21-9e53-85d4b82a7214" (UID: "781f545f-9659-4d21-9e53-85d4b82a7214"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:46.875238 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.875179 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/781f545f-9659-4d21-9e53-85d4b82a7214-istiod-ca-cert" pod "781f545f-9659-4d21-9e53-85d4b82a7214" (UID: "781f545f-9659-4d21-9e53-85d4b82a7214"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:46.876299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.876275 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-istio-data" pod "781f545f-9659-4d21-9e53-85d4b82a7214" (UID: "781f545f-9659-4d21-9e53-85d4b82a7214"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:46.877085 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.877057 5549 generic.go:356] "Generic (PLEG): container finished" podID="ca0043f1-fb84-4f38-b637-2315256e0c46" containerID="52aa504a43c50ad1cd12420fa399ab6a81cdd39423c82e42c208f1297a4456ce" exitCode=0 Nov 02 18:21:46.877355 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.877331 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" event={"ID":"ca0043f1-fb84-4f38-b637-2315256e0c46","Type":"ContainerDied","Data":"52aa504a43c50ad1cd12420fa399ab6a81cdd39423c82e42c208f1297a4456ce"} Nov 02 18:21:46.882005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.878799 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:46.882173 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.882153 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781f545f-9659-4d21-9e53-85d4b82a7214-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:46.882450 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.882433 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:46.882585 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.882572 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:46.882701 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.882687 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz4xs\" (UniqueName: \"kubernetes.io/projected/781f545f-9659-4d21-9e53-85d4b82a7214-kube-api-access-tz4xs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:46.882806 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.878881 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-credential-socket" pod "781f545f-9659-4d21-9e53-85d4b82a7214" (UID: "781f545f-9659-4d21-9e53-85d4b82a7214"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:46.882916 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.880915 5549 generic.go:356] "Generic (PLEG): container finished" podID="781f545f-9659-4d21-9e53-85d4b82a7214" containerID="78366fe1c80b10886716cfa09181443404c28b41f9d40e19e0a91d9ff0a26d3d" exitCode=0 Nov 02 18:21:46.883243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.880935 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" event={"ID":"781f545f-9659-4d21-9e53-85d4b82a7214","Type":"ContainerDied","Data":"78366fe1c80b10886716cfa09181443404c28b41f9d40e19e0a91d9ff0a26d3d"} Nov 02 18:21:46.883408 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.883395 5549 scope.go:122] "RemoveContainer" containerID="78366fe1c80b10886716cfa09181443404c28b41f9d40e19e0a91d9ff0a26d3d" Nov 02 18:21:46.883665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.881060 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd" Nov 02 18:21:46.885860 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.884706 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:21:46.944074 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.940842 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 02 18:21:46.944074 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.940909 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" podUID="55fbc482-0099-414a-a9d0-87d3d0256de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 02 18:21:46.944074 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.941163 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-istio-envoy" pod "781f545f-9659-4d21-9e53-85d4b82a7214" (UID: "781f545f-9659-4d21-9e53-85d4b82a7214"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:46.944074 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.941411 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/781f545f-9659-4d21-9e53-85d4b82a7214-istio-podinfo" pod "781f545f-9659-4d21-9e53-85d4b82a7214" (UID: "781f545f-9659-4d21-9e53-85d4b82a7214"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:21:46.944074 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.941556 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781f545f-9659-4d21-9e53-85d4b82a7214-istio-token" pod "781f545f-9659-4d21-9e53-85d4b82a7214" (UID: "781f545f-9659-4d21-9e53-85d4b82a7214"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:46.987045 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.986739 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca0043f1-fb84-4f38-b637-2315256e0c46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0043f1-fb84-4f38-b637-2315256e0c46-istiod-ca-cert\") pod \"ca0043f1-fb84-4f38-b637-2315256e0c46\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " Nov 02 18:21:46.987045 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.986804 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-data\") pod \"ca0043f1-fb84-4f38-b637-2315256e0c46\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " Nov 02 18:21:46.987045 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.986944 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-podinfo\") pod \"ca0043f1-fb84-4f38-b637-2315256e0c46\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " Nov 02 18:21:46.987045 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.987018 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca0043f1-fb84-4f38-b637-2315256e0c46-kube-api-access-gk72s\" (UniqueName: \"kubernetes.io/projected/ca0043f1-fb84-4f38-b637-2315256e0c46-kube-api-access-gk72s\") pod \"ca0043f1-fb84-4f38-b637-2315256e0c46\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " Nov 02 18:21:46.989670 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.989266 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-token\" (UniqueName: \"kubernetes.io/projected/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-token\") pod \"ca0043f1-fb84-4f38-b637-2315256e0c46\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " Nov 02 18:21:46.989670 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.989399 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-credential-socket\") pod \"ca0043f1-fb84-4f38-b637-2315256e0c46\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " Nov 02 18:21:46.989670 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.989531 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-envoy\") pod \"ca0043f1-fb84-4f38-b637-2315256e0c46\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " Nov 02 18:21:46.989670 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.989610 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-workload-socket\") pod \"ca0043f1-fb84-4f38-b637-2315256e0c46\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " Nov 02 18:21:46.990020 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.989783 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-workload-certs\") pod \"ca0043f1-fb84-4f38-b637-2315256e0c46\" (UID: \"ca0043f1-fb84-4f38-b637-2315256e0c46\") " Nov 02 18:21:46.991778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.990706 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781f545f-9659-4d21-9e53-85d4b82a7214-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:46.991778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.990740 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781f545f-9659-4d21-9e53-85d4b82a7214-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:46.991778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.990756 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:46.991778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.990773 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781f545f-9659-4d21-9e53-85d4b82a7214-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:46.997485 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.997395 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-workload-certs" pod "ca0043f1-fb84-4f38-b637-2315256e0c46" (UID: "ca0043f1-fb84-4f38-b637-2315256e0c46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:46.998145 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:46.998115 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca0043f1-fb84-4f38-b637-2315256e0c46-istiod-ca-cert" pod "ca0043f1-fb84-4f38-b637-2315256e0c46" (UID: "ca0043f1-fb84-4f38-b637-2315256e0c46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:47.002272 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.002240 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-data" pod "ca0043f1-fb84-4f38-b637-2315256e0c46" (UID: "ca0043f1-fb84-4f38-b637-2315256e0c46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:47.005481 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.005450 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-podinfo" pod "ca0043f1-fb84-4f38-b637-2315256e0c46" (UID: "ca0043f1-fb84-4f38-b637-2315256e0c46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:21:47.008326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.008211 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-credential-socket" pod "ca0043f1-fb84-4f38-b637-2315256e0c46" (UID: "ca0043f1-fb84-4f38-b637-2315256e0c46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:47.008906 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.008876 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0043f1-fb84-4f38-b637-2315256e0c46-kube-api-access-gk72s" pod "ca0043f1-fb84-4f38-b637-2315256e0c46" (UID: "ca0043f1-fb84-4f38-b637-2315256e0c46"). InnerVolumeSpecName "kube-api-access-gk72s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:47.012585 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.012553 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-workload-socket" pod "ca0043f1-fb84-4f38-b637-2315256e0c46" (UID: "ca0043f1-fb84-4f38-b637-2315256e0c46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:47.013030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.013005 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-envoy" pod "ca0043f1-fb84-4f38-b637-2315256e0c46" (UID: "ca0043f1-fb84-4f38-b637-2315256e0c46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:47.015092 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.015068 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-token" pod "ca0043f1-fb84-4f38-b637-2315256e0c46" (UID: "ca0043f1-fb84-4f38-b637-2315256e0c46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:47.060146 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.060116 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 18:21:47.119010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.113504 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-credential-socket\") pod \"5639111b-8dcd-4b4a-a845-f21825bda0d9\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " Nov 02 18:21:47.119010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.113558 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-workload-certs\") pod \"5639111b-8dcd-4b4a-a845-f21825bda0d9\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " Nov 02 18:21:47.119010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.113685 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-token\" (UniqueName: \"kubernetes.io/projected/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-token\") pod \"5639111b-8dcd-4b4a-a845-f21825bda0d9\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " Nov 02 18:21:47.119010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.113751 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-envoy\") pod \"5639111b-8dcd-4b4a-a845-f21825bda0d9\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " Nov 02 18:21:47.119010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.113866 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5639111b-8dcd-4b4a-a845-f21825bda0d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5639111b-8dcd-4b4a-a845-f21825bda0d9-istiod-ca-cert\") pod \"5639111b-8dcd-4b4a-a845-f21825bda0d9\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " Nov 02 18:21:47.119010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.113934 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5639111b-8dcd-4b4a-a845-f21825bda0d9-kube-api-access-tcsnc\" (UniqueName: \"kubernetes.io/projected/5639111b-8dcd-4b4a-a845-f21825bda0d9-kube-api-access-tcsnc\") pod \"5639111b-8dcd-4b4a-a845-f21825bda0d9\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " Nov 02 18:21:47.119010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.114099 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-workload-socket\") pod \"5639111b-8dcd-4b4a-a845-f21825bda0d9\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " Nov 02 18:21:47.119010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.114240 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-data\") pod \"5639111b-8dcd-4b4a-a845-f21825bda0d9\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " Nov 02 18:21:47.119010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.114328 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-podinfo\") pod \"5639111b-8dcd-4b4a-a845-f21825bda0d9\" (UID: \"5639111b-8dcd-4b4a-a845-f21825bda0d9\") " Nov 02 18:21:47.119010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.118250 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-workload-certs" pod "5639111b-8dcd-4b4a-a845-f21825bda0d9" (UID: "5639111b-8dcd-4b4a-a845-f21825bda0d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:47.124974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.121335 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:47.124974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.121366 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:47.124974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.121379 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk72s\" (UniqueName: \"kubernetes.io/projected/ca0043f1-fb84-4f38-b637-2315256e0c46-kube-api-access-gk72s\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:47.124974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.121393 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:47.124974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.121405 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:47.124974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.121417 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:47.124974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.121429 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:47.124974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.121442 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:47.124974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.121454 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0043f1-fb84-4f38-b637-2315256e0c46-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:47.124974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.121466 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0043f1-fb84-4f38-b637-2315256e0c46-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:47.124974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.121817 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-credential-socket" pod "5639111b-8dcd-4b4a-a845-f21825bda0d9" (UID: "5639111b-8dcd-4b4a-a845-f21825bda0d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:47.124974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.123638 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-workload-socket" pod "5639111b-8dcd-4b4a-a845-f21825bda0d9" (UID: "5639111b-8dcd-4b4a-a845-f21825bda0d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:47.124974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.123869 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-envoy" pod "5639111b-8dcd-4b4a-a845-f21825bda0d9" (UID: "5639111b-8dcd-4b4a-a845-f21825bda0d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:47.124974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.124245 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5639111b-8dcd-4b4a-a845-f21825bda0d9-istiod-ca-cert" pod "5639111b-8dcd-4b4a-a845-f21825bda0d9" (UID: "5639111b-8dcd-4b4a-a845-f21825bda0d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:47.124974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.124330 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-data" pod "5639111b-8dcd-4b4a-a845-f21825bda0d9" (UID: "5639111b-8dcd-4b4a-a845-f21825bda0d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:47.125750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.125074 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-podinfo" pod "5639111b-8dcd-4b4a-a845-f21825bda0d9" (UID: "5639111b-8dcd-4b4a-a845-f21825bda0d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:21:47.125750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.125236 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-token" pod "5639111b-8dcd-4b4a-a845-f21825bda0d9" (UID: "5639111b-8dcd-4b4a-a845-f21825bda0d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:47.125750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.125448 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5639111b-8dcd-4b4a-a845-f21825bda0d9-kube-api-access-tcsnc" pod "5639111b-8dcd-4b4a-a845-f21825bda0d9" (UID: "5639111b-8dcd-4b4a-a845-f21825bda0d9"). InnerVolumeSpecName "kube-api-access-tcsnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:47.229706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.227032 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:47.229706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.227066 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:47.229706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.227083 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:47.229706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.227097 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:47.229706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.227110 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:47.229706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.227123 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5639111b-8dcd-4b4a-a845-f21825bda0d9-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:47.229706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.227137 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5639111b-8dcd-4b4a-a845-f21825bda0d9-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:47.229706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.227150 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcsnc\" (UniqueName: \"kubernetes.io/projected/5639111b-8dcd-4b4a-a845-f21825bda0d9-kube-api-access-tcsnc\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:47.247443 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.247406 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd"] Nov 02 18:21:47.252264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.252232 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5lphd"] Nov 02 18:21:47.409444 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.409403 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:47.647733 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:21:47.647669 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:21:47.914387 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.914268 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" event={"ID":"5639111b-8dcd-4b4a-a845-f21825bda0d9","Type":"ContainerDied","Data":"0aa719953778a77b6edd72d3ff1cc4113ccb6262b13fd7c93bd23c7b56e0e50d"} Nov 02 18:21:47.914387 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.914294 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc" Nov 02 18:21:47.914387 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.914367 5549 scope.go:122] "RemoveContainer" containerID="7901a099298e95a847238d46d4ffe24d63475a2d9809b3a8fcee3557307afc66" Nov 02 18:21:47.918659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.918621 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 02 18:21:47.918814 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.918681 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" podUID="55fbc482-0099-414a-a9d0-87d3d0256de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 02 18:21:47.919996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.919931 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" event={"ID":"ca0043f1-fb84-4f38-b637-2315256e0c46","Type":"ContainerDied","Data":"291d829d2649c92b73ee7549731c3a123dc8bffa44e0b667a2db61be016c4204"} Nov 02 18:21:47.921527 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.920182 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t" Nov 02 18:21:47.933036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.933016 5549 scope.go:122] "RemoveContainer" containerID="52aa504a43c50ad1cd12420fa399ab6a81cdd39423c82e42c208f1297a4456ce" Nov 02 18:21:47.948815 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.948792 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc"] Nov 02 18:21:47.952520 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.952496 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-vbrfc"] Nov 02 18:21:47.964047 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.964023 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t"] Nov 02 18:21:47.968987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:47.968964 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-msm5t"] Nov 02 18:21:48.200566 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:48.200531 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:48.326699 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:48.326668 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5639111b-8dcd-4b4a-a845-f21825bda0d9" path="/var/lib/kubelet/pods/5639111b-8dcd-4b4a-a845-f21825bda0d9/volumes" Nov 02 18:21:48.327286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:48.327270 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="781f545f-9659-4d21-9e53-85d4b82a7214" path="/var/lib/kubelet/pods/781f545f-9659-4d21-9e53-85d4b82a7214/volumes" Nov 02 18:21:48.327749 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:48.327735 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca0043f1-fb84-4f38-b637-2315256e0c46" path="/var/lib/kubelet/pods/ca0043f1-fb84-4f38-b637-2315256e0c46/volumes" Nov 02 18:21:48.396160 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:48.396122 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q"] Nov 02 18:21:48.396347 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:48.396327 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" podUID="6eb2197f-704a-47aa-8b7b-0094baa86a96" containerName="istio-proxy" containerID="cri-o://50c5136b91fca96d8018cf7779eb32ef3abf4b742bfcc5edb50ff1951d00fea6" gracePeriod=30 Nov 02 18:21:48.893150 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:48.893103 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:49.432897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:49.432866 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:49.433900 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:49.433866 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:49.436238 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:49.436214 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:21:49.656751 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:49.656653 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:50.412981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:50.410755 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:51.041560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:51.041495 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:51.247588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:51.247557 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" Nov 02 18:21:51.509428 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:51.509387 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:51.649348 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:51.649310 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:51.785962 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:51.785840 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:52.004694 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:52.000228 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:52.303126 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:52.303090 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:52.650853 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:52.650771 5549 scope.go:122] "RemoveContainer" containerID="31819625a2d7ec8d9093073cbf61a5836615e99c76bab68e58a6f5637711dfed" Nov 02 18:21:52.904645 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:52.902567 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:53.088690 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.088624 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:53.557971 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.545081 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:53.733153 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.733120 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 18:21:53.815271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.815228 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-token\" (UniqueName: \"kubernetes.io/projected/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-token\") pod \"6eb2197f-704a-47aa-8b7b-0094baa86a96\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " Nov 02 18:21:53.815475 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.815308 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-workload-certs\") pod \"6eb2197f-704a-47aa-8b7b-0094baa86a96\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " Nov 02 18:21:53.815475 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.815354 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-envoy\") pod \"6eb2197f-704a-47aa-8b7b-0094baa86a96\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " Nov 02 18:21:53.815475 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.815406 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-credential-socket\") pod \"6eb2197f-704a-47aa-8b7b-0094baa86a96\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " Nov 02 18:21:53.815475 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.815452 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-podinfo\") pod \"6eb2197f-704a-47aa-8b7b-0094baa86a96\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " Nov 02 18:21:53.816072 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.816035 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-credential-socket" pod "6eb2197f-704a-47aa-8b7b-0094baa86a96" (UID: "6eb2197f-704a-47aa-8b7b-0094baa86a96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:53.816279 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.816252 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-workload-certs" pod "6eb2197f-704a-47aa-8b7b-0094baa86a96" (UID: "6eb2197f-704a-47aa-8b7b-0094baa86a96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:53.816372 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.815892 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb2197f-704a-47aa-8b7b-0094baa86a96-kube-api-access-zt6js\" (UniqueName: \"kubernetes.io/projected/6eb2197f-704a-47aa-8b7b-0094baa86a96-kube-api-access-zt6js\") pod \"6eb2197f-704a-47aa-8b7b-0094baa86a96\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " Nov 02 18:21:53.816449 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.816385 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-workload-socket\") pod \"6eb2197f-704a-47aa-8b7b-0094baa86a96\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " Nov 02 18:21:53.816620 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.816598 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eb2197f-704a-47aa-8b7b-0094baa86a96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb2197f-704a-47aa-8b7b-0094baa86a96-istiod-ca-cert\") pod \"6eb2197f-704a-47aa-8b7b-0094baa86a96\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " Nov 02 18:21:53.816699 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.816615 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-workload-socket" pod "6eb2197f-704a-47aa-8b7b-0094baa86a96" (UID: "6eb2197f-704a-47aa-8b7b-0094baa86a96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:53.816762 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.816732 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-data\") pod \"6eb2197f-704a-47aa-8b7b-0094baa86a96\" (UID: \"6eb2197f-704a-47aa-8b7b-0094baa86a96\") " Nov 02 18:21:53.816941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.816911 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb2197f-704a-47aa-8b7b-0094baa86a96-istiod-ca-cert" pod "6eb2197f-704a-47aa-8b7b-0094baa86a96" (UID: "6eb2197f-704a-47aa-8b7b-0094baa86a96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:53.817263 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.817240 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-data" pod "6eb2197f-704a-47aa-8b7b-0094baa86a96" (UID: "6eb2197f-704a-47aa-8b7b-0094baa86a96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:53.817716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.817692 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:53.817813 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.817723 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:53.817813 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.817742 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:53.817813 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.817760 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:53.817813 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.817776 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb2197f-704a-47aa-8b7b-0094baa86a96-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:53.819223 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.819185 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-token" pod "6eb2197f-704a-47aa-8b7b-0094baa86a96" (UID: "6eb2197f-704a-47aa-8b7b-0094baa86a96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:53.819450 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.819423 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-envoy" pod "6eb2197f-704a-47aa-8b7b-0094baa86a96" (UID: "6eb2197f-704a-47aa-8b7b-0094baa86a96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:53.819543 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.819469 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-podinfo" pod "6eb2197f-704a-47aa-8b7b-0094baa86a96" (UID: "6eb2197f-704a-47aa-8b7b-0094baa86a96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:21:53.820375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.820353 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb2197f-704a-47aa-8b7b-0094baa86a96-kube-api-access-zt6js" pod "6eb2197f-704a-47aa-8b7b-0094baa86a96" (UID: "6eb2197f-704a-47aa-8b7b-0094baa86a96"). InnerVolumeSpecName "kube-api-access-zt6js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:53.919880 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.919843 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:53.919880 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.919876 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:53.920112 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.919890 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb2197f-704a-47aa-8b7b-0094baa86a96-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:53.920112 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:53.919904 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt6js\" (UniqueName: \"kubernetes.io/projected/6eb2197f-704a-47aa-8b7b-0094baa86a96-kube-api-access-zt6js\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:54.036065 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.035974 5549 generic.go:356] "Generic (PLEG): container finished" podID="6eb2197f-704a-47aa-8b7b-0094baa86a96" containerID="50c5136b91fca96d8018cf7779eb32ef3abf4b742bfcc5edb50ff1951d00fea6" exitCode=0 Nov 02 18:21:54.036065 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.036018 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" event={"ID":"6eb2197f-704a-47aa-8b7b-0094baa86a96","Type":"ContainerDied","Data":"50c5136b91fca96d8018cf7779eb32ef3abf4b742bfcc5edb50ff1951d00fea6"} Nov 02 18:21:54.036278 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.036079 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" event={"ID":"6eb2197f-704a-47aa-8b7b-0094baa86a96","Type":"ContainerDied","Data":"23ed069a88afe0d187c24453bd6701f9289c839ff80b1b40de33e2920e836394"} Nov 02 18:21:54.036278 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.036102 5549 scope.go:122] "RemoveContainer" containerID="50c5136b91fca96d8018cf7779eb32ef3abf4b742bfcc5edb50ff1951d00fea6" Nov 02 18:21:54.036278 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.036029 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q" Nov 02 18:21:54.048179 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.048162 5549 scope.go:122] "RemoveContainer" containerID="50c5136b91fca96d8018cf7779eb32ef3abf4b742bfcc5edb50ff1951d00fea6" Nov 02 18:21:54.048463 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:21:54.048440 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50c5136b91fca96d8018cf7779eb32ef3abf4b742bfcc5edb50ff1951d00fea6\": container with ID starting with 50c5136b91fca96d8018cf7779eb32ef3abf4b742bfcc5edb50ff1951d00fea6 not found: ID does not exist" containerID="50c5136b91fca96d8018cf7779eb32ef3abf4b742bfcc5edb50ff1951d00fea6" Nov 02 18:21:54.048543 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.048469 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50c5136b91fca96d8018cf7779eb32ef3abf4b742bfcc5edb50ff1951d00fea6"} err="failed to get container status \"50c5136b91fca96d8018cf7779eb32ef3abf4b742bfcc5edb50ff1951d00fea6\": rpc error: code = NotFound desc = could not find container \"50c5136b91fca96d8018cf7779eb32ef3abf4b742bfcc5edb50ff1951d00fea6\": container with ID starting with 50c5136b91fca96d8018cf7779eb32ef3abf4b742bfcc5edb50ff1951d00fea6 not found: ID does not exist" Nov 02 18:21:54.066031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.066000 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q"] Nov 02 18:21:54.069413 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.069394 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-5465q"] Nov 02 18:21:54.179170 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.179123 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" podUID="4a853b47-0443-484f-b3a3-fdd13be51ba7" containerName="main" containerID="cri-o://0ff2564ba5a097315a4700e3c0ab392bbffbd036bfd1dc24393be2bbddb0d955" gracePeriod=45 Nov 02 18:21:54.357397 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.356011 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eb2197f-704a-47aa-8b7b-0094baa86a96" path="/var/lib/kubelet/pods/6eb2197f-704a-47aa-8b7b-0094baa86a96/volumes" Nov 02 18:21:54.357397 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.357040 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:54.529279 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.529251 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:21:54.657513 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.657423 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-home\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-home\") pod \"4a853b47-0443-484f-b3a3-fdd13be51ba7\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " Nov 02 18:21:54.658005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.657509 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-model-cache\") pod \"4a853b47-0443-484f-b3a3-fdd13be51ba7\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " Nov 02 18:21:54.658005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.657556 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-dshm\") pod \"4a853b47-0443-484f-b3a3-fdd13be51ba7\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " Nov 02 18:21:54.658005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.657620 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a853b47-0443-484f-b3a3-fdd13be51ba7-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a853b47-0443-484f-b3a3-fdd13be51ba7-tls-certs\") pod \"4a853b47-0443-484f-b3a3-fdd13be51ba7\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " Nov 02 18:21:54.658005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.657702 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-tmp-dir\") pod \"4a853b47-0443-484f-b3a3-fdd13be51ba7\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " Nov 02 18:21:54.658005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.657715 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-home" pod "4a853b47-0443-484f-b3a3-fdd13be51ba7" (UID: "4a853b47-0443-484f-b3a3-fdd13be51ba7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:54.658005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.657767 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-model-cache" pod "4a853b47-0443-484f-b3a3-fdd13be51ba7" (UID: "4a853b47-0443-484f-b3a3-fdd13be51ba7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:54.658005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.657866 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a853b47-0443-484f-b3a3-fdd13be51ba7-kube-api-access-fxr69\" (UniqueName: \"kubernetes.io/projected/4a853b47-0443-484f-b3a3-fdd13be51ba7-kube-api-access-fxr69\") pod \"4a853b47-0443-484f-b3a3-fdd13be51ba7\" (UID: \"4a853b47-0443-484f-b3a3-fdd13be51ba7\") " Nov 02 18:21:54.658005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.657992 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-tmp-dir" pod "4a853b47-0443-484f-b3a3-fdd13be51ba7" (UID: "4a853b47-0443-484f-b3a3-fdd13be51ba7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:54.658619 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.658599 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:54.658724 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.658625 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:54.658724 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.658657 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:54.659987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.659919 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a853b47-0443-484f-b3a3-fdd13be51ba7-tls-certs" pod "4a853b47-0443-484f-b3a3-fdd13be51ba7" (UID: "4a853b47-0443-484f-b3a3-fdd13be51ba7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:21:54.660260 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.660239 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a853b47-0443-484f-b3a3-fdd13be51ba7-kube-api-access-fxr69" pod "4a853b47-0443-484f-b3a3-fdd13be51ba7" (UID: "4a853b47-0443-484f-b3a3-fdd13be51ba7"). InnerVolumeSpecName "kube-api-access-fxr69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:54.660325 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.660265 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-dshm" pod "4a853b47-0443-484f-b3a3-fdd13be51ba7" (UID: "4a853b47-0443-484f-b3a3-fdd13be51ba7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:54.760118 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.760086 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxr69\" (UniqueName: \"kubernetes.io/projected/4a853b47-0443-484f-b3a3-fdd13be51ba7-kube-api-access-fxr69\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:54.760118 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.760114 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a853b47-0443-484f-b3a3-fdd13be51ba7-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:54.760118 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:54.760124 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a853b47-0443-484f-b3a3-fdd13be51ba7-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:55.028343 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:55.012591 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:55.050567 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:55.050479 5549 generic.go:356] "Generic (PLEG): container finished" podID="4a853b47-0443-484f-b3a3-fdd13be51ba7" containerID="0ff2564ba5a097315a4700e3c0ab392bbffbd036bfd1dc24393be2bbddb0d955" exitCode=0 Nov 02 18:21:55.050567 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:55.050537 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" Nov 02 18:21:55.050800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:55.050588 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" event={"ID":"4a853b47-0443-484f-b3a3-fdd13be51ba7","Type":"ContainerDied","Data":"0ff2564ba5a097315a4700e3c0ab392bbffbd036bfd1dc24393be2bbddb0d955"} Nov 02 18:21:55.050800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:55.050643 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj" event={"ID":"4a853b47-0443-484f-b3a3-fdd13be51ba7","Type":"ContainerDied","Data":"b8a02f66f54fb39f38ba53a972ff32c61ef51e2261624d1a197af3476bc41809"} Nov 02 18:21:55.050800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:55.050666 5549 scope.go:122] "RemoveContainer" containerID="0ff2564ba5a097315a4700e3c0ab392bbffbd036bfd1dc24393be2bbddb0d955" Nov 02 18:21:55.072021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:55.070557 5549 scope.go:122] "RemoveContainer" containerID="0ff2564ba5a097315a4700e3c0ab392bbffbd036bfd1dc24393be2bbddb0d955" Nov 02 18:21:55.072021 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:21:55.071301 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff2564ba5a097315a4700e3c0ab392bbffbd036bfd1dc24393be2bbddb0d955\": container with ID starting with 0ff2564ba5a097315a4700e3c0ab392bbffbd036bfd1dc24393be2bbddb0d955 not found: ID does not exist" containerID="0ff2564ba5a097315a4700e3c0ab392bbffbd036bfd1dc24393be2bbddb0d955" Nov 02 18:21:55.072021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:55.071332 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff2564ba5a097315a4700e3c0ab392bbffbd036bfd1dc24393be2bbddb0d955"} err="failed to get container status \"0ff2564ba5a097315a4700e3c0ab392bbffbd036bfd1dc24393be2bbddb0d955\": rpc error: code = NotFound desc = could not find container \"0ff2564ba5a097315a4700e3c0ab392bbffbd036bfd1dc24393be2bbddb0d955\": container with ID starting with 0ff2564ba5a097315a4700e3c0ab392bbffbd036bfd1dc24393be2bbddb0d955 not found: ID does not exist" Nov 02 18:21:55.083649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:55.083619 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj"] Nov 02 18:21:55.087784 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:55.087752 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0b0610/test-model-0b0610-kserve-684695468f-kfhfj"] Nov 02 18:21:55.658184 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:55.658147 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:56.324300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:56.324268 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a853b47-0443-484f-b3a3-fdd13be51ba7" path="/var/lib/kubelet/pods/4a853b47-0443-484f-b3a3-fdd13be51ba7/volumes" Nov 02 18:21:56.438206 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:56.438149 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:57.123865 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:57.123826 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:57.871719 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:57.871683 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:58.694340 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:58.694307 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:58.838217 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:58.838117 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:59.026127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:59.026094 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:21:59.964838 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:21:59.964803 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:00.143646 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:00.133523 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:00.771265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:00.771210 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:01.323629 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.323590 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k"] Nov 02 18:22:01.325051 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.325024 5549 cpu_manager.go:414] "containerMap: removing container" podUID="781f545f-9659-4d21-9e53-85d4b82a7214" containerName="istio-proxy" Nov 02 18:22:01.326348 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326319 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="781f545f-9659-4d21-9e53-85d4b82a7214" containerName="istio-proxy" Nov 02 18:22:01.326464 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326401 5549 cpu_manager.go:414] "containerMap: removing container" podUID="5639111b-8dcd-4b4a-a845-f21825bda0d9" containerName="istio-proxy" Nov 02 18:22:01.326464 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326411 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5639111b-8dcd-4b4a-a845-f21825bda0d9" containerName="istio-proxy" Nov 02 18:22:01.326464 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326438 5549 cpu_manager.go:414] "containerMap: removing container" podUID="b20603e9-d7ea-4bac-bf54-c1c59d343fa8" containerName="istio-proxy" Nov 02 18:22:01.326464 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326445 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b20603e9-d7ea-4bac-bf54-c1c59d343fa8" containerName="istio-proxy" Nov 02 18:22:01.326464 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326461 5549 cpu_manager.go:414] "containerMap: removing container" podUID="ca0043f1-fb84-4f38-b637-2315256e0c46" containerName="istio-proxy" Nov 02 18:22:01.326647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326469 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0043f1-fb84-4f38-b637-2315256e0c46" containerName="istio-proxy" Nov 02 18:22:01.326647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326482 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7990f1b0-ff87-4445-8a17-93052b4c9586" containerName="main" Nov 02 18:22:01.326647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326489 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7990f1b0-ff87-4445-8a17-93052b4c9586" containerName="main" Nov 02 18:22:01.326647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326529 5549 cpu_manager.go:414] "containerMap: removing container" podUID="468e1807-5ca2-4bef-8674-0879f2b3a77e" containerName="istio-proxy" Nov 02 18:22:01.326647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326536 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468e1807-5ca2-4bef-8674-0879f2b3a77e" containerName="istio-proxy" Nov 02 18:22:01.326647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326553 5549 cpu_manager.go:414] "containerMap: removing container" podUID="4a853b47-0443-484f-b3a3-fdd13be51ba7" containerName="main" Nov 02 18:22:01.326647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326560 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a853b47-0443-484f-b3a3-fdd13be51ba7" containerName="main" Nov 02 18:22:01.326647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326589 5549 cpu_manager.go:414] "containerMap: removing container" podUID="6eb2197f-704a-47aa-8b7b-0094baa86a96" containerName="istio-proxy" Nov 02 18:22:01.326647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326596 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb2197f-704a-47aa-8b7b-0094baa86a96" containerName="istio-proxy" Nov 02 18:22:01.326933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326816 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="781f545f-9659-4d21-9e53-85d4b82a7214" containerName="istio-proxy" Nov 02 18:22:01.326933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326845 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5639111b-8dcd-4b4a-a845-f21825bda0d9" containerName="istio-proxy" Nov 02 18:22:01.326933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326858 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="b20603e9-d7ea-4bac-bf54-c1c59d343fa8" containerName="istio-proxy" Nov 02 18:22:01.326933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326872 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0043f1-fb84-4f38-b637-2315256e0c46" containerName="istio-proxy" Nov 02 18:22:01.326933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326883 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7990f1b0-ff87-4445-8a17-93052b4c9586" containerName="main" Nov 02 18:22:01.326933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326896 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="468e1807-5ca2-4bef-8674-0879f2b3a77e" containerName="istio-proxy" Nov 02 18:22:01.326933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326916 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a853b47-0443-484f-b3a3-fdd13be51ba7" containerName="main" Nov 02 18:22:01.326933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.326931 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb2197f-704a-47aa-8b7b-0094baa86a96" containerName="istio-proxy" Nov 02 18:22:01.343388 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.343355 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k"] Nov 02 18:22:01.343548 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.343480 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" Nov 02 18:22:01.348304 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.348275 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2966e7-serving-cert\"" Nov 02 18:22:01.414236 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.414201 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:01.416815 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.416785 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/216112c3-41c7-45f6-9553-628fa92c38cb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k\" (UID: \"216112c3-41c7-45f6-9553-628fa92c38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" Nov 02 18:22:01.417003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.416987 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/216112c3-41c7-45f6-9553-628fa92c38cb-maas-api-tls\") pod \"maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k\" (UID: \"216112c3-41c7-45f6-9553-628fa92c38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" Nov 02 18:22:01.417292 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.417269 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/216112c3-41c7-45f6-9553-628fa92c38cb-metrics-certs\") pod \"maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k\" (UID: \"216112c3-41c7-45f6-9553-628fa92c38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" Nov 02 18:22:01.417517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.417497 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2ghz\" (UniqueName: \"kubernetes.io/projected/216112c3-41c7-45f6-9553-628fa92c38cb-kube-api-access-x2ghz\") pod \"maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k\" (UID: \"216112c3-41c7-45f6-9553-628fa92c38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" Nov 02 18:22:01.518799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.518766 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/216112c3-41c7-45f6-9553-628fa92c38cb-metrics-certs\") pod \"maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k\" (UID: \"216112c3-41c7-45f6-9553-628fa92c38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" Nov 02 18:22:01.519036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.518896 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2ghz\" (UniqueName: \"kubernetes.io/projected/216112c3-41c7-45f6-9553-628fa92c38cb-kube-api-access-x2ghz\") pod \"maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k\" (UID: \"216112c3-41c7-45f6-9553-628fa92c38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" Nov 02 18:22:01.519036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.518946 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/216112c3-41c7-45f6-9553-628fa92c38cb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k\" (UID: \"216112c3-41c7-45f6-9553-628fa92c38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" Nov 02 18:22:01.519179 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.519062 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/216112c3-41c7-45f6-9553-628fa92c38cb-maas-api-tls\") pod \"maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k\" (UID: \"216112c3-41c7-45f6-9553-628fa92c38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" Nov 02 18:22:01.519637 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.519604 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/216112c3-41c7-45f6-9553-628fa92c38cb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k\" (UID: \"216112c3-41c7-45f6-9553-628fa92c38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" Nov 02 18:22:01.521447 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.521420 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/216112c3-41c7-45f6-9553-628fa92c38cb-metrics-certs\") pod \"maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k\" (UID: \"216112c3-41c7-45f6-9553-628fa92c38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" Nov 02 18:22:01.521509 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.521456 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/216112c3-41c7-45f6-9553-628fa92c38cb-maas-api-tls\") pod \"maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k\" (UID: \"216112c3-41c7-45f6-9553-628fa92c38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" Nov 02 18:22:01.569974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.568925 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:01.589630 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.589547 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2ghz\" (UniqueName: \"kubernetes.io/projected/216112c3-41c7-45f6-9553-628fa92c38cb-kube-api-access-x2ghz\") pod \"maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k\" (UID: \"216112c3-41c7-45f6-9553-628fa92c38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" Nov 02 18:22:01.665702 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:01.665671 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" Nov 02 18:22:02.023466 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:22:02.023433 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216112c3_41c7_45f6_9553_628fa92c38cb.slice/crio-181d7b0f68826fe2d4165fe8188aa7b8a9e72c156b0af689818db98059c13f66 WatchSource:0}: Error finding container 181d7b0f68826fe2d4165fe8188aa7b8a9e72c156b0af689818db98059c13f66: Status 404 returned error can't find the container with id 181d7b0f68826fe2d4165fe8188aa7b8a9e72c156b0af689818db98059c13f66 Nov 02 18:22:02.023466 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:02.023457 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k"] Nov 02 18:22:02.169027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:02.168980 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" event={"ID":"216112c3-41c7-45f6-9553-628fa92c38cb","Type":"ContainerStarted","Data":"181d7b0f68826fe2d4165fe8188aa7b8a9e72c156b0af689818db98059c13f66"} Nov 02 18:22:02.293335 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:02.293254 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:02.423319 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:02.423286 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:03.031974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:03.031165 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:03.178574 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:03.178530 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" event={"ID":"216112c3-41c7-45f6-9553-628fa92c38cb","Type":"ContainerStarted","Data":"6ee759cc3da469533c1d78af08aa050398e4e010bc98d350d56afbd46d633dc5"} Nov 02 18:22:03.178766 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:03.178616 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" Nov 02 18:22:03.855011 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:03.854880 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:04.512539 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:04.512500 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:05.134118 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:05.134075 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:05.838642 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:05.838605 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:05.989189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:05.989154 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:06.509049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:06.509013 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:07.155981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:07.143479 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:07.948021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:07.947899 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:08.651716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:08.651680 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:09.192112 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:09.192079 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" Nov 02 18:22:09.214616 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:09.214559 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" podStartSLOduration=7.755815436 podStartE2EDuration="8.214545421s" totalImagesPullingTime="458.729985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:22:03.201449305 +0000 UTC m=+2894.989954832" watchObservedRunningTime="2026-11-02 18:22:09.214545421 +0000 UTC m=+2901.003050950" Nov 02 18:22:09.350186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:09.350147 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:10.073620 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:10.073569 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:10.764683 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:10.764648 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:11.572422 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:11.572345 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:12.232985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:12.225224 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:12.801718 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:12.801683 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:13.076583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:13.076469 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:13.959597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:13.959481 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:14.644431 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:14.644389 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:15.266233 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:15.266188 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:15.945980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:15.942231 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:16.607596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:16.606317 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:17.346464 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:17.346321 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:17.590702 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:22:17.590660 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:22:18.024747 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:18.024709 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:18.652457 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:18.652420 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:19.339039 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:19.339002 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:19.977972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:19.977920 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:20.688516 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:20.688475 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:21.014393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:21.014350 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj"] Nov 02 18:22:21.118985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:21.110157 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:21.727323 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:21.727270 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:21.934091 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:21.934054 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:22.072375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:22.072283 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:22.353583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:22.353500 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:22.621662 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:22.621548 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:22.755584 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:22.755494 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:23.499729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:23.499688 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:23.649407 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:23.649375 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:23.785099 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:23.785016 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:24.253522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:24.253482 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:24.512812 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:24.512710 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:25.552976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:25.549248 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:26.390161 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:26.390129 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:26.547236 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:26.547190 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:26.759789 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:26.759740 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:27.174229 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:27.174152 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:27.322514 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:27.322477 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:27.517610 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:27.517508 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:28.491044 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:28.491008 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:29.216804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:29.216757 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:29.907429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:29.907369 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:30.788184 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:30.788099 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:31.308643 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:31.308603 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:31.444692 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:31.444646 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:31.625615 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:31.625526 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:32.202063 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:32.202021 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:32.621335 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:32.621234 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:32.851210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:32.851175 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:32.971783 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:32.971753 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl"] Nov 02 18:22:32.976943 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:32.976923 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb"] Nov 02 18:22:32.977071 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:32.977052 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" Nov 02 18:22:32.981787 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:32.981763 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb" Nov 02 18:22:32.983226 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:32.983201 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-cc0b41\"" Nov 02 18:22:32.987261 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:32.987237 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl"] Nov 02 18:22:33.005752 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.005726 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb"] Nov 02 18:22:33.006403 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.006374 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:33.112383 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.107490 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kfl5\" (UniqueName: \"kubernetes.io/projected/ff3423b0-fb1a-4082-8227-bd154ed372d7-kube-api-access-9kfl5\") pod \"payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb\" (UID: \"ff3423b0-fb1a-4082-8227-bd154ed372d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb" Nov 02 18:22:33.112383 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.107720 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3423b0-fb1a-4082-8227-bd154ed372d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb\" (UID: \"ff3423b0-fb1a-4082-8227-bd154ed372d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb" Nov 02 18:22:33.112383 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.107912 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d87vw\" (UniqueName: \"kubernetes.io/projected/5b44de0c-f730-47cd-80e0-4f6b88f5b1fc-kube-api-access-d87vw\") pod \"payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl\" (UID: \"5b44de0c-f730-47cd-80e0-4f6b88f5b1fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" Nov 02 18:22:33.112383 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.108121 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b44de0c-f730-47cd-80e0-4f6b88f5b1fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl\" (UID: \"5b44de0c-f730-47cd-80e0-4f6b88f5b1fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" Nov 02 18:22:33.142704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.142665 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:33.220974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.216381 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3423b0-fb1a-4082-8227-bd154ed372d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb\" (UID: \"ff3423b0-fb1a-4082-8227-bd154ed372d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb" Nov 02 18:22:33.220974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.216582 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d87vw\" (UniqueName: \"kubernetes.io/projected/5b44de0c-f730-47cd-80e0-4f6b88f5b1fc-kube-api-access-d87vw\") pod \"payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl\" (UID: \"5b44de0c-f730-47cd-80e0-4f6b88f5b1fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" Nov 02 18:22:33.220974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.216761 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b44de0c-f730-47cd-80e0-4f6b88f5b1fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl\" (UID: \"5b44de0c-f730-47cd-80e0-4f6b88f5b1fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" Nov 02 18:22:33.220974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.216845 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kfl5\" (UniqueName: \"kubernetes.io/projected/ff3423b0-fb1a-4082-8227-bd154ed372d7-kube-api-access-9kfl5\") pod \"payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb\" (UID: \"ff3423b0-fb1a-4082-8227-bd154ed372d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb" Nov 02 18:22:33.220974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.218279 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3423b0-fb1a-4082-8227-bd154ed372d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb\" (UID: \"ff3423b0-fb1a-4082-8227-bd154ed372d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb" Nov 02 18:22:33.227550 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.227473 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b44de0c-f730-47cd-80e0-4f6b88f5b1fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl\" (UID: \"5b44de0c-f730-47cd-80e0-4f6b88f5b1fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" Nov 02 18:22:33.261880 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.261743 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kfl5\" (UniqueName: \"kubernetes.io/projected/ff3423b0-fb1a-4082-8227-bd154ed372d7-kube-api-access-9kfl5\") pod \"payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb\" (UID: \"ff3423b0-fb1a-4082-8227-bd154ed372d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb" Nov 02 18:22:33.285108 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.282615 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d87vw\" (UniqueName: \"kubernetes.io/projected/5b44de0c-f730-47cd-80e0-4f6b88f5b1fc-kube-api-access-d87vw\") pod \"payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl\" (UID: \"5b44de0c-f730-47cd-80e0-4f6b88f5b1fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" Nov 02 18:22:33.305975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.305493 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" Nov 02 18:22:33.312521 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.312317 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb" Nov 02 18:22:33.622564 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:22:33.622519 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b44de0c_f730_47cd_80e0_4f6b88f5b1fc.slice/crio-9c95c6620c2dfa33badb0201a5d3a8d15bf461cb0da88f7ac68613c3291c062b WatchSource:0}: Error finding container 9c95c6620c2dfa33badb0201a5d3a8d15bf461cb0da88f7ac68613c3291c062b: Status 404 returned error can't find the container with id 9c95c6620c2dfa33badb0201a5d3a8d15bf461cb0da88f7ac68613c3291c062b Nov 02 18:22:33.624001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.623972 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl"] Nov 02 18:22:33.640997 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:22:33.640932 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3423b0_fb1a_4082_8227_bd154ed372d7.slice/crio-901ea22623f8600608414d0324595c93b6caf05c8438cef9a250d7ab69d38d38 WatchSource:0}: Error finding container 901ea22623f8600608414d0324595c93b6caf05c8438cef9a250d7ab69d38d38: Status 404 returned error can't find the container with id 901ea22623f8600608414d0324595c93b6caf05c8438cef9a250d7ab69d38d38 Nov 02 18:22:33.641830 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.641804 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb"] Nov 02 18:22:33.746667 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:33.743692 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:34.530786 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:34.530744 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb" event={"ID":"ff3423b0-fb1a-4082-8227-bd154ed372d7","Type":"ContainerStarted","Data":"2c48b63b9dcff7b42388b9cca89f4d46bf85d5c8f98a27b0ad6269f75e873faa"} Nov 02 18:22:34.531061 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:34.530810 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb" event={"ID":"ff3423b0-fb1a-4082-8227-bd154ed372d7","Type":"ContainerStarted","Data":"901ea22623f8600608414d0324595c93b6caf05c8438cef9a250d7ab69d38d38"} Nov 02 18:22:34.531061 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:34.530985 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb" Nov 02 18:22:34.532758 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:34.532726 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" event={"ID":"5b44de0c-f730-47cd-80e0-4f6b88f5b1fc","Type":"ContainerStarted","Data":"68d3c4ec5094b9c8b1a25acc1dbe99052485c54143fb006ab7218b596d0a7ca6"} Nov 02 18:22:34.532894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:34.532775 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" event={"ID":"5b44de0c-f730-47cd-80e0-4f6b88f5b1fc","Type":"ContainerStarted","Data":"9c95c6620c2dfa33badb0201a5d3a8d15bf461cb0da88f7ac68613c3291c062b"} Nov 02 18:22:34.532894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:34.532823 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" Nov 02 18:22:34.592976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:34.584022 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:34.606871 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:34.605670 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb" podStartSLOduration=2.60564865 podStartE2EDuration="2.60564865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:22:34.584282254 +0000 UTC m=+2926.372787793" watchObservedRunningTime="2026-11-02 18:22:34.60564865 +0000 UTC m=+2926.394154180" Nov 02 18:22:34.608727 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:34.608677 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" podStartSLOduration=2.608659495 podStartE2EDuration="2.608659495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:22:34.601210265 +0000 UTC m=+2926.389715793" watchObservedRunningTime="2026-11-02 18:22:34.608659495 +0000 UTC m=+2926.397165024" Nov 02 18:22:35.518974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:35.517032 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:36.047250 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.047184 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" podUID="63194adc-5a21-49ca-a101-da5038c43036" containerName="main" containerID="cri-o://4d6b1ba35e2c08d8a590a49727a56a36a8fd669ad90ddec6fc735a36f8e41bc1" gracePeriod=45 Nov 02 18:22:36.396903 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.396863 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:22:36.415449 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.415410 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63194adc-5a21-49ca-a101-da5038c43036-tls-certs\" (UniqueName: \"kubernetes.io/secret/63194adc-5a21-49ca-a101-da5038c43036-tls-certs\") pod \"63194adc-5a21-49ca-a101-da5038c43036\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " Nov 02 18:22:36.415627 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.415616 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-dshm\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-dshm\") pod \"63194adc-5a21-49ca-a101-da5038c43036\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " Nov 02 18:22:36.415707 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.415690 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-tmp-dir\") pod \"63194adc-5a21-49ca-a101-da5038c43036\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " Nov 02 18:22:36.415845 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.415823 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-home\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-home\") pod \"63194adc-5a21-49ca-a101-da5038c43036\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " Nov 02 18:22:36.415977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.415884 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-model-cache\") pod \"63194adc-5a21-49ca-a101-da5038c43036\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " Nov 02 18:22:36.415977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.415916 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63194adc-5a21-49ca-a101-da5038c43036-kube-api-access-6hrgk\" (UniqueName: \"kubernetes.io/projected/63194adc-5a21-49ca-a101-da5038c43036-kube-api-access-6hrgk\") pod \"63194adc-5a21-49ca-a101-da5038c43036\" (UID: \"63194adc-5a21-49ca-a101-da5038c43036\") " Nov 02 18:22:36.416125 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.415967 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-tmp-dir" pod "63194adc-5a21-49ca-a101-da5038c43036" (UID: "63194adc-5a21-49ca-a101-da5038c43036"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:22:36.416223 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.416196 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-home" pod "63194adc-5a21-49ca-a101-da5038c43036" (UID: "63194adc-5a21-49ca-a101-da5038c43036"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:22:36.416223 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.416214 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-model-cache" pod "63194adc-5a21-49ca-a101-da5038c43036" (UID: "63194adc-5a21-49ca-a101-da5038c43036"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:22:36.417068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.416756 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:36.417068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.416785 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:36.417068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.416801 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:36.418644 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.418609 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-dshm" pod "63194adc-5a21-49ca-a101-da5038c43036" (UID: "63194adc-5a21-49ca-a101-da5038c43036"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:22:36.420600 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.420064 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63194adc-5a21-49ca-a101-da5038c43036-tls-certs" pod "63194adc-5a21-49ca-a101-da5038c43036" (UID: "63194adc-5a21-49ca-a101-da5038c43036"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:22:36.420600 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.420133 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63194adc-5a21-49ca-a101-da5038c43036-kube-api-access-6hrgk" pod "63194adc-5a21-49ca-a101-da5038c43036" (UID: "63194adc-5a21-49ca-a101-da5038c43036"). InnerVolumeSpecName "kube-api-access-6hrgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:22:36.518517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.518477 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hrgk\" (UniqueName: \"kubernetes.io/projected/63194adc-5a21-49ca-a101-da5038c43036-kube-api-access-6hrgk\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:36.518517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.518510 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63194adc-5a21-49ca-a101-da5038c43036-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:36.518517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.518524 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63194adc-5a21-49ca-a101-da5038c43036-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:36.562292 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.562262 5549 generic.go:356] "Generic (PLEG): container finished" podID="63194adc-5a21-49ca-a101-da5038c43036" containerID="4d6b1ba35e2c08d8a590a49727a56a36a8fd669ad90ddec6fc735a36f8e41bc1" exitCode=0 Nov 02 18:22:36.562757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.562310 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" Nov 02 18:22:36.562757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.562345 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" event={"ID":"63194adc-5a21-49ca-a101-da5038c43036","Type":"ContainerDied","Data":"4d6b1ba35e2c08d8a590a49727a56a36a8fd669ad90ddec6fc735a36f8e41bc1"} Nov 02 18:22:36.562757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.562390 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj" event={"ID":"63194adc-5a21-49ca-a101-da5038c43036","Type":"ContainerDied","Data":"4518963ab92cac3527593a18eb25bf20396544686ea758609cafc3cd7ee10e19"} Nov 02 18:22:36.562757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.562412 5549 scope.go:122] "RemoveContainer" containerID="4d6b1ba35e2c08d8a590a49727a56a36a8fd669ad90ddec6fc735a36f8e41bc1" Nov 02 18:22:36.578137 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.578118 5549 scope.go:122] "RemoveContainer" containerID="4d6b1ba35e2c08d8a590a49727a56a36a8fd669ad90ddec6fc735a36f8e41bc1" Nov 02 18:22:36.578416 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:22:36.578394 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d6b1ba35e2c08d8a590a49727a56a36a8fd669ad90ddec6fc735a36f8e41bc1\": container with ID starting with 4d6b1ba35e2c08d8a590a49727a56a36a8fd669ad90ddec6fc735a36f8e41bc1 not found: ID does not exist" containerID="4d6b1ba35e2c08d8a590a49727a56a36a8fd669ad90ddec6fc735a36f8e41bc1" Nov 02 18:22:36.578467 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.578427 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d6b1ba35e2c08d8a590a49727a56a36a8fd669ad90ddec6fc735a36f8e41bc1"} err="failed to get container status \"4d6b1ba35e2c08d8a590a49727a56a36a8fd669ad90ddec6fc735a36f8e41bc1\": rpc error: code = NotFound desc = could not find container \"4d6b1ba35e2c08d8a590a49727a56a36a8fd669ad90ddec6fc735a36f8e41bc1\": container with ID starting with 4d6b1ba35e2c08d8a590a49727a56a36a8fd669ad90ddec6fc735a36f8e41bc1 not found: ID does not exist" Nov 02 18:22:36.596303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.596270 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj"] Nov 02 18:22:36.600288 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.600226 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-22dfc1/test-model-22dfc1-kserve-698cd6fc87-xdwdj"] Nov 02 18:22:36.706993 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:36.706929 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:37.851808 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:37.848078 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:38.325914 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:38.325875 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63194adc-5a21-49ca-a101-da5038c43036" path="/var/lib/kubelet/pods/63194adc-5a21-49ca-a101-da5038c43036/volumes" Nov 02 18:22:38.342169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:38.342138 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:38.511981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:38.504774 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:38.887992 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:38.887645 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:39.384765 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:39.384475 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:40.081236 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.081201 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:40.515267 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.515233 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv"] Nov 02 18:22:40.516356 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.516332 5549 cpu_manager.go:414] "containerMap: removing container" podUID="63194adc-5a21-49ca-a101-da5038c43036" containerName="main" Nov 02 18:22:40.516356 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.516355 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63194adc-5a21-49ca-a101-da5038c43036" containerName="main" Nov 02 18:22:40.516555 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.516545 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="63194adc-5a21-49ca-a101-da5038c43036" containerName="main" Nov 02 18:22:40.522683 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.522662 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.530862 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.530782 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n"] Nov 02 18:22:40.537215 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.537191 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv"] Nov 02 18:22:40.537360 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.537279 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.552136 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.548531 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n"] Nov 02 18:22:40.572845 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.572814 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7066250e-b36d-4065-a5fa-265c358a1257-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.573049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.572961 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4740514e-6d04-468b-930f-8be299ef0f4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.573049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.573022 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxncx\" (UniqueName: \"kubernetes.io/projected/4740514e-6d04-468b-930f-8be299ef0f4e-kube-api-access-hxncx\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.573205 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.573093 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl87r\" (UniqueName: \"kubernetes.io/projected/7066250e-b36d-4065-a5fa-265c358a1257-kube-api-access-nl87r\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.573205 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.573122 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4740514e-6d04-468b-930f-8be299ef0f4e-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.573205 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.573189 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4740514e-6d04-468b-930f-8be299ef0f4e-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.573378 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.573252 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7066250e-b36d-4065-a5fa-265c358a1257-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.574618 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.574002 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4740514e-6d04-468b-930f-8be299ef0f4e-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.574741 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.574185 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4740514e-6d04-468b-930f-8be299ef0f4e-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.574741 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.574711 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4740514e-6d04-468b-930f-8be299ef0f4e-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.574869 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.574817 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.574943 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.574866 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.574943 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.574906 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4740514e-6d04-468b-930f-8be299ef0f4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.575083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.574971 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.575083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.575012 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4740514e-6d04-468b-930f-8be299ef0f4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.575083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.575041 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7066250e-b36d-4065-a5fa-265c358a1257-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.575083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.575073 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.575295 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.575101 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.676931 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.676892 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl87r\" (UniqueName: \"kubernetes.io/projected/7066250e-b36d-4065-a5fa-265c358a1257-kube-api-access-nl87r\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.676931 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.676930 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4740514e-6d04-468b-930f-8be299ef0f4e-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.677217 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677049 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4740514e-6d04-468b-930f-8be299ef0f4e-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.677217 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677109 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7066250e-b36d-4065-a5fa-265c358a1257-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.677217 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677148 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4740514e-6d04-468b-930f-8be299ef0f4e-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.677402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677255 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4740514e-6d04-468b-930f-8be299ef0f4e-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.677402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677301 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4740514e-6d04-468b-930f-8be299ef0f4e-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.677402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677392 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.677581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677438 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.677581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677462 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4740514e-6d04-468b-930f-8be299ef0f4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.677581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677471 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4740514e-6d04-468b-930f-8be299ef0f4e-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.677581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677503 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.677581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677513 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4740514e-6d04-468b-930f-8be299ef0f4e-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.677581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677541 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4740514e-6d04-468b-930f-8be299ef0f4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.677581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677569 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4740514e-6d04-468b-930f-8be299ef0f4e-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.678027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677573 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7066250e-b36d-4065-a5fa-265c358a1257-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.678027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677626 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.678027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677655 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.678027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677693 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7066250e-b36d-4065-a5fa-265c358a1257-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.678027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677720 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4740514e-6d04-468b-930f-8be299ef0f4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.678027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677772 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxncx\" (UniqueName: \"kubernetes.io/projected/4740514e-6d04-468b-930f-8be299ef0f4e-kube-api-access-hxncx\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.678027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677784 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.678027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677813 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.678027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677867 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4740514e-6d04-468b-930f-8be299ef0f4e-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.678027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.677904 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7066250e-b36d-4065-a5fa-265c358a1257-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.678547 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.678218 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.678547 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.678306 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.678709 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.678658 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4740514e-6d04-468b-930f-8be299ef0f4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.680209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.680181 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4740514e-6d04-468b-930f-8be299ef0f4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.680209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.680196 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7066250e-b36d-4065-a5fa-265c358a1257-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.680209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.680207 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.680709 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.680687 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4740514e-6d04-468b-930f-8be299ef0f4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.686604 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.686577 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4740514e-6d04-468b-930f-8be299ef0f4e-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.686703 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.686632 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl87r\" (UniqueName: \"kubernetes.io/projected/7066250e-b36d-4065-a5fa-265c358a1257-kube-api-access-nl87r\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.687030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.686942 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7066250e-b36d-4065-a5fa-265c358a1257-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-wgjhv\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.687330 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.687306 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxncx\" (UniqueName: \"kubernetes.io/projected/4740514e-6d04-468b-930f-8be299ef0f4e-kube-api-access-hxncx\") pod \"maas-default-gateway-openshift-default-65c448b547-xsg9n\" (UID: \"4740514e-6d04-468b-930f-8be299ef0f4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:40.871703 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.871622 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:40.888900 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.888865 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:41.001719 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:40.999500 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:41.176309 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:22:41.176276 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7066250e_b36d_4065_a5fa_265c358a1257.slice/crio-85f9d862895585a0f601e8a21949aa876a34b341712ded022c2d6bf469e3d0b6 WatchSource:0}: Error finding container 85f9d862895585a0f601e8a21949aa876a34b341712ded022c2d6bf469e3d0b6: Status 404 returned error can't find the container with id 85f9d862895585a0f601e8a21949aa876a34b341712ded022c2d6bf469e3d0b6 Nov 02 18:22:41.176722 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:41.176690 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv"] Nov 02 18:22:41.179887 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:41.179848 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:22:41.180039 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:41.179936 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:22:41.180039 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:41.179986 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:22:41.242824 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:41.242794 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n"] Nov 02 18:22:41.246676 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:22:41.246649 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4740514e_6d04_468b_930f_8be299ef0f4e.slice/crio-9e35402749406d818126b73f3a39f2702e6ae366d3e268700769cb5c66dc9dbf WatchSource:0}: Error finding container 9e35402749406d818126b73f3a39f2702e6ae366d3e268700769cb5c66dc9dbf: Status 404 returned error can't find the container with id 9e35402749406d818126b73f3a39f2702e6ae366d3e268700769cb5c66dc9dbf Nov 02 18:22:41.249752 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:41.249718 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:22:41.249870 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:41.249805 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:22:41.249870 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:41.249833 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:22:41.677725 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:41.677682 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" event={"ID":"4740514e-6d04-468b-930f-8be299ef0f4e","Type":"ContainerStarted","Data":"756438c352d9b87ab81199126cae8cac0b4c73beaf62285e88b309c65015f5fe"} Nov 02 18:22:41.677921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:41.677788 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" event={"ID":"4740514e-6d04-468b-930f-8be299ef0f4e","Type":"ContainerStarted","Data":"9e35402749406d818126b73f3a39f2702e6ae366d3e268700769cb5c66dc9dbf"} Nov 02 18:22:41.679888 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:41.679859 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" event={"ID":"7066250e-b36d-4065-a5fa-265c358a1257","Type":"ContainerStarted","Data":"afd5eb9961584b3f0c55e98ffc21c95b2f11bc73130d890aa633a3d413cb3628"} Nov 02 18:22:41.680045 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:41.679910 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" event={"ID":"7066250e-b36d-4065-a5fa-265c358a1257","Type":"ContainerStarted","Data":"85f9d862895585a0f601e8a21949aa876a34b341712ded022c2d6bf469e3d0b6"} Nov 02 18:22:41.762835 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:41.761602 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:41.780712 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:41.780487 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" podStartSLOduration=1.7804680400000001 podStartE2EDuration="1.78046804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:22:41.73801622 +0000 UTC m=+2933.526521747" watchObservedRunningTime="2026-11-02 18:22:41.78046804 +0000 UTC m=+2933.568973572" Nov 02 18:22:41.800765 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:41.800701 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" podStartSLOduration=1.800687082 podStartE2EDuration="1.800687082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:22:41.773495564 +0000 UTC m=+2933.562001116" watchObservedRunningTime="2026-11-02 18:22:41.800687082 +0000 UTC m=+2933.589192612" Nov 02 18:22:41.872420 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:41.872387 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:41.893662 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:41.889300 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:42.610975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:42.610556 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:42.874714 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:42.873764 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-wgjhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body= Nov 02 18:22:42.874714 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:42.873819 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" podUID="7066250e-b36d-4065-a5fa-265c358a1257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" Nov 02 18:22:42.923978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:42.923456 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xsg9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 02 18:22:42.923978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:42.923518 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" podUID="4740514e-6d04-468b-930f-8be299ef0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 02 18:22:43.447975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:43.442501 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:43.872380 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:43.872292 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-wgjhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body= Nov 02 18:22:43.872380 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:43.872360 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" podUID="7066250e-b36d-4065-a5fa-265c358a1257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" Nov 02 18:22:43.917974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:43.917053 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xsg9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 02 18:22:43.917974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:43.917119 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" podUID="4740514e-6d04-468b-930f-8be299ef0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 02 18:22:44.144135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:44.144028 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:44.807977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:44.806559 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:44.876021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:44.874119 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-wgjhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body= Nov 02 18:22:44.876021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:44.874182 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" podUID="7066250e-b36d-4065-a5fa-265c358a1257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" Nov 02 18:22:44.936022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:44.935234 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xsg9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 02 18:22:44.936022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:44.935311 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" podUID="4740514e-6d04-468b-930f-8be299ef0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 02 18:22:45.403978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:45.398652 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:45.535782 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:45.535595 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:45.555148 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:45.555120 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb" Nov 02 18:22:45.556154 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:45.556131 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" Nov 02 18:22:45.872120 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:45.872084 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-wgjhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body= Nov 02 18:22:45.872310 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:45.872151 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" podUID="7066250e-b36d-4065-a5fa-265c358a1257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" Nov 02 18:22:45.894978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:45.893984 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xsg9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 02 18:22:45.894978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:45.894049 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" podUID="4740514e-6d04-468b-930f-8be299ef0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 02 18:22:46.142032 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:46.135592 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:46.783980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:46.778483 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:46.873726 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:46.872849 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-wgjhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body= Nov 02 18:22:46.873726 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:46.872927 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" podUID="7066250e-b36d-4065-a5fa-265c358a1257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" Nov 02 18:22:46.911978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:46.911419 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xsg9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 02 18:22:46.911978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:46.911512 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" podUID="4740514e-6d04-468b-930f-8be299ef0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 02 18:22:47.539070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:47.538736 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:47.643262 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:22:47.643218 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:22:47.872510 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:47.872419 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-wgjhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body= Nov 02 18:22:47.872681 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:47.872503 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" podUID="7066250e-b36d-4065-a5fa-265c358a1257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" Nov 02 18:22:47.889312 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:47.889265 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xsg9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 02 18:22:47.889497 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:47.889321 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" podUID="4740514e-6d04-468b-930f-8be299ef0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 02 18:22:48.192291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:48.192172 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:48.825615 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:48.825054 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:48.875824 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:48.875782 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-wgjhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body= Nov 02 18:22:48.876101 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:48.875858 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" podUID="7066250e-b36d-4065-a5fa-265c358a1257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" Nov 02 18:22:48.889288 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:48.889249 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xsg9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 02 18:22:48.889483 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:48.889314 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" podUID="4740514e-6d04-468b-930f-8be299ef0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 02 18:22:49.674159 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:49.674098 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:50.216025 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:50.215946 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:50.217090 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:50.217063 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:50.217824 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:50.217798 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" Nov 02 18:22:50.219570 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:50.219551 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:50.220524 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:50.220489 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:50.221193 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:50.221177 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:22:50.272473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:50.272428 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:50.470502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:50.470395 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:51.116151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:51.116118 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:51.811030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:51.808641 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:52.548094 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:52.548041 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:53.127471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:53.127363 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:53.756405 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:53.756293 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:54.453988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:54.444824 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:55.131172 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:55.128016 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:55.790456 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:55.790423 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:55.922596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:55.922557 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:56.291802 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:56.291750 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:56.656093 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:56.655998 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:57.222474 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:57.222440 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:57.481765 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:57.481678 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:58.197605 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:58.197554 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:58.923104 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:58.923066 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:22:59.715988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:22:59.715937 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:00.454049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:00.452205 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:01.300770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:01.300730 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:01.463875 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:01.463841 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:01.603317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:01.603228 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:01.932704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:01.932563 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:02.152554 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:02.152515 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:02.361600 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:02.361539 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:02.530149 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:02.530098 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:03.125771 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:03.125724 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:03.261086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:03.261051 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:03.848536 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:03.848492 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:05.661879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:05.661844 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:05.840131 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:05.840060 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:06.522382 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:06.522347 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:07.225346 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:07.225304 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:08.049973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:08.047571 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:08.701546 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:08.701463 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:09.490790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:09.490667 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:09.671490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:09.671457 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:09.824028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:09.823668 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:10.223181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:10.223145 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:10.678421 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:10.678362 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:11.205255 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:11.205221 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:11.364033 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:11.363994 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:12.111296 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:12.111256 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:12.805227 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:12.805190 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:13.551977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:13.550232 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:14.240839 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:14.240795 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:14.949932 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:14.949896 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:15.642199 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:15.642146 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:16.359384 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:16.359307 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:16.519794 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:16.519757 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:17.009624 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:17.009582 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:17.590374 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:23:17.590330 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:23:17.695753 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:17.695661 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:18.430339 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:18.430264 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:19.208171 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:19.206549 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:20.168545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:20.168504 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:20.932445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:20.932405 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:21.687062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:21.687023 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:22.173650 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:22.173616 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:22.304420 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:22.304378 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:22.956653 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:22.956621 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:22.980254 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:22.980206 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:23:23.619171 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:23.619131 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:23.823321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:23.823284 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:24.077400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:24.077356 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:24.222170 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:24.222138 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:24.601110 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:24.601075 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:24.857290 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:24.857197 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx"] Nov 02 18:23:24.857671 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:24.857637 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" podUID="8998b9b1-c956-4ffb-9274-9eb5443894f9" containerName="istio-proxy" containerID="cri-o://4eb20ff7a517e4db37b943aa9e483440758a7c6d7e2e848145a3a5f52f73f22b" gracePeriod=30 Nov 02 18:23:24.864917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:24.864890 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8"] Nov 02 18:23:24.865339 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:24.865311 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" podUID="5a3d31ae-cabe-438d-91c5-d5852c562ab3" containerName="istio-proxy" containerID="cri-o://3f8e373ecaf16eedfa395c1fc82cb66d3fc7e4e5d2a737d2ab85b902c7be7412" gracePeriod=30 Nov 02 18:23:24.874047 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:24.874015 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z"] Nov 02 18:23:24.874265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:24.874240 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" podUID="55fbc482-0099-414a-a9d0-87d3d0256de7" containerName="istio-proxy" containerID="cri-o://03d4fdd935f368db0c6b4219e79bf62c9244cbdf09de3e0d14b9d440714d4b82" gracePeriod=30 Nov 02 18:23:24.882072 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:24.882046 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9"] Nov 02 18:23:24.882221 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:24.882203 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" podUID="41c5644b-6e3c-45b3-83da-f8c89a081c19" containerName="istio-proxy" containerID="cri-o://9ca961c54ee96766071f62b9bf92d1d53ecbc9596bdc5563145b66f04d2cc8c2" gracePeriod=30 Nov 02 18:23:24.886324 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:24.886301 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5"] Nov 02 18:23:24.886470 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:24.886453 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" podUID="752af71e-0e0d-48b9-9555-9ce325e913ce" containerName="istio-proxy" containerID="cri-o://563f0c0517aeeeb1bfc22c321770cf482d3c5c899aa76b90faf19e8f049ee3fa" gracePeriod=30 Nov 02 18:23:25.018992 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:25.005622 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:25.213487 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:25.213451 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:23:25.213869 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:25.213522 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" podUID="41c5644b-6e3c-45b3-83da-f8c89a081c19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:23:25.238000 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:25.237967 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:25.749156 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:25.749114 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:25.902826 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:25.902789 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:26.055698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:26.055613 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:23:26.055698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:26.055673 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" podUID="752af71e-0e0d-48b9-9555-9ce325e913ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:23:26.504975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:26.503008 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:27.309340 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:27.297353 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:28.126783 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:28.126742 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:28.833850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:28.833776 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:29.231329 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:29.231288 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:23:29.231840 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:29.231349 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" podUID="8998b9b1-c956-4ffb-9274-9eb5443894f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:23:29.515522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:29.515421 5549 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:23:29.515708 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:29.515517 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" podUID="5a3d31ae-cabe-438d-91c5-d5852c562ab3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:23:29.653935 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:29.653903 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:30.220491 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.220411 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:23:30.285503 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.285478 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:23:30.289931 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.289903 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:23:30.294632 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.294605 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:23:30.307444 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.307418 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-workload-certs\") pod \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " Nov 02 18:23:30.307599 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.307457 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-credential-socket\") pod \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " Nov 02 18:23:30.307599 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.307547 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-token\" (UniqueName: \"kubernetes.io/projected/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-token\") pod \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " Nov 02 18:23:30.307737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.307616 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-podinfo\") pod \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " Nov 02 18:23:30.307798 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.307737 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-envoy\") pod \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " Nov 02 18:23:30.307798 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.307759 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istiod-ca-cert\") pod \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " Nov 02 18:23:30.307920 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.307798 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-workload-socket\") pod \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " Nov 02 18:23:30.307920 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.307838 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-data\") pod \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " Nov 02 18:23:30.308065 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.308040 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3d31ae-cabe-438d-91c5-d5852c562ab3-kube-api-access-lx9m5\" (UniqueName: \"kubernetes.io/projected/5a3d31ae-cabe-438d-91c5-d5852c562ab3-kube-api-access-lx9m5\") pod \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\" (UID: \"5a3d31ae-cabe-438d-91c5-d5852c562ab3\") " Nov 02 18:23:30.319790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.319478 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-workload-certs" pod "5a3d31ae-cabe-438d-91c5-d5852c562ab3" (UID: "5a3d31ae-cabe-438d-91c5-d5852c562ab3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.319790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.319573 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-podinfo" pod "5a3d31ae-cabe-438d-91c5-d5852c562ab3" (UID: "5a3d31ae-cabe-438d-91c5-d5852c562ab3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:23:30.319996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.319911 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istiod-ca-cert" pod "5a3d31ae-cabe-438d-91c5-d5852c562ab3" (UID: "5a3d31ae-cabe-438d-91c5-d5852c562ab3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:23:30.320309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.320290 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-credential-socket" pod "5a3d31ae-cabe-438d-91c5-d5852c562ab3" (UID: "5a3d31ae-cabe-438d-91c5-d5852c562ab3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.320622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.320604 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-data" pod "5a3d31ae-cabe-438d-91c5-d5852c562ab3" (UID: "5a3d31ae-cabe-438d-91c5-d5852c562ab3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.321463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.320430 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-workload-socket" pod "5a3d31ae-cabe-438d-91c5-d5852c562ab3" (UID: "5a3d31ae-cabe-438d-91c5-d5852c562ab3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.321828 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.321806 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3d31ae-cabe-438d-91c5-d5852c562ab3-kube-api-access-lx9m5" pod "5a3d31ae-cabe-438d-91c5-d5852c562ab3" (UID: "5a3d31ae-cabe-438d-91c5-d5852c562ab3"). InnerVolumeSpecName "kube-api-access-lx9m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:30.327699 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.327672 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-token" pod "5a3d31ae-cabe-438d-91c5-d5852c562ab3" (UID: "5a3d31ae-cabe-438d-91c5-d5852c562ab3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:30.329709 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.329673 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-envoy" pod "5a3d31ae-cabe-438d-91c5-d5852c562ab3" (UID: "5a3d31ae-cabe-438d-91c5-d5852c562ab3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.353353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.353322 5549 generic.go:356] "Generic (PLEG): container finished" podID="752af71e-0e0d-48b9-9555-9ce325e913ce" containerID="563f0c0517aeeeb1bfc22c321770cf482d3c5c899aa76b90faf19e8f049ee3fa" exitCode=0 Nov 02 18:23:30.353528 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.353447 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" event={"ID":"752af71e-0e0d-48b9-9555-9ce325e913ce","Type":"ContainerDied","Data":"563f0c0517aeeeb1bfc22c321770cf482d3c5c899aa76b90faf19e8f049ee3fa"} Nov 02 18:23:30.355073 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.355040 5549 generic.go:356] "Generic (PLEG): container finished" podID="41c5644b-6e3c-45b3-83da-f8c89a081c19" containerID="9ca961c54ee96766071f62b9bf92d1d53ecbc9596bdc5563145b66f04d2cc8c2" exitCode=0 Nov 02 18:23:30.355213 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.355192 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" event={"ID":"41c5644b-6e3c-45b3-83da-f8c89a081c19","Type":"ContainerDied","Data":"9ca961c54ee96766071f62b9bf92d1d53ecbc9596bdc5563145b66f04d2cc8c2"} Nov 02 18:23:30.355283 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.355244 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" event={"ID":"41c5644b-6e3c-45b3-83da-f8c89a081c19","Type":"ContainerDied","Data":"125fb1f413e41e3eb9d7e7c264a916164a693f674b4207ab066daa1b678362ea"} Nov 02 18:23:30.355283 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.355276 5549 scope.go:122] "RemoveContainer" containerID="9ca961c54ee96766071f62b9bf92d1d53ecbc9596bdc5563145b66f04d2cc8c2" Nov 02 18:23:30.355471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.355452 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9" Nov 02 18:23:30.357767 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.357739 5549 generic.go:356] "Generic (PLEG): container finished" podID="55fbc482-0099-414a-a9d0-87d3d0256de7" containerID="03d4fdd935f368db0c6b4219e79bf62c9244cbdf09de3e0d14b9d440714d4b82" exitCode=0 Nov 02 18:23:30.357880 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.357838 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" event={"ID":"55fbc482-0099-414a-a9d0-87d3d0256de7","Type":"ContainerDied","Data":"03d4fdd935f368db0c6b4219e79bf62c9244cbdf09de3e0d14b9d440714d4b82"} Nov 02 18:23:30.357990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.357880 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" event={"ID":"55fbc482-0099-414a-a9d0-87d3d0256de7","Type":"ContainerDied","Data":"533d39784fd440fdbc987777dc0ee3e1d11ed552e81ad9969de94cf8f458f3c7"} Nov 02 18:23:30.358060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.357986 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z" Nov 02 18:23:30.359999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.359976 5549 generic.go:356] "Generic (PLEG): container finished" podID="8998b9b1-c956-4ffb-9274-9eb5443894f9" containerID="4eb20ff7a517e4db37b943aa9e483440758a7c6d7e2e848145a3a5f52f73f22b" exitCode=0 Nov 02 18:23:30.360342 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.360318 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" event={"ID":"8998b9b1-c956-4ffb-9274-9eb5443894f9","Type":"ContainerDied","Data":"4eb20ff7a517e4db37b943aa9e483440758a7c6d7e2e848145a3a5f52f73f22b"} Nov 02 18:23:30.360499 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.360481 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" event={"ID":"8998b9b1-c956-4ffb-9274-9eb5443894f9","Type":"ContainerDied","Data":"efe3bb009d6604629419e814b7c58ec51896ca01ae7e7a9a1c0c0d9120177c3a"} Nov 02 18:23:30.360681 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.360664 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx" Nov 02 18:23:30.362765 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.362739 5549 generic.go:356] "Generic (PLEG): container finished" podID="5a3d31ae-cabe-438d-91c5-d5852c562ab3" containerID="3f8e373ecaf16eedfa395c1fc82cb66d3fc7e4e5d2a737d2ab85b902c7be7412" exitCode=0 Nov 02 18:23:30.362877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.362787 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" event={"ID":"5a3d31ae-cabe-438d-91c5-d5852c562ab3","Type":"ContainerDied","Data":"3f8e373ecaf16eedfa395c1fc82cb66d3fc7e4e5d2a737d2ab85b902c7be7412"} Nov 02 18:23:30.362877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.362825 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" event={"ID":"5a3d31ae-cabe-438d-91c5-d5852c562ab3","Type":"ContainerDied","Data":"fd729bd9a876352988315e6bdd04e548ab0408cf7aba1c49042a26960a1e93dc"} Nov 02 18:23:30.363044 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.362888 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8" Nov 02 18:23:30.379804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.378487 5549 scope.go:122] "RemoveContainer" containerID="9ca961c54ee96766071f62b9bf92d1d53ecbc9596bdc5563145b66f04d2cc8c2" Nov 02 18:23:30.379804 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:23:30.378871 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca961c54ee96766071f62b9bf92d1d53ecbc9596bdc5563145b66f04d2cc8c2\": container with ID starting with 9ca961c54ee96766071f62b9bf92d1d53ecbc9596bdc5563145b66f04d2cc8c2 not found: ID does not exist" containerID="9ca961c54ee96766071f62b9bf92d1d53ecbc9596bdc5563145b66f04d2cc8c2" Nov 02 18:23:30.379804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.378913 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca961c54ee96766071f62b9bf92d1d53ecbc9596bdc5563145b66f04d2cc8c2"} err="failed to get container status \"9ca961c54ee96766071f62b9bf92d1d53ecbc9596bdc5563145b66f04d2cc8c2\": rpc error: code = NotFound desc = could not find container \"9ca961c54ee96766071f62b9bf92d1d53ecbc9596bdc5563145b66f04d2cc8c2\": container with ID starting with 9ca961c54ee96766071f62b9bf92d1d53ecbc9596bdc5563145b66f04d2cc8c2 not found: ID does not exist" Nov 02 18:23:30.379804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.378939 5549 scope.go:122] "RemoveContainer" containerID="03d4fdd935f368db0c6b4219e79bf62c9244cbdf09de3e0d14b9d440714d4b82" Nov 02 18:23:30.397100 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.397076 5549 scope.go:122] "RemoveContainer" containerID="03d4fdd935f368db0c6b4219e79bf62c9244cbdf09de3e0d14b9d440714d4b82" Nov 02 18:23:30.397836 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:23:30.397805 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d4fdd935f368db0c6b4219e79bf62c9244cbdf09de3e0d14b9d440714d4b82\": container with ID starting with 03d4fdd935f368db0c6b4219e79bf62c9244cbdf09de3e0d14b9d440714d4b82 not found: ID does not exist" containerID="03d4fdd935f368db0c6b4219e79bf62c9244cbdf09de3e0d14b9d440714d4b82" Nov 02 18:23:30.398847 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.398812 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d4fdd935f368db0c6b4219e79bf62c9244cbdf09de3e0d14b9d440714d4b82"} err="failed to get container status \"03d4fdd935f368db0c6b4219e79bf62c9244cbdf09de3e0d14b9d440714d4b82\": rpc error: code = NotFound desc = could not find container \"03d4fdd935f368db0c6b4219e79bf62c9244cbdf09de3e0d14b9d440714d4b82\": container with ID starting with 03d4fdd935f368db0c6b4219e79bf62c9244cbdf09de3e0d14b9d440714d4b82 not found: ID does not exist" Nov 02 18:23:30.399019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.399004 5549 scope.go:122] "RemoveContainer" containerID="4eb20ff7a517e4db37b943aa9e483440758a7c6d7e2e848145a3a5f52f73f22b" Nov 02 18:23:30.403097 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.403070 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:23:30.403721 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.403698 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8"] Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.410351 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-podinfo\") pod \"41c5644b-6e3c-45b3-83da-f8c89a081c19\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.410434 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-credential-socket\") pod \"41c5644b-6e3c-45b3-83da-f8c89a081c19\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.410477 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c5644b-6e3c-45b3-83da-f8c89a081c19-kube-api-access-gq29c\" (UniqueName: \"kubernetes.io/projected/41c5644b-6e3c-45b3-83da-f8c89a081c19-kube-api-access-gq29c\") pod \"41c5644b-6e3c-45b3-83da-f8c89a081c19\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.410523 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8998b9b1-c956-4ffb-9274-9eb5443894f9-kube-api-access-k6j5f\" (UniqueName: \"kubernetes.io/projected/8998b9b1-c956-4ffb-9274-9eb5443894f9-kube-api-access-k6j5f\") pod \"8998b9b1-c956-4ffb-9274-9eb5443894f9\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.410564 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8998b9b1-c956-4ffb-9274-9eb5443894f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8998b9b1-c956-4ffb-9274-9eb5443894f9-istiod-ca-cert\") pod \"8998b9b1-c956-4ffb-9274-9eb5443894f9\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.410592 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-envoy\") pod \"41c5644b-6e3c-45b3-83da-f8c89a081c19\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.410642 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-podinfo\") pod \"55fbc482-0099-414a-a9d0-87d3d0256de7\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.410674 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-workload-certs\") pod \"41c5644b-6e3c-45b3-83da-f8c89a081c19\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.410716 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-credential-socket\") pod \"55fbc482-0099-414a-a9d0-87d3d0256de7\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.410742 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-envoy\") pod \"55fbc482-0099-414a-a9d0-87d3d0256de7\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.410812 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-credential-socket\") pod \"8998b9b1-c956-4ffb-9274-9eb5443894f9\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.410854 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-envoy\") pod \"8998b9b1-c956-4ffb-9274-9eb5443894f9\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.410906 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-data\") pod \"41c5644b-6e3c-45b3-83da-f8c89a081c19\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.410993 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-tjpc8"] Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.411343 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-data\") pod \"8998b9b1-c956-4ffb-9274-9eb5443894f9\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.411378 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c5644b-6e3c-45b3-83da-f8c89a081c19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c5644b-6e3c-45b3-83da-f8c89a081c19-istiod-ca-cert\") pod \"41c5644b-6e3c-45b3-83da-f8c89a081c19\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.411458 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-token\" (UniqueName: \"kubernetes.io/projected/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-token\") pod \"41c5644b-6e3c-45b3-83da-f8c89a081c19\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.411503 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-data\") pod \"55fbc482-0099-414a-a9d0-87d3d0256de7\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.411629 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-token\" (UniqueName: \"kubernetes.io/projected/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-token\") pod \"55fbc482-0099-414a-a9d0-87d3d0256de7\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.411656 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55fbc482-0099-414a-a9d0-87d3d0256de7-kube-api-access-gd2jk\" (UniqueName: \"kubernetes.io/projected/55fbc482-0099-414a-a9d0-87d3d0256de7-kube-api-access-gd2jk\") pod \"55fbc482-0099-414a-a9d0-87d3d0256de7\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.411696 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-workload-certs\") pod \"55fbc482-0099-414a-a9d0-87d3d0256de7\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.411735 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-token\" (UniqueName: \"kubernetes.io/projected/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-token\") pod \"8998b9b1-c956-4ffb-9274-9eb5443894f9\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.411780 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-workload-socket\") pod \"55fbc482-0099-414a-a9d0-87d3d0256de7\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.411813 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-workload-certs\") pod \"8998b9b1-c956-4ffb-9274-9eb5443894f9\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.411847 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-workload-socket\") pod \"41c5644b-6e3c-45b3-83da-f8c89a081c19\" (UID: \"41c5644b-6e3c-45b3-83da-f8c89a081c19\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.411893 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55fbc482-0099-414a-a9d0-87d3d0256de7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55fbc482-0099-414a-a9d0-87d3d0256de7-istiod-ca-cert\") pod \"55fbc482-0099-414a-a9d0-87d3d0256de7\" (UID: \"55fbc482-0099-414a-a9d0-87d3d0256de7\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.412000 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-workload-socket\") pod \"8998b9b1-c956-4ffb-9274-9eb5443894f9\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " Nov 02 18:23:30.412192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.412027 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-podinfo\") pod \"8998b9b1-c956-4ffb-9274-9eb5443894f9\" (UID: \"8998b9b1-c956-4ffb-9274-9eb5443894f9\") " Nov 02 18:23:30.413827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.413090 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx9m5\" (UniqueName: \"kubernetes.io/projected/5a3d31ae-cabe-438d-91c5-d5852c562ab3-kube-api-access-lx9m5\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.413827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.413115 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.413827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.413131 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.413827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.413147 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.413827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.413161 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.413827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.413175 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.413827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.413191 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.413827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.413206 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.413827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.413221 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3d31ae-cabe-438d-91c5-d5852c562ab3-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.419268 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.418741 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-workload-socket" pod "55fbc482-0099-414a-a9d0-87d3d0256de7" (UID: "55fbc482-0099-414a-a9d0-87d3d0256de7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.419268 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.419040 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8998b9b1-c956-4ffb-9274-9eb5443894f9-istiod-ca-cert" pod "8998b9b1-c956-4ffb-9274-9eb5443894f9" (UID: "8998b9b1-c956-4ffb-9274-9eb5443894f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:23:30.419515 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.419488 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-workload-certs" pod "8998b9b1-c956-4ffb-9274-9eb5443894f9" (UID: "8998b9b1-c956-4ffb-9274-9eb5443894f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.419661 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.419641 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55fbc482-0099-414a-a9d0-87d3d0256de7-istiod-ca-cert" pod "55fbc482-0099-414a-a9d0-87d3d0256de7" (UID: "55fbc482-0099-414a-a9d0-87d3d0256de7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:23:30.419875 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.419840 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-workload-socket" pod "41c5644b-6e3c-45b3-83da-f8c89a081c19" (UID: "41c5644b-6e3c-45b3-83da-f8c89a081c19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.420055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.420035 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-workload-socket" pod "8998b9b1-c956-4ffb-9274-9eb5443894f9" (UID: "8998b9b1-c956-4ffb-9274-9eb5443894f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.420122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.420092 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-credential-socket" pod "55fbc482-0099-414a-a9d0-87d3d0256de7" (UID: "55fbc482-0099-414a-a9d0-87d3d0256de7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.420250 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.420231 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-data" pod "41c5644b-6e3c-45b3-83da-f8c89a081c19" (UID: "41c5644b-6e3c-45b3-83da-f8c89a081c19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.420314 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.420300 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-workload-certs" pod "41c5644b-6e3c-45b3-83da-f8c89a081c19" (UID: "41c5644b-6e3c-45b3-83da-f8c89a081c19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.421321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.421122 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-data" pod "8998b9b1-c956-4ffb-9274-9eb5443894f9" (UID: "8998b9b1-c956-4ffb-9274-9eb5443894f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.421321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.421170 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-credential-socket" pod "8998b9b1-c956-4ffb-9274-9eb5443894f9" (UID: "8998b9b1-c956-4ffb-9274-9eb5443894f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.421632 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.421506 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-credential-socket" pod "41c5644b-6e3c-45b3-83da-f8c89a081c19" (UID: "41c5644b-6e3c-45b3-83da-f8c89a081c19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.421898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.421751 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-workload-certs" pod "55fbc482-0099-414a-a9d0-87d3d0256de7" (UID: "55fbc482-0099-414a-a9d0-87d3d0256de7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.422265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.422242 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c5644b-6e3c-45b3-83da-f8c89a081c19-istiod-ca-cert" pod "41c5644b-6e3c-45b3-83da-f8c89a081c19" (UID: "41c5644b-6e3c-45b3-83da-f8c89a081c19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:23:30.422360 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.422295 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-data" pod "55fbc482-0099-414a-a9d0-87d3d0256de7" (UID: "55fbc482-0099-414a-a9d0-87d3d0256de7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.428327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.428308 5549 scope.go:122] "RemoveContainer" containerID="4eb20ff7a517e4db37b943aa9e483440758a7c6d7e2e848145a3a5f52f73f22b" Nov 02 18:23:30.429174 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:23:30.428698 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb20ff7a517e4db37b943aa9e483440758a7c6d7e2e848145a3a5f52f73f22b\": container with ID starting with 4eb20ff7a517e4db37b943aa9e483440758a7c6d7e2e848145a3a5f52f73f22b not found: ID does not exist" containerID="4eb20ff7a517e4db37b943aa9e483440758a7c6d7e2e848145a3a5f52f73f22b" Nov 02 18:23:30.429174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.428744 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb20ff7a517e4db37b943aa9e483440758a7c6d7e2e848145a3a5f52f73f22b"} err="failed to get container status \"4eb20ff7a517e4db37b943aa9e483440758a7c6d7e2e848145a3a5f52f73f22b\": rpc error: code = NotFound desc = could not find container \"4eb20ff7a517e4db37b943aa9e483440758a7c6d7e2e848145a3a5f52f73f22b\": container with ID starting with 4eb20ff7a517e4db37b943aa9e483440758a7c6d7e2e848145a3a5f52f73f22b not found: ID does not exist" Nov 02 18:23:30.429174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.428778 5549 scope.go:122] "RemoveContainer" containerID="3f8e373ecaf16eedfa395c1fc82cb66d3fc7e4e5d2a737d2ab85b902c7be7412" Nov 02 18:23:30.430080 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.430057 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55fbc482-0099-414a-a9d0-87d3d0256de7-kube-api-access-gd2jk" pod "55fbc482-0099-414a-a9d0-87d3d0256de7" (UID: "55fbc482-0099-414a-a9d0-87d3d0256de7"). InnerVolumeSpecName "kube-api-access-gd2jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:30.430711 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.430675 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-envoy" pod "8998b9b1-c956-4ffb-9274-9eb5443894f9" (UID: "8998b9b1-c956-4ffb-9274-9eb5443894f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.430790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.430736 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-podinfo" pod "55fbc482-0099-414a-a9d0-87d3d0256de7" (UID: "55fbc482-0099-414a-a9d0-87d3d0256de7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:23:30.430790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.430753 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-token" pod "55fbc482-0099-414a-a9d0-87d3d0256de7" (UID: "55fbc482-0099-414a-a9d0-87d3d0256de7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:30.430790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.430739 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c5644b-6e3c-45b3-83da-f8c89a081c19-kube-api-access-gq29c" pod "41c5644b-6e3c-45b3-83da-f8c89a081c19" (UID: "41c5644b-6e3c-45b3-83da-f8c89a081c19"). InnerVolumeSpecName "kube-api-access-gq29c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:30.431052 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.430853 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-token" pod "41c5644b-6e3c-45b3-83da-f8c89a081c19" (UID: "41c5644b-6e3c-45b3-83da-f8c89a081c19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:30.431052 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.430880 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8998b9b1-c956-4ffb-9274-9eb5443894f9-kube-api-access-k6j5f" pod "8998b9b1-c956-4ffb-9274-9eb5443894f9" (UID: "8998b9b1-c956-4ffb-9274-9eb5443894f9"). InnerVolumeSpecName "kube-api-access-k6j5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:30.431052 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.430990 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-envoy" pod "41c5644b-6e3c-45b3-83da-f8c89a081c19" (UID: "41c5644b-6e3c-45b3-83da-f8c89a081c19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.431052 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.431023 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-token" pod "8998b9b1-c956-4ffb-9274-9eb5443894f9" (UID: "8998b9b1-c956-4ffb-9274-9eb5443894f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:30.431052 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.431043 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-podinfo" pod "8998b9b1-c956-4ffb-9274-9eb5443894f9" (UID: "8998b9b1-c956-4ffb-9274-9eb5443894f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:23:30.431281 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.431096 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-podinfo" pod "41c5644b-6e3c-45b3-83da-f8c89a081c19" (UID: "41c5644b-6e3c-45b3-83da-f8c89a081c19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:23:30.434335 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.434310 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-envoy" pod "55fbc482-0099-414a-a9d0-87d3d0256de7" (UID: "55fbc482-0099-414a-a9d0-87d3d0256de7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.440116 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.440099 5549 scope.go:122] "RemoveContainer" containerID="3f8e373ecaf16eedfa395c1fc82cb66d3fc7e4e5d2a737d2ab85b902c7be7412" Nov 02 18:23:30.440415 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:23:30.440393 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f8e373ecaf16eedfa395c1fc82cb66d3fc7e4e5d2a737d2ab85b902c7be7412\": container with ID starting with 3f8e373ecaf16eedfa395c1fc82cb66d3fc7e4e5d2a737d2ab85b902c7be7412 not found: ID does not exist" containerID="3f8e373ecaf16eedfa395c1fc82cb66d3fc7e4e5d2a737d2ab85b902c7be7412" Nov 02 18:23:30.440485 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.440421 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8e373ecaf16eedfa395c1fc82cb66d3fc7e4e5d2a737d2ab85b902c7be7412"} err="failed to get container status \"3f8e373ecaf16eedfa395c1fc82cb66d3fc7e4e5d2a737d2ab85b902c7be7412\": rpc error: code = NotFound desc = could not find container \"3f8e373ecaf16eedfa395c1fc82cb66d3fc7e4e5d2a737d2ab85b902c7be7412\": container with ID starting with 3f8e373ecaf16eedfa395c1fc82cb66d3fc7e4e5d2a737d2ab85b902c7be7412 not found: ID does not exist" Nov 02 18:23:30.514010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.513974 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-workload-socket\") pod \"752af71e-0e0d-48b9-9555-9ce325e913ce\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " Nov 02 18:23:30.514197 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.514120 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-podinfo\") pod \"752af71e-0e0d-48b9-9555-9ce325e913ce\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " Nov 02 18:23:30.514269 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.514244 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-workload-socket" pod "752af71e-0e0d-48b9-9555-9ce325e913ce" (UID: "752af71e-0e0d-48b9-9555-9ce325e913ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.514396 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.514377 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-workload-certs\") pod \"752af71e-0e0d-48b9-9555-9ce325e913ce\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " Nov 02 18:23:30.514465 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.514420 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-data\") pod \"752af71e-0e0d-48b9-9555-9ce325e913ce\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " Nov 02 18:23:30.514538 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.514504 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-token\" (UniqueName: \"kubernetes.io/projected/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-token\") pod \"752af71e-0e0d-48b9-9555-9ce325e913ce\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " Nov 02 18:23:30.514597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.514548 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752af71e-0e0d-48b9-9555-9ce325e913ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752af71e-0e0d-48b9-9555-9ce325e913ce-istiod-ca-cert\") pod \"752af71e-0e0d-48b9-9555-9ce325e913ce\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " Nov 02 18:23:30.514597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.514575 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752af71e-0e0d-48b9-9555-9ce325e913ce-kube-api-access-7prdx\" (UniqueName: \"kubernetes.io/projected/752af71e-0e0d-48b9-9555-9ce325e913ce-kube-api-access-7prdx\") pod \"752af71e-0e0d-48b9-9555-9ce325e913ce\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " Nov 02 18:23:30.514712 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.514623 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-credential-socket\") pod \"752af71e-0e0d-48b9-9555-9ce325e913ce\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " Nov 02 18:23:30.514712 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.514687 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-envoy\") pod \"752af71e-0e0d-48b9-9555-9ce325e913ce\" (UID: \"752af71e-0e0d-48b9-9555-9ce325e913ce\") " Nov 02 18:23:30.515613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515414 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-data" pod "752af71e-0e0d-48b9-9555-9ce325e913ce" (UID: "752af71e-0e0d-48b9-9555-9ce325e913ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.515613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515421 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752af71e-0e0d-48b9-9555-9ce325e913ce-istiod-ca-cert" pod "752af71e-0e0d-48b9-9555-9ce325e913ce" (UID: "752af71e-0e0d-48b9-9555-9ce325e913ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:23:30.515613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515583 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55fbc482-0099-414a-a9d0-87d3d0256de7-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515607 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515622 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515624 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-credential-socket" pod "752af71e-0e0d-48b9-9555-9ce325e913ce" (UID: "752af71e-0e0d-48b9-9555-9ce325e913ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515638 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515654 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515668 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515682 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq29c\" (UniqueName: \"kubernetes.io/projected/41c5644b-6e3c-45b3-83da-f8c89a081c19-kube-api-access-gq29c\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515690 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-workload-certs" pod "752af71e-0e0d-48b9-9555-9ce325e913ce" (UID: "752af71e-0e0d-48b9-9555-9ce325e913ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515697 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6j5f\" (UniqueName: \"kubernetes.io/projected/8998b9b1-c956-4ffb-9274-9eb5443894f9-kube-api-access-k6j5f\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515741 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8998b9b1-c956-4ffb-9274-9eb5443894f9-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515758 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515774 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515788 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515804 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515818 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515831 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515845 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515857 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515870 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515885 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c5644b-6e3c-45b3-83da-f8c89a081c19-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515898 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c5644b-6e3c-45b3-83da-f8c89a081c19-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515912 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515935 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.515927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515964 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55fbc482-0099-414a-a9d0-87d3d0256de7-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.517231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515979 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd2jk\" (UniqueName: \"kubernetes.io/projected/55fbc482-0099-414a-a9d0-87d3d0256de7-kube-api-access-gd2jk\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.517231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.515995 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.517231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.516008 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8998b9b1-c956-4ffb-9274-9eb5443894f9-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.517231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.516022 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fbc482-0099-414a-a9d0-87d3d0256de7-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.517231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.516036 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752af71e-0e0d-48b9-9555-9ce325e913ce-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.517231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.516051 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8998b9b1-c956-4ffb-9274-9eb5443894f9-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.517231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.516065 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c5644b-6e3c-45b3-83da-f8c89a081c19-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.517231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.516972 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-podinfo" pod "752af71e-0e0d-48b9-9555-9ce325e913ce" (UID: "752af71e-0e0d-48b9-9555-9ce325e913ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:23:30.517697 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.517673 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-token" pod "752af71e-0e0d-48b9-9555-9ce325e913ce" (UID: "752af71e-0e0d-48b9-9555-9ce325e913ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:30.518347 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.518323 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752af71e-0e0d-48b9-9555-9ce325e913ce-kube-api-access-7prdx" pod "752af71e-0e0d-48b9-9555-9ce325e913ce" (UID: "752af71e-0e0d-48b9-9555-9ce325e913ce"). InnerVolumeSpecName "kube-api-access-7prdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:30.518635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.518608 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-envoy" pod "752af71e-0e0d-48b9-9555-9ce325e913ce" (UID: "752af71e-0e0d-48b9-9555-9ce325e913ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.618013 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.617979 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.618013 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.618010 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.618251 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.618024 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7prdx\" (UniqueName: \"kubernetes.io/projected/752af71e-0e0d-48b9-9555-9ce325e913ce-kube-api-access-7prdx\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.618251 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.618037 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.618251 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.618046 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.618251 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.618055 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752af71e-0e0d-48b9-9555-9ce325e913ce-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.697544 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.685294 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9"] Nov 02 18:23:30.725005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.717671 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vkmf9"] Nov 02 18:23:30.725005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.719911 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:30.746091 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.746039 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z"] Nov 02 18:23:30.751072 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.751043 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-kv89z"] Nov 02 18:23:30.762032 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.762006 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx"] Nov 02 18:23:30.769262 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:30.769239 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-n6bsx"] Nov 02 18:23:31.373147 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:31.373113 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" Nov 02 18:23:31.373147 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:31.373133 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5" event={"ID":"752af71e-0e0d-48b9-9555-9ce325e913ce","Type":"ContainerDied","Data":"23abc1ec3b1566bc768ee466709ef801950e4c9020550a327ff4c850d9b2632d"} Nov 02 18:23:31.373664 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:31.373210 5549 scope.go:122] "RemoveContainer" containerID="563f0c0517aeeeb1bfc22c321770cf482d3c5c899aa76b90faf19e8f049ee3fa" Nov 02 18:23:31.415983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:31.413440 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5"] Nov 02 18:23:31.449756 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:31.434005 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:31.449756 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:31.443856 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-gfzb5"] Nov 02 18:23:32.326121 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:32.325972 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:32.344024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:32.343991 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c5644b-6e3c-45b3-83da-f8c89a081c19" path="/var/lib/kubelet/pods/41c5644b-6e3c-45b3-83da-f8c89a081c19/volumes" Nov 02 18:23:32.344713 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:32.344688 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55fbc482-0099-414a-a9d0-87d3d0256de7" path="/var/lib/kubelet/pods/55fbc482-0099-414a-a9d0-87d3d0256de7/volumes" Nov 02 18:23:32.345427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:32.345410 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a3d31ae-cabe-438d-91c5-d5852c562ab3" path="/var/lib/kubelet/pods/5a3d31ae-cabe-438d-91c5-d5852c562ab3/volumes" Nov 02 18:23:32.346316 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:32.346297 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="752af71e-0e0d-48b9-9555-9ce325e913ce" path="/var/lib/kubelet/pods/752af71e-0e0d-48b9-9555-9ce325e913ce/volumes" Nov 02 18:23:32.346797 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:32.346784 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8998b9b1-c956-4ffb-9274-9eb5443894f9" path="/var/lib/kubelet/pods/8998b9b1-c956-4ffb-9274-9eb5443894f9/volumes" Nov 02 18:23:33.116463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.112804 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:33.632736 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.632701 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc"] Nov 02 18:23:33.633738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.633714 5549 cpu_manager.go:414] "containerMap: removing container" podUID="8998b9b1-c956-4ffb-9274-9eb5443894f9" containerName="istio-proxy" Nov 02 18:23:33.633738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.633737 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8998b9b1-c956-4ffb-9274-9eb5443894f9" containerName="istio-proxy" Nov 02 18:23:33.633738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.633751 5549 cpu_manager.go:414] "containerMap: removing container" podUID="5a3d31ae-cabe-438d-91c5-d5852c562ab3" containerName="istio-proxy" Nov 02 18:23:33.634020 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.633758 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a3d31ae-cabe-438d-91c5-d5852c562ab3" containerName="istio-proxy" Nov 02 18:23:33.634020 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.633771 5549 cpu_manager.go:414] "containerMap: removing container" podUID="752af71e-0e0d-48b9-9555-9ce325e913ce" containerName="istio-proxy" Nov 02 18:23:33.634020 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.633778 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752af71e-0e0d-48b9-9555-9ce325e913ce" containerName="istio-proxy" Nov 02 18:23:33.634020 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.633803 5549 cpu_manager.go:414] "containerMap: removing container" podUID="41c5644b-6e3c-45b3-83da-f8c89a081c19" containerName="istio-proxy" Nov 02 18:23:33.634020 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.633811 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c5644b-6e3c-45b3-83da-f8c89a081c19" containerName="istio-proxy" Nov 02 18:23:33.634020 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.633885 5549 cpu_manager.go:414] "containerMap: removing container" podUID="55fbc482-0099-414a-a9d0-87d3d0256de7" containerName="istio-proxy" Nov 02 18:23:33.634020 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.633893 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55fbc482-0099-414a-a9d0-87d3d0256de7" containerName="istio-proxy" Nov 02 18:23:33.634398 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.634134 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="55fbc482-0099-414a-a9d0-87d3d0256de7" containerName="istio-proxy" Nov 02 18:23:33.634398 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.634154 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8998b9b1-c956-4ffb-9274-9eb5443894f9" containerName="istio-proxy" Nov 02 18:23:33.634398 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.634167 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a3d31ae-cabe-438d-91c5-d5852c562ab3" containerName="istio-proxy" Nov 02 18:23:33.634398 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.634177 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="752af71e-0e0d-48b9-9555-9ce325e913ce" containerName="istio-proxy" Nov 02 18:23:33.634398 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.634192 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c5644b-6e3c-45b3-83da-f8c89a081c19" containerName="istio-proxy" Nov 02 18:23:33.640328 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.640307 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.645330 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.645308 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cc0b41\"/\"default-dockercfg-nnhsf\"" Nov 02 18:23:33.646158 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.646115 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cc0b41\"/\"test-model-cc0b41-kserve-self-signed-certs\"" Nov 02 18:23:33.646526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.646503 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cc0b41\"/\"kube-root-ca.crt\"" Nov 02 18:23:33.646648 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.646597 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cc0b41\"/\"openshift-service-ca.crt\"" Nov 02 18:23:33.649238 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.649218 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc"] Nov 02 18:23:33.716114 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.716082 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:33.771675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.771646 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-home\") pod \"test-model-cc0b41-kserve-677864c7b5-27rcc\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.771848 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.771680 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5febcd-72c8-415a-888a-edf57777464c-tls-certs\") pod \"test-model-cc0b41-kserve-677864c7b5-27rcc\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.771848 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.771700 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvdzr\" (UniqueName: \"kubernetes.io/projected/bc5febcd-72c8-415a-888a-edf57777464c-kube-api-access-dvdzr\") pod \"test-model-cc0b41-kserve-677864c7b5-27rcc\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.771942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.771844 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-model-cache\") pod \"test-model-cc0b41-kserve-677864c7b5-27rcc\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.772023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.771998 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-tmp-dir\") pod \"test-model-cc0b41-kserve-677864c7b5-27rcc\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.772111 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.772097 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-dshm\") pod \"test-model-cc0b41-kserve-677864c7b5-27rcc\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.867975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.863097 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:33.878978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.875982 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-model-cache\") pod \"test-model-cc0b41-kserve-677864c7b5-27rcc\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.878978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.876138 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-tmp-dir\") pod \"test-model-cc0b41-kserve-677864c7b5-27rcc\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.878978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.876265 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-dshm\") pod \"test-model-cc0b41-kserve-677864c7b5-27rcc\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.878978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.876454 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-home\") pod \"test-model-cc0b41-kserve-677864c7b5-27rcc\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.878978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.876480 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5febcd-72c8-415a-888a-edf57777464c-tls-certs\") pod \"test-model-cc0b41-kserve-677864c7b5-27rcc\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.878978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.876505 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvdzr\" (UniqueName: \"kubernetes.io/projected/bc5febcd-72c8-415a-888a-edf57777464c-kube-api-access-dvdzr\") pod \"test-model-cc0b41-kserve-677864c7b5-27rcc\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.878978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.877266 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-model-cache\") pod \"test-model-cc0b41-kserve-677864c7b5-27rcc\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.878978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.877529 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-tmp-dir\") pod \"test-model-cc0b41-kserve-677864c7b5-27rcc\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.885014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.881312 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-home\") pod \"test-model-cc0b41-kserve-677864c7b5-27rcc\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.906987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.900642 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5febcd-72c8-415a-888a-edf57777464c-tls-certs\") pod \"test-model-cc0b41-kserve-677864c7b5-27rcc\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.919706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.919628 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-dshm\") pod \"test-model-cc0b41-kserve-677864c7b5-27rcc\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.920583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.920368 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvdzr\" (UniqueName: \"kubernetes.io/projected/bc5febcd-72c8-415a-888a-edf57777464c-kube-api-access-dvdzr\") pod \"test-model-cc0b41-kserve-677864c7b5-27rcc\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:33.957465 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:33.957433 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:34.256423 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:23:34.256388 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc5febcd_72c8_415a_888a_edf57777464c.slice/crio-c82d95f15c10dfe900863553205b37dd545af29131e6b43511ff4ea984863d37 WatchSource:0}: Error finding container c82d95f15c10dfe900863553205b37dd545af29131e6b43511ff4ea984863d37: Status 404 returned error can't find the container with id c82d95f15c10dfe900863553205b37dd545af29131e6b43511ff4ea984863d37 Nov 02 18:23:34.256884 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:34.256522 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc"] Nov 02 18:23:34.414650 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:34.414606 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" event={"ID":"bc5febcd-72c8-415a-888a-edf57777464c","Type":"ContainerStarted","Data":"296dc13cadebbed52320ae5dafe032e8ae0deb647e2bafd70a59da90472c5aac"} Nov 02 18:23:34.414832 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:34.414669 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" event={"ID":"bc5febcd-72c8-415a-888a-edf57777464c","Type":"ContainerStarted","Data":"c82d95f15c10dfe900863553205b37dd545af29131e6b43511ff4ea984863d37"} Nov 02 18:23:34.438022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:34.437914 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" podStartSLOduration=1.437901065 podStartE2EDuration="1.437901065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:23:34.434396141 +0000 UTC m=+2986.222901669" watchObservedRunningTime="2026-11-02 18:23:34.437901065 +0000 UTC m=+2986.226406615" Nov 02 18:23:34.777985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:34.777926 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:34.957760 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:34.957716 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:35.527610 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:35.520624 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:36.203151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:36.203118 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:36.477327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:36.477246 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:36.933292 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:36.931642 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:37.309498 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:37.309461 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:37.648066 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:37.647699 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:38.321137 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:38.321046 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:39.017186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:39.017130 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:39.920982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:39.917390 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:40.624568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:40.624528 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:41.288837 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:41.288761 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:41.933332 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:41.933293 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:42.703990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:42.698869 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:43.353513 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:43.353474 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:43.958153 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:43.958117 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:43.970928 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:43.970901 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:44.136887 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:44.136848 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:44.310806 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:44.310717 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:44.504763 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:44.504733 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:23:44.851897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:44.845696 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:45.519777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:45.518465 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:46.328987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:46.328138 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:46.990105 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:46.990066 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:47.588795 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:23:47.588752 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:23:47.729978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:47.728631 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:48.349105 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:48.349072 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:48.481395 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:48.481358 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:48.619472 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:48.619360 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:49.062984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:49.062932 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:49.829544 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:49.829503 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:49.912123 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:49.912096 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:23:50.152551 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:50.152477 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:23:50.501381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:50.501210 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:51.202684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:51.202651 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:51.942976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:51.938035 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:52.603539 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:52.603498 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:53.359788 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:53.359730 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:54.270289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:54.270253 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:55.210037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:55.209984 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:55.483249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:55.483168 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:56.042023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:56.041984 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:56.727324 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:56.727242 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:57.497444 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:57.497408 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:58.343975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:58.341896 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:59.055555 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:59.055506 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:23:59.709652 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:23:59.709609 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:00.480111 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:00.480079 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:01.113462 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:01.113429 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:01.745414 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:01.745375 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:02.409776 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:02.409743 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:02.545846 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:02.545814 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:03.135778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:03.135727 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:04.098164 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:04.098129 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:04.747132 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:04.747066 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:05.425887 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:05.425185 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:06.119810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:06.119774 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:06.735976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:06.734989 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:07.437977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:07.435547 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:08.131976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:08.123550 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:08.755785 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:08.755746 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:09.439323 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:09.434999 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:10.099189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:10.099088 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:10.759542 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:10.759504 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:11.445417 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:11.445372 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:12.135280 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:12.135228 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:12.958660 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:12.958617 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:13.635502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:13.635464 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:14.347978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:14.347391 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:15.140035 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:15.139995 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:15.876450 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:15.876410 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:16.598729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:16.598689 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:17.327502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:17.327350 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:17.589551 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:24:17.589470 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:24:18.044666 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:18.043358 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:18.818014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:18.817943 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:19.593039 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:19.593001 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:20.425552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:20.425504 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:20.738425 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:20.738383 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:20.907796 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:20.907763 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:21.043312 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:21.043220 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:21.214873 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:21.214745 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:21.818383 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:21.818343 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:21.952871 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:21.952832 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:22.088329 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:22.088241 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:22.789486 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:22.789449 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:23.442560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:23.440945 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:24.220946 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:24.220911 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:24.986788 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:24.986755 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:25.614797 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:25.614763 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:26.434471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:26.434226 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:27.088150 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:27.075339 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:27.749211 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:27.749172 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:28.167608 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:28.167509 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:24:28.426769 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:28.426601 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:29.123738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:29.123694 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:30.196196 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:30.196146 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:31.049978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:31.049847 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:32.062975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:32.060257 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:32.674978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:32.672517 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:33.352634 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:33.352595 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:33.703614 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:33.703580 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:34.132895 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:34.132802 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:34.803622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:34.803548 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:35.436137 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:35.436082 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:36.239635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:36.239589 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:37.020597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:37.020560 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:37.719984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:37.718497 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:38.425011 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:38.424937 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:39.092603 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:39.092567 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:39.927673 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:39.923608 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:40.164326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:40.164292 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:40.651234 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:40.651198 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:41.291774 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:41.291733 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:42.012643 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:42.012586 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:42.649686 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:42.649585 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:43.328792 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:43.328759 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:44.071898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:44.071859 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:44.842006 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:44.841932 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:45.499398 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:45.499361 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:46.144979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:46.142004 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:46.811979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:46.807559 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:47.551367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:47.551314 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:47.589916 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:24:47.589867 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:24:48.223826 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:48.222477 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:48.912649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:48.909638 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:49.543200 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:49.543155 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:50.244225 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:50.244181 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:50.920171 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:50.920136 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:51.896976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:51.893643 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:52.795977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:52.795220 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:54.141426 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:54.141054 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:54.799842 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:54.799800 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:55.439860 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:55.439821 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:56.146008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:56.144055 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:56.960853 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:56.960359 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:57.655686 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:57.655646 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:58.414724 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:58.414681 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:59.188858 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:59.188822 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:24:59.832975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:24:59.823743 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:00.574909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:00.574689 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:01.407979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:01.405080 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:02.133446 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:02.130775 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:02.449387 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:02.449355 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:02.795029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:02.792112 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:03.431847 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:03.431808 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:04.138428 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:04.137697 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:04.810486 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:04.809938 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:05.532795 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:05.532760 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:06.125199 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:06.125163 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:06.334296 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:06.334252 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:06.692793 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:06.692256 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:07.135557 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:07.135509 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:07.959720 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:07.959689 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:08.090844 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:08.090796 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:08.425850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:08.425756 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:08.608973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:08.608513 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:09.020444 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:09.020392 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:09.809321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:09.809284 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:10.632293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:10.632140 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:11.054845 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:11.054810 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:11.196188 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:11.196142 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:11.565129 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:11.565093 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:12.136139 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:12.136096 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:12.273850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:12.273816 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:12.560967 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:12.560924 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:13.329408 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:13.329373 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:13.473647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:13.473612 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:13.610501 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:13.610416 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:14.225532 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:14.225493 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:14.914028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:14.903558 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:15.171127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:15.171022 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:25:15.645027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:15.644944 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:16.316393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:16.316354 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:17.016709 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:17.016673 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:17.588099 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:17.588059 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh"] Nov 02 18:25:17.616033 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:25:17.615986 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:25:17.658330 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:17.658296 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:18.013370 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:18.013330 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:18.241736 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:18.241705 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:18.712285 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:18.712251 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:18.866007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:18.865969 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:19.522173 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:19.519419 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:20.241760 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:20.241720 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:20.930495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:20.930420 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:21.627606 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:21.627571 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:22.335655 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:22.335622 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:23.057176 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:23.057137 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:23.252113 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:23.252080 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh"] Nov 02 18:25:23.367782 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:23.367680 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:23.504615 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:23.504564 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:24.117086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:24.117044 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:24.825266 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:24.825217 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:25.129530 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:25.129442 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:25.621087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:25.620988 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:26.383499 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:26.383459 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:27.104905 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:27.104861 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:27.782839 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:27.782789 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:28.253136 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:28.253095 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh" podUID="ec0b8f3e-b65f-4709-95ad-8dab56153f36" containerName="payload-processing" containerID="cri-o://4a1e6f017e8edddac7d9b290849d4e3af58bb2e0bb239bc33294324a0e2fddfe" gracePeriod=25 Nov 02 18:25:28.415710 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:28.415562 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:28.597379 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:28.597350 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh" Nov 02 18:25:28.696485 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:28.696451 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec0b8f3e-b65f-4709-95ad-8dab56153f36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec0b8f3e-b65f-4709-95ad-8dab56153f36-plugins-config-volume\") pod \"ec0b8f3e-b65f-4709-95ad-8dab56153f36\" (UID: \"ec0b8f3e-b65f-4709-95ad-8dab56153f36\") " Nov 02 18:25:28.696657 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:28.696504 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0b8f3e-b65f-4709-95ad-8dab56153f36-kube-api-access-px9sf\" (UniqueName: \"kubernetes.io/projected/ec0b8f3e-b65f-4709-95ad-8dab56153f36-kube-api-access-px9sf\") pod \"ec0b8f3e-b65f-4709-95ad-8dab56153f36\" (UID: \"ec0b8f3e-b65f-4709-95ad-8dab56153f36\") " Nov 02 18:25:28.696844 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:28.696822 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec0b8f3e-b65f-4709-95ad-8dab56153f36-plugins-config-volume" pod "ec0b8f3e-b65f-4709-95ad-8dab56153f36" (UID: "ec0b8f3e-b65f-4709-95ad-8dab56153f36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:25:28.697233 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:28.697213 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec0b8f3e-b65f-4709-95ad-8dab56153f36-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:28.698481 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:28.698460 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0b8f3e-b65f-4709-95ad-8dab56153f36-kube-api-access-px9sf" pod "ec0b8f3e-b65f-4709-95ad-8dab56153f36" (UID: "ec0b8f3e-b65f-4709-95ad-8dab56153f36"). InnerVolumeSpecName "kube-api-access-px9sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:25:28.799270 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:28.799237 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px9sf\" (UniqueName: \"kubernetes.io/projected/ec0b8f3e-b65f-4709-95ad-8dab56153f36-kube-api-access-px9sf\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:28.891014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:28.890904 5549 generic.go:356] "Generic (PLEG): container finished" podID="ec0b8f3e-b65f-4709-95ad-8dab56153f36" containerID="4a1e6f017e8edddac7d9b290849d4e3af58bb2e0bb239bc33294324a0e2fddfe" exitCode=0 Nov 02 18:25:28.891014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:28.890969 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh" Nov 02 18:25:28.891014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:28.890998 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh" event={"ID":"ec0b8f3e-b65f-4709-95ad-8dab56153f36","Type":"ContainerDied","Data":"4a1e6f017e8edddac7d9b290849d4e3af58bb2e0bb239bc33294324a0e2fddfe"} Nov 02 18:25:28.891277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:28.891080 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh" event={"ID":"ec0b8f3e-b65f-4709-95ad-8dab56153f36","Type":"ContainerDied","Data":"629cf25d53aa16db78dc4e67154f00bf4a9e764304f8c2836b6fe807344242d3"} Nov 02 18:25:28.891277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:28.891102 5549 scope.go:122] "RemoveContainer" containerID="4a1e6f017e8edddac7d9b290849d4e3af58bb2e0bb239bc33294324a0e2fddfe" Nov 02 18:25:28.903145 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:28.903128 5549 scope.go:122] "RemoveContainer" containerID="4a1e6f017e8edddac7d9b290849d4e3af58bb2e0bb239bc33294324a0e2fddfe" Nov 02 18:25:28.903462 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:25:28.903440 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1e6f017e8edddac7d9b290849d4e3af58bb2e0bb239bc33294324a0e2fddfe\": container with ID starting with 4a1e6f017e8edddac7d9b290849d4e3af58bb2e0bb239bc33294324a0e2fddfe not found: ID does not exist" containerID="4a1e6f017e8edddac7d9b290849d4e3af58bb2e0bb239bc33294324a0e2fddfe" Nov 02 18:25:28.903519 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:28.903470 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1e6f017e8edddac7d9b290849d4e3af58bb2e0bb239bc33294324a0e2fddfe"} err="failed to get container status \"4a1e6f017e8edddac7d9b290849d4e3af58bb2e0bb239bc33294324a0e2fddfe\": rpc error: code = NotFound desc = could not find container \"4a1e6f017e8edddac7d9b290849d4e3af58bb2e0bb239bc33294324a0e2fddfe\": container with ID starting with 4a1e6f017e8edddac7d9b290849d4e3af58bb2e0bb239bc33294324a0e2fddfe not found: ID does not exist" Nov 02 18:25:28.918629 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:28.918601 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh"] Nov 02 18:25:28.923262 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:28.923239 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b3a6-58d74c4dc5-msbsh"] Nov 02 18:25:29.150852 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:29.150741 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:29.727907 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:29.727865 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:30.211568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:30.211524 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:30.323909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:30.323874 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec0b8f3e-b65f-4709-95ad-8dab56153f36" path="/var/lib/kubelet/pods/ec0b8f3e-b65f-4709-95ad-8dab56153f36/volumes" Nov 02 18:25:30.955781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:30.955746 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:31.331534 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:31.331447 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:31.480752 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:31.480700 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:31.773234 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:31.773202 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:31.918414 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:31.918375 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:32.094547 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.094461 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr"] Nov 02 18:25:32.094750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.094726 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" podUID="b3110f2f-be09-4cbb-9687-e642a0268441" containerName="istio-proxy" containerID="cri-o://16d2b804ef90eb3ab1637fe274ef6e14f8ec47a94773305df2fd15997344ae16" gracePeriod=30 Nov 02 18:25:32.103602 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.103574 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw"] Nov 02 18:25:32.103830 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.103810 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" podUID="408a8909-9fa2-43ab-8bd9-1dbc52cefb95" containerName="istio-proxy" containerID="cri-o://3c68bbf2eed35a8261c795b30ce8cf65c0b863d0658d56ec7fc914c40cdbfe01" gracePeriod=30 Nov 02 18:25:32.173840 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.173800 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:32.628402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.628359 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" podUID="96fcaedb-4f59-4385-92c4-5c04bbef5d20" containerName="main" containerID="cri-o://593e7b0e6766d0be75c34dd949aefe7176ff90dcf900a2ffddb0b96061906934" gracePeriod=45 Nov 02 18:25:32.904713 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.904677 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:32.960607 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.960568 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:25:32.980906 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.980873 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-tmp-dir\") pod \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " Nov 02 18:25:32.981112 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.980987 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-model-cache\") pod \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " Nov 02 18:25:32.981184 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.981169 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-dshm\") pod \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " Nov 02 18:25:32.981241 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.981208 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-tmp-dir" pod "96fcaedb-4f59-4385-92c4-5c04bbef5d20" (UID: "96fcaedb-4f59-4385-92c4-5c04bbef5d20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:32.981339 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.981304 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96fcaedb-4f59-4385-92c4-5c04bbef5d20-tls-certs\" (UniqueName: \"kubernetes.io/secret/96fcaedb-4f59-4385-92c4-5c04bbef5d20-tls-certs\") pod \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " Nov 02 18:25:32.981412 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.981346 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96fcaedb-4f59-4385-92c4-5c04bbef5d20-kube-api-access-xqr7q\" (UniqueName: \"kubernetes.io/projected/96fcaedb-4f59-4385-92c4-5c04bbef5d20-kube-api-access-xqr7q\") pod \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " Nov 02 18:25:32.981412 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.981353 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-model-cache" pod "96fcaedb-4f59-4385-92c4-5c04bbef5d20" (UID: "96fcaedb-4f59-4385-92c4-5c04bbef5d20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:32.981523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.981468 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-home\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-home\") pod \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\" (UID: \"96fcaedb-4f59-4385-92c4-5c04bbef5d20\") " Nov 02 18:25:32.981969 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.981931 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-home" pod "96fcaedb-4f59-4385-92c4-5c04bbef5d20" (UID: "96fcaedb-4f59-4385-92c4-5c04bbef5d20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:32.982314 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.982296 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:32.982368 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.982324 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:32.982368 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.982340 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:32.984269 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.984239 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96fcaedb-4f59-4385-92c4-5c04bbef5d20-tls-certs" pod "96fcaedb-4f59-4385-92c4-5c04bbef5d20" (UID: "96fcaedb-4f59-4385-92c4-5c04bbef5d20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:25:32.984375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.984248 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-dshm" pod "96fcaedb-4f59-4385-92c4-5c04bbef5d20" (UID: "96fcaedb-4f59-4385-92c4-5c04bbef5d20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:32.985678 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:32.985651 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96fcaedb-4f59-4385-92c4-5c04bbef5d20-kube-api-access-xqr7q" pod "96fcaedb-4f59-4385-92c4-5c04bbef5d20" (UID: "96fcaedb-4f59-4385-92c4-5c04bbef5d20"). InnerVolumeSpecName "kube-api-access-xqr7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:25:33.039327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:33.039288 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:33.053255 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:33.053221 5549 generic.go:356] "Generic (PLEG): container finished" podID="96fcaedb-4f59-4385-92c4-5c04bbef5d20" containerID="593e7b0e6766d0be75c34dd949aefe7176ff90dcf900a2ffddb0b96061906934" exitCode=0 Nov 02 18:25:33.053412 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:33.053274 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" Nov 02 18:25:33.053412 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:33.053316 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" event={"ID":"96fcaedb-4f59-4385-92c4-5c04bbef5d20","Type":"ContainerDied","Data":"593e7b0e6766d0be75c34dd949aefe7176ff90dcf900a2ffddb0b96061906934"} Nov 02 18:25:33.053412 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:33.053359 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" event={"ID":"96fcaedb-4f59-4385-92c4-5c04bbef5d20","Type":"ContainerDied","Data":"aa6f179742932efd8590a26bc627c486191b88f10c22b3e8d2ce8d99f3937604"} Nov 02 18:25:33.053412 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:33.053383 5549 scope.go:122] "RemoveContainer" containerID="593e7b0e6766d0be75c34dd949aefe7176ff90dcf900a2ffddb0b96061906934" Nov 02 18:25:33.075233 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:33.075196 5549 scope.go:122] "RemoveContainer" containerID="593e7b0e6766d0be75c34dd949aefe7176ff90dcf900a2ffddb0b96061906934" Nov 02 18:25:33.076268 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:25:33.076223 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"593e7b0e6766d0be75c34dd949aefe7176ff90dcf900a2ffddb0b96061906934\": container with ID starting with 593e7b0e6766d0be75c34dd949aefe7176ff90dcf900a2ffddb0b96061906934 not found: ID does not exist" containerID="593e7b0e6766d0be75c34dd949aefe7176ff90dcf900a2ffddb0b96061906934" Nov 02 18:25:33.076395 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:33.076273 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"593e7b0e6766d0be75c34dd949aefe7176ff90dcf900a2ffddb0b96061906934"} err="failed to get container status \"593e7b0e6766d0be75c34dd949aefe7176ff90dcf900a2ffddb0b96061906934\": rpc error: code = NotFound desc = could not find container \"593e7b0e6766d0be75c34dd949aefe7176ff90dcf900a2ffddb0b96061906934\": container with ID starting with 593e7b0e6766d0be75c34dd949aefe7176ff90dcf900a2ffddb0b96061906934 not found: ID does not exist" Nov 02 18:25:33.083665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:33.083641 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96fcaedb-4f59-4385-92c4-5c04bbef5d20-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:33.083665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:33.083666 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96fcaedb-4f59-4385-92c4-5c04bbef5d20-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:33.083795 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:33.083697 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqr7q\" (UniqueName: \"kubernetes.io/projected/96fcaedb-4f59-4385-92c4-5c04bbef5d20-kube-api-access-xqr7q\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:33.093520 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:33.093496 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh"] Nov 02 18:25:33.097278 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:33.097257 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh"] Nov 02 18:25:33.792850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:33.792819 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:33.868273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:33.868236 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-06644b/test-model-06644b-kserve-66c55b658c-hjnvh" podUID="96fcaedb-4f59-4385-92c4-5c04bbef5d20" containerName="main" probeResult="failure" output="Get \"https://10.133.1.40:8000/ready\": context deadline exceeded" Nov 02 18:25:34.322602 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:34.322568 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96fcaedb-4f59-4385-92c4-5c04bbef5d20" path="/var/lib/kubelet/pods/96fcaedb-4f59-4385-92c4-5c04bbef5d20/volumes" Nov 02 18:25:34.516702 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:34.516666 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:34.661678 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:34.661583 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:34.996226 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:34.996176 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:35.415273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:35.412399 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:36.310247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:36.310208 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:37.184729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.183634 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:37.431697 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.431671 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:25:37.436204 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.436181 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:25:37.590554 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.590519 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-credential-socket\") pod \"b3110f2f-be09-4cbb-9687-e642a0268441\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " Nov 02 18:25:37.590729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.590593 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-token\" (UniqueName: \"kubernetes.io/projected/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-token\") pod \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " Nov 02 18:25:37.590729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.590614 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3110f2f-be09-4cbb-9687-e642a0268441-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3110f2f-be09-4cbb-9687-e642a0268441-istio-podinfo\") pod \"b3110f2f-be09-4cbb-9687-e642a0268441\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " Nov 02 18:25:37.590729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.590649 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-istio-envoy\") pod \"b3110f2f-be09-4cbb-9687-e642a0268441\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " Nov 02 18:25:37.590729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.590680 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-workload-certs\") pod \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " Nov 02 18:25:37.590729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.590711 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-envoy\") pod \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " Nov 02 18:25:37.591135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.590756 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-workload-socket\") pod \"b3110f2f-be09-4cbb-9687-e642a0268441\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " Nov 02 18:25:37.591135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.590794 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3110f2f-be09-4cbb-9687-e642a0268441-istio-token\" (UniqueName: \"kubernetes.io/projected/b3110f2f-be09-4cbb-9687-e642a0268441-istio-token\") pod \"b3110f2f-be09-4cbb-9687-e642a0268441\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " Nov 02 18:25:37.591135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.590841 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3110f2f-be09-4cbb-9687-e642a0268441-kube-api-access-pwqlz\" (UniqueName: \"kubernetes.io/projected/b3110f2f-be09-4cbb-9687-e642a0268441-kube-api-access-pwqlz\") pod \"b3110f2f-be09-4cbb-9687-e642a0268441\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " Nov 02 18:25:37.591135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.590857 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-credential-socket" pod "b3110f2f-be09-4cbb-9687-e642a0268441" (UID: "b3110f2f-be09-4cbb-9687-e642a0268441"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:37.591135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.590892 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-kube-api-access-zr6h8\" (UniqueName: \"kubernetes.io/projected/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-kube-api-access-zr6h8\") pod \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " Nov 02 18:25:37.591135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.591007 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-data\") pod \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " Nov 02 18:25:37.591135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.591021 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-workload-certs" pod "408a8909-9fa2-43ab-8bd9-1dbc52cefb95" (UID: "408a8909-9fa2-43ab-8bd9-1dbc52cefb95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:37.591135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.591064 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-podinfo\") pod \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " Nov 02 18:25:37.591588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.591147 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-istio-data\") pod \"b3110f2f-be09-4cbb-9687-e642a0268441\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " Nov 02 18:25:37.591588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.591191 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-credential-socket\") pod \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " Nov 02 18:25:37.591588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.591242 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-workload-socket" pod "b3110f2f-be09-4cbb-9687-e642a0268441" (UID: "b3110f2f-be09-4cbb-9687-e642a0268441"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:37.591588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.591280 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3110f2f-be09-4cbb-9687-e642a0268441-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3110f2f-be09-4cbb-9687-e642a0268441-istiod-ca-cert\") pod \"b3110f2f-be09-4cbb-9687-e642a0268441\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " Nov 02 18:25:37.591588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.591357 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-workload-socket\") pod \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " Nov 02 18:25:37.591588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.591422 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-data" pod "408a8909-9fa2-43ab-8bd9-1dbc52cefb95" (UID: "408a8909-9fa2-43ab-8bd9-1dbc52cefb95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:37.591588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.591487 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istiod-ca-cert\") pod \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\" (UID: \"408a8909-9fa2-43ab-8bd9-1dbc52cefb95\") " Nov 02 18:25:37.591588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.591540 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-workload-certs\") pod \"b3110f2f-be09-4cbb-9687-e642a0268441\" (UID: \"b3110f2f-be09-4cbb-9687-e642a0268441\") " Nov 02 18:25:37.592070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.591643 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-istio-data" pod "b3110f2f-be09-4cbb-9687-e642a0268441" (UID: "b3110f2f-be09-4cbb-9687-e642a0268441"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:37.592070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.591830 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-workload-socket" pod "408a8909-9fa2-43ab-8bd9-1dbc52cefb95" (UID: "408a8909-9fa2-43ab-8bd9-1dbc52cefb95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:37.592187 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.592090 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-credential-socket" pod "408a8909-9fa2-43ab-8bd9-1dbc52cefb95" (UID: "408a8909-9fa2-43ab-8bd9-1dbc52cefb95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:37.592187 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.592076 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-workload-certs" pod "b3110f2f-be09-4cbb-9687-e642a0268441" (UID: "b3110f2f-be09-4cbb-9687-e642a0268441"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:37.592309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.592216 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3110f2f-be09-4cbb-9687-e642a0268441-istiod-ca-cert" pod "b3110f2f-be09-4cbb-9687-e642a0268441" (UID: "b3110f2f-be09-4cbb-9687-e642a0268441"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:25:37.592309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.592270 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istiod-ca-cert" pod "408a8909-9fa2-43ab-8bd9-1dbc52cefb95" (UID: "408a8909-9fa2-43ab-8bd9-1dbc52cefb95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:25:37.592933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.592904 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:37.593060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.592936 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:37.593060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.592974 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3110f2f-be09-4cbb-9687-e642a0268441-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:37.593060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.592990 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:37.593060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.593004 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:37.593060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.593020 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:37.593060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.593034 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:37.593060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.593049 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:37.593502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.593064 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:37.593502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.593134 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:37.594208 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.594180 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-token" pod "408a8909-9fa2-43ab-8bd9-1dbc52cefb95" (UID: "408a8909-9fa2-43ab-8bd9-1dbc52cefb95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:25:37.594340 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.594239 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3110f2f-be09-4cbb-9687-e642a0268441-istio-podinfo" pod "b3110f2f-be09-4cbb-9687-e642a0268441" (UID: "b3110f2f-be09-4cbb-9687-e642a0268441"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:25:37.594340 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.594250 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-kube-api-access-zr6h8" pod "408a8909-9fa2-43ab-8bd9-1dbc52cefb95" (UID: "408a8909-9fa2-43ab-8bd9-1dbc52cefb95"). InnerVolumeSpecName "kube-api-access-zr6h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:25:37.594340 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.594283 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3110f2f-be09-4cbb-9687-e642a0268441-istio-token" pod "b3110f2f-be09-4cbb-9687-e642a0268441" (UID: "b3110f2f-be09-4cbb-9687-e642a0268441"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:25:37.594340 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.594289 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-istio-envoy" pod "b3110f2f-be09-4cbb-9687-e642a0268441" (UID: "b3110f2f-be09-4cbb-9687-e642a0268441"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:37.594651 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.594629 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-envoy" pod "408a8909-9fa2-43ab-8bd9-1dbc52cefb95" (UID: "408a8909-9fa2-43ab-8bd9-1dbc52cefb95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:37.594751 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.594632 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3110f2f-be09-4cbb-9687-e642a0268441-kube-api-access-pwqlz" pod "b3110f2f-be09-4cbb-9687-e642a0268441" (UID: "b3110f2f-be09-4cbb-9687-e642a0268441"). InnerVolumeSpecName "kube-api-access-pwqlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:25:37.595026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.595008 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-podinfo" pod "408a8909-9fa2-43ab-8bd9-1dbc52cefb95" (UID: "408a8909-9fa2-43ab-8bd9-1dbc52cefb95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:25:37.694781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.694687 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:37.694781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.694725 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3110f2f-be09-4cbb-9687-e642a0268441-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:37.694781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.694735 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3110f2f-be09-4cbb-9687-e642a0268441-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:37.694781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.694747 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:37.694781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.694756 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3110f2f-be09-4cbb-9687-e642a0268441-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:37.694781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.694765 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwqlz\" (UniqueName: \"kubernetes.io/projected/b3110f2f-be09-4cbb-9687-e642a0268441-kube-api-access-pwqlz\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:37.694781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.694775 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr6h8\" (UniqueName: \"kubernetes.io/projected/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-kube-api-access-zr6h8\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:37.694781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.694783 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408a8909-9fa2-43ab-8bd9-1dbc52cefb95-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:37.990695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:37.990633 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:38.101681 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.101641 5549 generic.go:356] "Generic (PLEG): container finished" podID="408a8909-9fa2-43ab-8bd9-1dbc52cefb95" containerID="3c68bbf2eed35a8261c795b30ce8cf65c0b863d0658d56ec7fc914c40cdbfe01" exitCode=0 Nov 02 18:25:38.101852 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.101691 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" Nov 02 18:25:38.101852 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.101729 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" event={"ID":"408a8909-9fa2-43ab-8bd9-1dbc52cefb95","Type":"ContainerDied","Data":"3c68bbf2eed35a8261c795b30ce8cf65c0b863d0658d56ec7fc914c40cdbfe01"} Nov 02 18:25:38.101852 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.101787 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw" event={"ID":"408a8909-9fa2-43ab-8bd9-1dbc52cefb95","Type":"ContainerDied","Data":"f7150f585c713e7be1ea46d5f198a427f467cfe6a2771dcb37661275576efa1c"} Nov 02 18:25:38.101852 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.101819 5549 scope.go:122] "RemoveContainer" containerID="3c68bbf2eed35a8261c795b30ce8cf65c0b863d0658d56ec7fc914c40cdbfe01" Nov 02 18:25:38.104041 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.104011 5549 generic.go:356] "Generic (PLEG): container finished" podID="b3110f2f-be09-4cbb-9687-e642a0268441" containerID="16d2b804ef90eb3ab1637fe274ef6e14f8ec47a94773305df2fd15997344ae16" exitCode=0 Nov 02 18:25:38.104139 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.104061 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" Nov 02 18:25:38.104139 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.104085 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" event={"ID":"b3110f2f-be09-4cbb-9687-e642a0268441","Type":"ContainerDied","Data":"16d2b804ef90eb3ab1637fe274ef6e14f8ec47a94773305df2fd15997344ae16"} Nov 02 18:25:38.104139 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.104132 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr" event={"ID":"b3110f2f-be09-4cbb-9687-e642a0268441","Type":"ContainerDied","Data":"ec30ff6c4c2d19253fb6f459f93305a1f4416b8bfaa59d5370eefa837f81edaf"} Nov 02 18:25:38.113967 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.113926 5549 scope.go:122] "RemoveContainer" containerID="3c68bbf2eed35a8261c795b30ce8cf65c0b863d0658d56ec7fc914c40cdbfe01" Nov 02 18:25:38.114263 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:25:38.114241 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c68bbf2eed35a8261c795b30ce8cf65c0b863d0658d56ec7fc914c40cdbfe01\": container with ID starting with 3c68bbf2eed35a8261c795b30ce8cf65c0b863d0658d56ec7fc914c40cdbfe01 not found: ID does not exist" containerID="3c68bbf2eed35a8261c795b30ce8cf65c0b863d0658d56ec7fc914c40cdbfe01" Nov 02 18:25:38.114338 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.114269 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c68bbf2eed35a8261c795b30ce8cf65c0b863d0658d56ec7fc914c40cdbfe01"} err="failed to get container status \"3c68bbf2eed35a8261c795b30ce8cf65c0b863d0658d56ec7fc914c40cdbfe01\": rpc error: code = NotFound desc = could not find container \"3c68bbf2eed35a8261c795b30ce8cf65c0b863d0658d56ec7fc914c40cdbfe01\": container with ID starting with 3c68bbf2eed35a8261c795b30ce8cf65c0b863d0658d56ec7fc914c40cdbfe01 not found: ID does not exist" Nov 02 18:25:38.114338 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.114286 5549 scope.go:122] "RemoveContainer" containerID="16d2b804ef90eb3ab1637fe274ef6e14f8ec47a94773305df2fd15997344ae16" Nov 02 18:25:38.138984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.138931 5549 scope.go:122] "RemoveContainer" containerID="bc33d8602d13b5912358056b5ca160550230851af9a698f74bfa68fb19a90f6a" Nov 02 18:25:38.144655 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.144631 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:38.149858 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.149837 5549 scope.go:122] "RemoveContainer" containerID="16d2b804ef90eb3ab1637fe274ef6e14f8ec47a94773305df2fd15997344ae16" Nov 02 18:25:38.150197 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:25:38.150175 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d2b804ef90eb3ab1637fe274ef6e14f8ec47a94773305df2fd15997344ae16\": container with ID starting with 16d2b804ef90eb3ab1637fe274ef6e14f8ec47a94773305df2fd15997344ae16 not found: ID does not exist" containerID="16d2b804ef90eb3ab1637fe274ef6e14f8ec47a94773305df2fd15997344ae16" Nov 02 18:25:38.150280 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.150207 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d2b804ef90eb3ab1637fe274ef6e14f8ec47a94773305df2fd15997344ae16"} err="failed to get container status \"16d2b804ef90eb3ab1637fe274ef6e14f8ec47a94773305df2fd15997344ae16\": rpc error: code = NotFound desc = could not find container \"16d2b804ef90eb3ab1637fe274ef6e14f8ec47a94773305df2fd15997344ae16\": container with ID starting with 16d2b804ef90eb3ab1637fe274ef6e14f8ec47a94773305df2fd15997344ae16 not found: ID does not exist" Nov 02 18:25:38.150280 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.150227 5549 scope.go:122] "RemoveContainer" containerID="bc33d8602d13b5912358056b5ca160550230851af9a698f74bfa68fb19a90f6a" Nov 02 18:25:38.150444 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:25:38.150424 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc33d8602d13b5912358056b5ca160550230851af9a698f74bfa68fb19a90f6a\": container with ID starting with bc33d8602d13b5912358056b5ca160550230851af9a698f74bfa68fb19a90f6a not found: ID does not exist" containerID="bc33d8602d13b5912358056b5ca160550230851af9a698f74bfa68fb19a90f6a" Nov 02 18:25:38.150482 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.150449 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc33d8602d13b5912358056b5ca160550230851af9a698f74bfa68fb19a90f6a"} err="failed to get container status \"bc33d8602d13b5912358056b5ca160550230851af9a698f74bfa68fb19a90f6a\": rpc error: code = NotFound desc = could not find container \"bc33d8602d13b5912358056b5ca160550230851af9a698f74bfa68fb19a90f6a\": container with ID starting with bc33d8602d13b5912358056b5ca160550230851af9a698f74bfa68fb19a90f6a not found: ID does not exist" Nov 02 18:25:38.153493 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.153468 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr"] Nov 02 18:25:38.157557 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.157533 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-wznbr"] Nov 02 18:25:38.179208 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.179172 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw"] Nov 02 18:25:38.182081 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.181869 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-50b3a6-openshift-default-77b64b67fc-6k9cw"] Nov 02 18:25:38.328798 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.328716 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408a8909-9fa2-43ab-8bd9-1dbc52cefb95" path="/var/lib/kubelet/pods/408a8909-9fa2-43ab-8bd9-1dbc52cefb95/volumes" Nov 02 18:25:38.329822 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:38.329797 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3110f2f-be09-4cbb-9687-e642a0268441" path="/var/lib/kubelet/pods/b3110f2f-be09-4cbb-9687-e642a0268441/volumes" Nov 02 18:25:39.114350 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:39.113353 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:39.751308 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:39.751275 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:40.003149 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:40.003058 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:40.358489 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:40.358399 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt"] Nov 02 18:25:40.359021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:40.358655 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" podUID="b85a86f2-b60d-4dc6-aeb6-be4872586f40" containerName="maas-api" containerID="cri-o://d6c7935d3ee434b7ce1866f97043a4d0a3551a83365bd3a1e3168e45de75161e" gracePeriod=30 Nov 02 18:25:40.439081 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:40.439049 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:40.755687 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:40.755308 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" Nov 02 18:25:40.807393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:40.806096 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b85a86f2-b60d-4dc6-aeb6-be4872586f40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85a86f2-b60d-4dc6-aeb6-be4872586f40-openshift-service-ca\") pod \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\" (UID: \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\") " Nov 02 18:25:40.807393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:40.806181 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b85a86f2-b60d-4dc6-aeb6-be4872586f40-kube-api-access-s4ztl\" (UniqueName: \"kubernetes.io/projected/b85a86f2-b60d-4dc6-aeb6-be4872586f40-kube-api-access-s4ztl\") pod \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\" (UID: \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\") " Nov 02 18:25:40.807393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:40.806227 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b85a86f2-b60d-4dc6-aeb6-be4872586f40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85a86f2-b60d-4dc6-aeb6-be4872586f40-metrics-certs\") pod \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\" (UID: \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\") " Nov 02 18:25:40.807393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:40.806331 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b85a86f2-b60d-4dc6-aeb6-be4872586f40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85a86f2-b60d-4dc6-aeb6-be4872586f40-maas-api-tls\") pod \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\" (UID: \"b85a86f2-b60d-4dc6-aeb6-be4872586f40\") " Nov 02 18:25:40.807393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:40.806868 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b85a86f2-b60d-4dc6-aeb6-be4872586f40-openshift-service-ca" pod "b85a86f2-b60d-4dc6-aeb6-be4872586f40" (UID: "b85a86f2-b60d-4dc6-aeb6-be4872586f40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:25:40.807393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:40.807342 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85a86f2-b60d-4dc6-aeb6-be4872586f40-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:40.810658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:40.810622 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b85a86f2-b60d-4dc6-aeb6-be4872586f40-metrics-certs" pod "b85a86f2-b60d-4dc6-aeb6-be4872586f40" (UID: "b85a86f2-b60d-4dc6-aeb6-be4872586f40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:25:40.820210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:40.820173 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b85a86f2-b60d-4dc6-aeb6-be4872586f40-maas-api-tls" pod "b85a86f2-b60d-4dc6-aeb6-be4872586f40" (UID: "b85a86f2-b60d-4dc6-aeb6-be4872586f40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:25:40.821115 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:40.821086 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85a86f2-b60d-4dc6-aeb6-be4872586f40-kube-api-access-s4ztl" pod "b85a86f2-b60d-4dc6-aeb6-be4872586f40" (UID: "b85a86f2-b60d-4dc6-aeb6-be4872586f40"). InnerVolumeSpecName "kube-api-access-s4ztl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:25:40.909533 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:40.909457 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4ztl\" (UniqueName: \"kubernetes.io/projected/b85a86f2-b60d-4dc6-aeb6-be4872586f40-kube-api-access-s4ztl\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:40.909533 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:40.909489 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85a86f2-b60d-4dc6-aeb6-be4872586f40-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:40.909533 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:40.909499 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85a86f2-b60d-4dc6-aeb6-be4872586f40-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:41.018265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:41.017998 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:41.140904 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:41.140860 5549 generic.go:356] "Generic (PLEG): container finished" podID="b85a86f2-b60d-4dc6-aeb6-be4872586f40" containerID="d6c7935d3ee434b7ce1866f97043a4d0a3551a83365bd3a1e3168e45de75161e" exitCode=0 Nov 02 18:25:41.141095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:41.140915 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" Nov 02 18:25:41.141095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:41.140945 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" event={"ID":"b85a86f2-b60d-4dc6-aeb6-be4872586f40","Type":"ContainerDied","Data":"d6c7935d3ee434b7ce1866f97043a4d0a3551a83365bd3a1e3168e45de75161e"} Nov 02 18:25:41.141095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:41.141030 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt" event={"ID":"b85a86f2-b60d-4dc6-aeb6-be4872586f40","Type":"ContainerDied","Data":"541ac9dacbbccf2fbb6e081a5712dd76f9b9255a8fab1ba9c567808ba0cd97cc"} Nov 02 18:25:41.141095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:41.141054 5549 scope.go:122] "RemoveContainer" containerID="d6c7935d3ee434b7ce1866f97043a4d0a3551a83365bd3a1e3168e45de75161e" Nov 02 18:25:41.154271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:41.154242 5549 scope.go:122] "RemoveContainer" containerID="d6c7935d3ee434b7ce1866f97043a4d0a3551a83365bd3a1e3168e45de75161e" Nov 02 18:25:41.154633 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:25:41.154601 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c7935d3ee434b7ce1866f97043a4d0a3551a83365bd3a1e3168e45de75161e\": container with ID starting with d6c7935d3ee434b7ce1866f97043a4d0a3551a83365bd3a1e3168e45de75161e not found: ID does not exist" containerID="d6c7935d3ee434b7ce1866f97043a4d0a3551a83365bd3a1e3168e45de75161e" Nov 02 18:25:41.154735 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:41.154639 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c7935d3ee434b7ce1866f97043a4d0a3551a83365bd3a1e3168e45de75161e"} err="failed to get container status \"d6c7935d3ee434b7ce1866f97043a4d0a3551a83365bd3a1e3168e45de75161e\": rpc error: code = NotFound desc = could not find container \"d6c7935d3ee434b7ce1866f97043a4d0a3551a83365bd3a1e3168e45de75161e\": container with ID starting with d6c7935d3ee434b7ce1866f97043a4d0a3551a83365bd3a1e3168e45de75161e not found: ID does not exist" Nov 02 18:25:41.172687 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:41.172606 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt"] Nov 02 18:25:41.177795 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:41.177766 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-06644b-775d57cdc9-cb8kt"] Nov 02 18:25:41.280737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:41.280701 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:41.778862 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:41.770609 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:42.322816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:42.322779 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b85a86f2-b60d-4dc6-aeb6-be4872586f40" path="/var/lib/kubelet/pods/b85a86f2-b60d-4dc6-aeb6-be4872586f40/volumes" Nov 02 18:25:42.450888 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:42.450853 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:43.116315 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:43.116162 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:43.850282 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:43.850236 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:44.536735 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:44.536690 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:45.203324 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:45.203282 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:45.848529 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:45.848460 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:46.480153 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:46.480114 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:46.620854 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:46.620810 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:46.927744 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:46.927652 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:47.268859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:47.268821 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:47.458537 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:47.458494 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:47.590616 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:25:47.590527 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:25:48.151154 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:48.151124 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:48.318850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:48.318804 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:48.467201 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:48.467162 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:48.609923 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:48.609884 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:49.168695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:49.168659 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:49.838947 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:49.831248 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:50.509021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:50.507944 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:51.280794 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:51.280753 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:51.926002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:51.925946 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:52.620765 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:52.620729 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:53.305291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:53.305253 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:54.140320 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:54.140283 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:54.955564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:54.955520 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:55.594278 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:55.594243 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:55.830133 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:55.830099 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:55.983238 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:55.983199 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:56.494741 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:56.494647 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:57.166371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.166277 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:57.693114 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.693082 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg"] Nov 02 18:25:57.693985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.693970 5549 cpu_manager.go:414] "containerMap: removing container" podUID="b85a86f2-b60d-4dc6-aeb6-be4872586f40" containerName="maas-api" Nov 02 18:25:57.694049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.693987 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b85a86f2-b60d-4dc6-aeb6-be4872586f40" containerName="maas-api" Nov 02 18:25:57.694049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.694006 5549 cpu_manager.go:414] "containerMap: removing container" podUID="ec0b8f3e-b65f-4709-95ad-8dab56153f36" containerName="payload-processing" Nov 02 18:25:57.694049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.694011 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec0b8f3e-b65f-4709-95ad-8dab56153f36" containerName="payload-processing" Nov 02 18:25:57.694049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.694031 5549 cpu_manager.go:414] "containerMap: removing container" podUID="408a8909-9fa2-43ab-8bd9-1dbc52cefb95" containerName="istio-proxy" Nov 02 18:25:57.694049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.694036 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408a8909-9fa2-43ab-8bd9-1dbc52cefb95" containerName="istio-proxy" Nov 02 18:25:57.694049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.694045 5549 cpu_manager.go:414] "containerMap: removing container" podUID="b3110f2f-be09-4cbb-9687-e642a0268441" containerName="istio-proxy" Nov 02 18:25:57.694249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.694052 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3110f2f-be09-4cbb-9687-e642a0268441" containerName="istio-proxy" Nov 02 18:25:57.694249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.694061 5549 cpu_manager.go:414] "containerMap: removing container" podUID="96fcaedb-4f59-4385-92c4-5c04bbef5d20" containerName="main" Nov 02 18:25:57.694249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.694066 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96fcaedb-4f59-4385-92c4-5c04bbef5d20" containerName="main" Nov 02 18:25:57.694249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.694086 5549 cpu_manager.go:414] "containerMap: removing container" podUID="b3110f2f-be09-4cbb-9687-e642a0268441" containerName="istio-proxy" Nov 02 18:25:57.694249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.694091 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3110f2f-be09-4cbb-9687-e642a0268441" containerName="istio-proxy" Nov 02 18:25:57.694249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.694205 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="408a8909-9fa2-43ab-8bd9-1dbc52cefb95" containerName="istio-proxy" Nov 02 18:25:57.694249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.694216 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3110f2f-be09-4cbb-9687-e642a0268441" containerName="istio-proxy" Nov 02 18:25:57.694249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.694223 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="96fcaedb-4f59-4385-92c4-5c04bbef5d20" containerName="main" Nov 02 18:25:57.694249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.694230 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3110f2f-be09-4cbb-9687-e642a0268441" containerName="istio-proxy" Nov 02 18:25:57.694249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.694240 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="b85a86f2-b60d-4dc6-aeb6-be4872586f40" containerName="maas-api" Nov 02 18:25:57.694249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.694248 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec0b8f3e-b65f-4709-95ad-8dab56153f36" containerName="payload-processing" Nov 02 18:25:57.696595 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.696571 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:57.700101 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.700085 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:57.706192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.706159 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-1e2281\"/\"rate-test-model-1e2281-kserve-self-signed-certs\"" Nov 02 18:25:57.706192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.706199 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-1e2281\"/\"kube-root-ca.crt\"" Nov 02 18:25:57.706427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.706235 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-1e2281\"/\"openshift-service-ca.crt\"" Nov 02 18:25:57.706427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.706274 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-1e2281\"/\"default-dockercfg-sd4jw\"" Nov 02 18:25:57.707826 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.707798 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg"] Nov 02 18:25:57.853864 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.853823 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/321897e4-9071-443e-a688-8c2aead4d6df-tls-certs\") pod \"rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:57.854427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.854401 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdnjj\" (UniqueName: \"kubernetes.io/projected/321897e4-9071-443e-a688-8c2aead4d6df-kube-api-access-kdnjj\") pod \"rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:57.854696 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.854678 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-model-cache\") pod \"rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:57.854829 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.854816 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-tmp-dir\") pod \"rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:57.855145 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.855127 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-dshm\") pod \"rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:57.872973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.855463 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-home\") pod \"rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:57.961028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.960947 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-home\") pod \"rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:57.961225 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.961087 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/321897e4-9071-443e-a688-8c2aead4d6df-tls-certs\") pod \"rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:57.961225 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.961114 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdnjj\" (UniqueName: \"kubernetes.io/projected/321897e4-9071-443e-a688-8c2aead4d6df-kube-api-access-kdnjj\") pod \"rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:57.961354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.961279 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-model-cache\") pod \"rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:57.961429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.961355 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-home\") pod \"rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:57.961491 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.961422 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-tmp-dir\") pod \"rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:57.961605 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.961588 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-dshm\") pod \"rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:57.961673 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.961631 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-model-cache\") pod \"rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:57.961673 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.961660 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-tmp-dir\") pod \"rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:57.964516 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.964495 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-dshm\") pod \"rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:57.964723 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.964705 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/321897e4-9071-443e-a688-8c2aead4d6df-tls-certs\") pod \"rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:57.969665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:57.969641 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdnjj\" (UniqueName: \"kubernetes.io/projected/321897e4-9071-443e-a688-8c2aead4d6df-kube-api-access-kdnjj\") pod \"rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:58.016809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:58.016768 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:25:58.135548 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:58.135477 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:58.301134 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:58.301102 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg"] Nov 02 18:25:58.302207 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:25:58.302183 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod321897e4_9071_443e_a688_8c2aead4d6df.slice/crio-75dbc888a56e84846f076a854faae26d7fdbcacf1538aff5193b4c3ca4134ba0 WatchSource:0}: Error finding container 75dbc888a56e84846f076a854faae26d7fdbcacf1538aff5193b4c3ca4134ba0: Status 404 returned error can't find the container with id 75dbc888a56e84846f076a854faae26d7fdbcacf1538aff5193b4c3ca4134ba0 Nov 02 18:25:58.884373 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:58.873083 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:25:59.311816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:59.311779 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" event={"ID":"321897e4-9071-443e-a688-8c2aead4d6df","Type":"ContainerStarted","Data":"e7b5673d23e63e7981b42c310309db7f2a1c4db4036ffcc9800e819db901c1ad"} Nov 02 18:25:59.311816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:59.311822 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" event={"ID":"321897e4-9071-443e-a688-8c2aead4d6df","Type":"ContainerStarted","Data":"75dbc888a56e84846f076a854faae26d7fdbcacf1538aff5193b4c3ca4134ba0"} Nov 02 18:25:59.330388 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:59.330331 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" podStartSLOduration=2.330313046 podStartE2EDuration="2.330313046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:25:59.32872489 +0000 UTC m=+3131.117230422" watchObservedRunningTime="2026-11-02 18:25:59.330313046 +0000 UTC m=+3131.118818579" Nov 02 18:25:59.645524 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:25:59.645429 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:00.017502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:00.017471 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:26:00.369923 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:00.369843 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:01.036169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:01.036127 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:01.673014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:01.672975 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:02.405152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:02.404239 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:02.538916 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:02.538875 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:02.956484 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:02.956446 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:03.259444 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:03.259350 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:03.793668 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:03.793633 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:03.933782 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:03.933750 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:04.121567 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:04.121485 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:04.361504 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:04.361473 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:04.872977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:04.867467 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:05.623407 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:05.623276 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:06.415879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:06.415837 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:07.015146 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:07.015097 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:07.831725 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:07.831682 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:08.017546 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:08.017507 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:26:08.030660 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:08.030633 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:26:08.392865 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:08.392828 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:26:08.609892 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:08.609854 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:09.403052 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:09.403010 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:10.206889 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:10.206791 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:10.921040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:10.920945 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:11.104924 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:11.104886 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:11.602367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:11.601721 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:12.260165 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:12.260032 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:12.769763 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:12.769619 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn"] Nov 02 18:26:12.778088 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:12.778057 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn"] Nov 02 18:26:12.778271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:12.778173 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:12.784530 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:12.784438 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-7771bf\"/\"openshift-service-ca.crt\"" Nov 02 18:26:12.784530 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:12.784506 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-7771bf\"/\"kube-root-ca.crt\"" Nov 02 18:26:12.785097 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:12.785071 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-7771bf\"/\"rate-test-model-7771bf-kserve-self-signed-certs\"" Nov 02 18:26:12.785234 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:12.785101 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-7771bf\"/\"default-dockercfg-fl2z6\"" Nov 02 18:26:12.861132 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:12.861089 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:12.910509 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:12.910478 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-dshm\") pod \"rate-test-model-7771bf-kserve-5797944949-vltvn\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:12.910682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:12.910540 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-model-cache\") pod \"rate-test-model-7771bf-kserve-5797944949-vltvn\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:12.910682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:12.910645 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs5lf\" (UniqueName: \"kubernetes.io/projected/672fc5b2-2136-4d06-bf2f-3f5581500ca3-kube-api-access-fs5lf\") pod \"rate-test-model-7771bf-kserve-5797944949-vltvn\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:12.910799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:12.910784 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-home\") pod \"rate-test-model-7771bf-kserve-5797944949-vltvn\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:12.910845 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:12.910814 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/672fc5b2-2136-4d06-bf2f-3f5581500ca3-tls-certs\") pod \"rate-test-model-7771bf-kserve-5797944949-vltvn\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:12.911001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:12.910966 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-tmp-dir\") pod \"rate-test-model-7771bf-kserve-5797944949-vltvn\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:13.012774 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:13.012725 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-dshm\") pod \"rate-test-model-7771bf-kserve-5797944949-vltvn\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:13.013040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:13.012824 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-model-cache\") pod \"rate-test-model-7771bf-kserve-5797944949-vltvn\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:13.013040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:13.012901 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs5lf\" (UniqueName: \"kubernetes.io/projected/672fc5b2-2136-4d06-bf2f-3f5581500ca3-kube-api-access-fs5lf\") pod \"rate-test-model-7771bf-kserve-5797944949-vltvn\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:13.013180 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:13.013056 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-home\") pod \"rate-test-model-7771bf-kserve-5797944949-vltvn\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:13.013180 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:13.013086 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/672fc5b2-2136-4d06-bf2f-3f5581500ca3-tls-certs\") pod \"rate-test-model-7771bf-kserve-5797944949-vltvn\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:13.013299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:13.013201 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-tmp-dir\") pod \"rate-test-model-7771bf-kserve-5797944949-vltvn\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:13.013684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:13.013651 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-tmp-dir\") pod \"rate-test-model-7771bf-kserve-5797944949-vltvn\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:13.013921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:13.013888 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-home\") pod \"rate-test-model-7771bf-kserve-5797944949-vltvn\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:13.014437 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:13.014408 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-model-cache\") pod \"rate-test-model-7771bf-kserve-5797944949-vltvn\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:13.016401 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:13.016375 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-dshm\") pod \"rate-test-model-7771bf-kserve-5797944949-vltvn\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:13.017122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:13.017100 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/672fc5b2-2136-4d06-bf2f-3f5581500ca3-tls-certs\") pod \"rate-test-model-7771bf-kserve-5797944949-vltvn\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:13.023546 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:13.023482 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs5lf\" (UniqueName: \"kubernetes.io/projected/672fc5b2-2136-4d06-bf2f-3f5581500ca3-kube-api-access-fs5lf\") pod \"rate-test-model-7771bf-kserve-5797944949-vltvn\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:13.100684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:13.100643 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:13.285707 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:13.285602 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:13.391706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:13.391674 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn"] Nov 02 18:26:13.392715 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:26:13.392684 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod672fc5b2_2136_4d06_bf2f_3f5581500ca3.slice/crio-7611f4558fe3608ad460ddc3e3a2bcae3c8b0399e0d5d965809163fe44c5db78 WatchSource:0}: Error finding container 7611f4558fe3608ad460ddc3e3a2bcae3c8b0399e0d5d965809163fe44c5db78: Status 404 returned error can't find the container with id 7611f4558fe3608ad460ddc3e3a2bcae3c8b0399e0d5d965809163fe44c5db78 Nov 02 18:26:13.439891 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:13.439862 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" event={"ID":"672fc5b2-2136-4d06-bf2f-3f5581500ca3","Type":"ContainerStarted","Data":"7611f4558fe3608ad460ddc3e3a2bcae3c8b0399e0d5d965809163fe44c5db78"} Nov 02 18:26:14.152312 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:14.152276 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:14.450290 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:14.450262 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" event={"ID":"672fc5b2-2136-4d06-bf2f-3f5581500ca3","Type":"ContainerStarted","Data":"88be9bebb782401d96b51603267a0fc4155f1d3336bbeae6326baccfeabd6ef6"} Nov 02 18:26:14.474145 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:14.474099 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" podStartSLOduration=2.474085972 podStartE2EDuration="2.474085972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:26:14.469240121 +0000 UTC m=+3146.257745694" watchObservedRunningTime="2026-11-02 18:26:14.474085972 +0000 UTC m=+3146.262591500" Nov 02 18:26:14.635454 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:14.635420 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:14.921046 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:14.917299 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:15.101308 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:15.101276 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:15.638052 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:15.636559 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:16.503052 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:16.503014 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:17.279589 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:17.271258 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:17.591171 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:26:17.591080 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:26:17.924648 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:17.924556 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:18.603494 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:18.603442 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:19.341904 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:19.341854 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:20.012115 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:20.011996 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:20.603912 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:20.603862 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:20.743295 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:20.743258 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:21.407131 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:21.407093 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:21.789187 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:21.789157 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:22.095683 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:22.095564 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:22.872996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:22.871937 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:23.101589 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:23.101548 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:23.114941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:23.114902 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:23.550939 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:23.550902 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:26:23.584573 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:23.584531 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:24.228201 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:24.228165 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:24.993459 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:24.993424 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:25.728802 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:25.728763 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:26.409092 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:26.409052 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:27.220826 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:27.220788 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:27.932416 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:27.932377 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:28.762340 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:28.762299 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:29.573404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:29.573349 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:29.757135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:29.757102 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:30.410317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:30.410192 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:31.114694 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:31.114651 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:31.836570 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:31.836533 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:32.528978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:32.526830 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:33.214113 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:33.214077 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:33.915804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:33.915765 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:34.703722 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:34.703684 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:35.394326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:35.394278 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:36.028908 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:36.028865 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:36.454428 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:36.454393 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:26:36.840089 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:36.839998 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:37.723631 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:37.723584 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:38.062033 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:38.061797 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn"] Nov 02 18:26:38.074872 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:38.068221 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" podUID="70149a4a-e73e-4b5e-8d39-3877dd4c5583" containerName="istio-proxy" containerID="cri-o://f5193e89ef6ab472e98e25776046fbd7ca0bca3d1f9e4ffc4f6411f223620a93" gracePeriod=30 Nov 02 18:26:38.742012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:38.741745 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:39.779886 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:39.776972 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:40.820456 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:40.820145 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:41.602001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:41.601197 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:42.487783 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:42.487751 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:43.154136 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.154018 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:43.756861 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.756835 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:26:43.776430 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.776401 5549 generic.go:356] "Generic (PLEG): container finished" podID="70149a4a-e73e-4b5e-8d39-3877dd4c5583" containerID="f5193e89ef6ab472e98e25776046fbd7ca0bca3d1f9e4ffc4f6411f223620a93" exitCode=0 Nov 02 18:26:43.776611 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.776537 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" Nov 02 18:26:43.776611 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.776576 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" event={"ID":"70149a4a-e73e-4b5e-8d39-3877dd4c5583","Type":"ContainerDied","Data":"f5193e89ef6ab472e98e25776046fbd7ca0bca3d1f9e4ffc4f6411f223620a93"} Nov 02 18:26:43.776719 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.776628 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn" event={"ID":"70149a4a-e73e-4b5e-8d39-3877dd4c5583","Type":"ContainerDied","Data":"4c8742aec990ddbfb0da47cb13364797f38e1af4f4988a6fdaf51fdebbe2f5aa"} Nov 02 18:26:43.776719 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.776648 5549 scope.go:122] "RemoveContainer" containerID="f5193e89ef6ab472e98e25776046fbd7ca0bca3d1f9e4ffc4f6411f223620a93" Nov 02 18:26:43.796354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.796326 5549 scope.go:122] "RemoveContainer" containerID="f5193e89ef6ab472e98e25776046fbd7ca0bca3d1f9e4ffc4f6411f223620a93" Nov 02 18:26:43.796735 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:26:43.796709 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5193e89ef6ab472e98e25776046fbd7ca0bca3d1f9e4ffc4f6411f223620a93\": container with ID starting with f5193e89ef6ab472e98e25776046fbd7ca0bca3d1f9e4ffc4f6411f223620a93 not found: ID does not exist" containerID="f5193e89ef6ab472e98e25776046fbd7ca0bca3d1f9e4ffc4f6411f223620a93" Nov 02 18:26:43.796812 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.796747 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5193e89ef6ab472e98e25776046fbd7ca0bca3d1f9e4ffc4f6411f223620a93"} err="failed to get container status \"f5193e89ef6ab472e98e25776046fbd7ca0bca3d1f9e4ffc4f6411f223620a93\": rpc error: code = NotFound desc = could not find container \"f5193e89ef6ab472e98e25776046fbd7ca0bca3d1f9e4ffc4f6411f223620a93\": container with ID starting with f5193e89ef6ab472e98e25776046fbd7ca0bca3d1f9e4ffc4f6411f223620a93 not found: ID does not exist" Nov 02 18:26:43.834596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.834557 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:43.897274 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.897181 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-credential-socket\") pod \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " Nov 02 18:26:43.897632 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.897589 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istiod-ca-cert\") pod \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " Nov 02 18:26:43.899548 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.899521 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70149a4a-e73e-4b5e-8d39-3877dd4c5583-kube-api-access-qhj65\" (UniqueName: \"kubernetes.io/projected/70149a4a-e73e-4b5e-8d39-3877dd4c5583-kube-api-access-qhj65\") pod \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " Nov 02 18:26:43.899753 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.899734 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-envoy\") pod \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " Nov 02 18:26:43.899912 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.899897 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-workload-certs\") pod \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " Nov 02 18:26:43.900720 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.898279 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-credential-socket" pod "70149a4a-e73e-4b5e-8d39-3877dd4c5583" (UID: "70149a4a-e73e-4b5e-8d39-3877dd4c5583"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:26:43.900720 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.900190 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istiod-ca-cert" pod "70149a4a-e73e-4b5e-8d39-3877dd4c5583" (UID: "70149a4a-e73e-4b5e-8d39-3877dd4c5583"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:26:43.902132 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.902096 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-workload-certs" pod "70149a4a-e73e-4b5e-8d39-3877dd4c5583" (UID: "70149a4a-e73e-4b5e-8d39-3877dd4c5583"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:26:43.904724 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.904683 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70149a4a-e73e-4b5e-8d39-3877dd4c5583-kube-api-access-qhj65" pod "70149a4a-e73e-4b5e-8d39-3877dd4c5583" (UID: "70149a4a-e73e-4b5e-8d39-3877dd4c5583"). InnerVolumeSpecName "kube-api-access-qhj65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:26:43.905240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.905216 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-data\") pod \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " Nov 02 18:26:43.905456 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.905344 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-workload-socket\") pod \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " Nov 02 18:26:43.905456 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.905417 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-token\" (UniqueName: \"kubernetes.io/projected/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-token\") pod \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " Nov 02 18:26:43.905880 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.905562 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-podinfo\") pod \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\" (UID: \"70149a4a-e73e-4b5e-8d39-3877dd4c5583\") " Nov 02 18:26:43.905880 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.905803 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-data" pod "70149a4a-e73e-4b5e-8d39-3877dd4c5583" (UID: "70149a4a-e73e-4b5e-8d39-3877dd4c5583"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:26:43.906494 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.906429 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-workload-socket" pod "70149a4a-e73e-4b5e-8d39-3877dd4c5583" (UID: "70149a4a-e73e-4b5e-8d39-3877dd4c5583"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:26:43.907037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.906889 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:26:43.907037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.906929 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:26:43.907037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.906944 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhj65\" (UniqueName: \"kubernetes.io/projected/70149a4a-e73e-4b5e-8d39-3877dd4c5583-kube-api-access-qhj65\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:26:43.907037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.906976 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:26:43.907037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.906990 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:26:43.907037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.907004 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:26:43.916521 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.916470 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-envoy" pod "70149a4a-e73e-4b5e-8d39-3877dd4c5583" (UID: "70149a4a-e73e-4b5e-8d39-3877dd4c5583"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:26:43.917678 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.917635 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-podinfo" pod "70149a4a-e73e-4b5e-8d39-3877dd4c5583" (UID: "70149a4a-e73e-4b5e-8d39-3877dd4c5583"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:26:43.921605 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:43.921501 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-token" pod "70149a4a-e73e-4b5e-8d39-3877dd4c5583" (UID: "70149a4a-e73e-4b5e-8d39-3877dd4c5583"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:26:44.008229 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:44.008189 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:26:44.008229 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:44.008222 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:26:44.008229 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:44.008232 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70149a4a-e73e-4b5e-8d39-3877dd4c5583-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:26:44.104186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:44.104149 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn"] Nov 02 18:26:44.108074 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:44.108042 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-kp2hn"] Nov 02 18:26:44.325904 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:44.325863 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70149a4a-e73e-4b5e-8d39-3877dd4c5583" path="/var/lib/kubelet/pods/70149a4a-e73e-4b5e-8d39-3877dd4c5583/volumes" Nov 02 18:26:44.531676 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:44.531640 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:45.216982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:45.216909 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:45.863411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:45.863145 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:46.595012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:46.594671 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:47.314075 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:47.314045 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:47.588695 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:26:47.588611 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:26:47.896310 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:47.895461 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:48.587320 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:48.587282 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:49.245816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:49.245650 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:49.926496 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:49.925340 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:50.701525 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:50.701348 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:51.352554 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:51.352519 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:52.018291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:52.018200 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:52.717765 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:52.717716 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:53.415877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:53.415785 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:54.100563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:54.100515 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:54.790973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:54.790882 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:55.512900 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:55.512861 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:56.213673 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:56.213518 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:56.346090 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:56.346056 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:56.936964 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:56.936617 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:57.803858 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:57.803805 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:58.616972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:58.615835 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:26:59.394043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:26:59.394004 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:00.093062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:00.093024 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:00.716433 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:00.716391 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:01.407046 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:01.407007 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:02.025293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:02.025240 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:02.603372 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:02.603337 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:02.801946 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:02.801909 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:03.440838 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.440795 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:03.811264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.811227 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp"] Nov 02 18:27:03.812167 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.812148 5549 cpu_manager.go:414] "containerMap: removing container" podUID="70149a4a-e73e-4b5e-8d39-3877dd4c5583" containerName="istio-proxy" Nov 02 18:27:03.812167 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.812168 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70149a4a-e73e-4b5e-8d39-3877dd4c5583" containerName="istio-proxy" Nov 02 18:27:03.812367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.812356 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="70149a4a-e73e-4b5e-8d39-3877dd4c5583" containerName="istio-proxy" Nov 02 18:27:03.817624 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.817603 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.822803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.822779 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-1e2281-openshift-default-dockercfg-xhl4q\"" Nov 02 18:27:03.828106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.828084 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp"] Nov 02 18:27:03.856730 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.856694 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.856903 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.856747 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.856995 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.856934 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wsgg\" (UniqueName: \"kubernetes.io/projected/da89e636-cb83-41a2-ae31-cfbbaff79e79-kube-api-access-6wsgg\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.857071 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.857048 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da89e636-cb83-41a2-ae31-cfbbaff79e79-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.857124 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.857086 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.857224 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.857205 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.857335 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.857319 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.857399 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.857349 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.857464 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.857402 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.958530 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.958492 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.958726 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.958594 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.958726 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.958613 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.958726 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.958647 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.958726 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.958710 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.959028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.958733 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.959028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.958867 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wsgg\" (UniqueName: \"kubernetes.io/projected/da89e636-cb83-41a2-ae31-cfbbaff79e79-kube-api-access-6wsgg\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.959028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.958908 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da89e636-cb83-41a2-ae31-cfbbaff79e79-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.959028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.958981 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.959028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.959014 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.959375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.959044 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.959375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.959316 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.959488 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.959376 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.959621 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.959603 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da89e636-cb83-41a2-ae31-cfbbaff79e79-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.961237 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.961206 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.961361 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.961237 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.967161 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.967135 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:03.967584 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:03.967568 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wsgg\" (UniqueName: \"kubernetes.io/projected/da89e636-cb83-41a2-ae31-cfbbaff79e79-kube-api-access-6wsgg\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:04.135110 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:04.135037 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:04.214975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:04.214606 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:04.422054 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:27:04.422023 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda89e636_cb83_41a2_ae31_cfbbaff79e79.slice/crio-07d316ec2d39af3b249bf4ec2a505536251e69ad9278357c1edb21a2de92c1e2 WatchSource:0}: Error finding container 07d316ec2d39af3b249bf4ec2a505536251e69ad9278357c1edb21a2de92c1e2: Status 404 returned error can't find the container with id 07d316ec2d39af3b249bf4ec2a505536251e69ad9278357c1edb21a2de92c1e2 Nov 02 18:27:04.422377 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:04.422355 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp"] Nov 02 18:27:04.424880 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:04.424839 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:27:04.425051 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:04.424929 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:27:04.425051 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:04.424992 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:27:04.814092 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:04.813530 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:04.965334 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:04.965289 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" event={"ID":"da89e636-cb83-41a2-ae31-cfbbaff79e79","Type":"ContainerStarted","Data":"bd174499444ccf7def32684f615c5c1df7b7e969875cdd96b9b47a61f215fe28"} Nov 02 18:27:04.965535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:04.965355 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" event={"ID":"da89e636-cb83-41a2-ae31-cfbbaff79e79","Type":"ContainerStarted","Data":"07d316ec2d39af3b249bf4ec2a505536251e69ad9278357c1edb21a2de92c1e2"} Nov 02 18:27:04.987214 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:04.987160 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" podStartSLOduration=1.987147494 podStartE2EDuration="1.987147494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:04.984357845 +0000 UTC m=+3196.772863373" watchObservedRunningTime="2026-11-02 18:27:04.987147494 +0000 UTC m=+3196.775653022" Nov 02 18:27:05.135672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:05.135580 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:05.137370 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:05.137344 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 02 18:27:05.137488 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:05.137398 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" podUID="da89e636-cb83-41a2-ae31-cfbbaff79e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 02 18:27:05.721417 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:05.721353 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:06.136489 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:06.136404 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 02 18:27:06.136489 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:06.136474 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" podUID="da89e636-cb83-41a2-ae31-cfbbaff79e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 02 18:27:06.403036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:06.400037 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:07.099979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:07.090437 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:07.135641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:07.135604 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 02 18:27:07.135818 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:07.135671 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" podUID="da89e636-cb83-41a2-ae31-cfbbaff79e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 02 18:27:07.833084 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:07.833045 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:08.136289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:08.136202 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 02 18:27:08.136289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:08.136260 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" podUID="da89e636-cb83-41a2-ae31-cfbbaff79e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 02 18:27:08.313164 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:08.313128 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:08.572996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:08.572273 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:09.135722 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:09.135684 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 02 18:27:09.136141 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:09.135736 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" podUID="da89e636-cb83-41a2-ae31-cfbbaff79e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 02 18:27:09.400064 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:09.399544 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:10.136131 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:10.136095 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 02 18:27:10.136552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:10.136150 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" podUID="da89e636-cb83-41a2-ae31-cfbbaff79e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 02 18:27:10.221138 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:10.207493 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:10.936976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:10.936481 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:11.141417 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:11.141385 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:11.141913 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:11.141894 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:11.142678 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:11.142656 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:27:11.666053 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:11.666012 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:12.311220 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:12.311182 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:13.003288 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:13.003249 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:13.659378 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:13.659342 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:14.431078 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:14.429942 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:15.011804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:15.006502 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:15.805025 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:15.799998 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:16.437757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:16.437693 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:16.503897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:16.503861 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:27:17.202542 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:17.202504 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:17.591598 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:27:17.591511 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:27:17.933271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:17.930533 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:18.529402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:18.529364 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:19.215601 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:19.215554 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:19.885381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:19.885325 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:20.599206 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:20.599165 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:21.324168 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:21.323938 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:22.115356 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:22.115314 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:22.782117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:22.782082 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:23.008471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:23.008438 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:23.721387 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:23.721344 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:24.396733 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:24.396702 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:29.063853 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:29.063817 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:29.217944 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:29.217874 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:29.428358 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:29.427383 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:30.231973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:30.230629 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:30.981000 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:30.980940 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:31.697781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:31.697719 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:32.347749 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:32.347704 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:32.936828 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:32.936785 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:33.583566 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:33.583533 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:33.717231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:33.717193 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:34.436983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:34.435593 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:35.177124 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:35.177088 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:35.336856 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:35.336820 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:36.204617 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:36.202815 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:37.040226 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:37.040191 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:37.624766 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:37.624633 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:38.325771 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:38.325732 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:39.134509 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:39.134102 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:39.801031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:39.799200 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:40.481411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:40.477335 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:41.444933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:41.444894 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:41.702063 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:41.701981 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:41.882485 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:41.882445 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:42.020050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:42.019930 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:42.440656 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:42.436258 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:43.190747 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:43.190644 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:43.899828 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:43.899787 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:44.405881 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:44.405848 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:44.686031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:44.659931 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:44.837892 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:44.837853 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc"] Nov 02 18:27:44.928519 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:44.928479 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:45.101876 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:45.101842 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:45.316759 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:45.316717 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:45.453813 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:45.453777 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:45.718666 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:45.718541 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:46.527638 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:46.527599 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:47.205102 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:47.205066 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:47.589629 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:27:47.589544 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:27:47.829583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:47.829544 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:48.536896 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:48.536836 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:48.688293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:48.688257 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:49.090508 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:49.090466 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:49.489585 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:49.489500 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:49.746459 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:49.746312 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:50.471619 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:50.470962 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:51.212858 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:51.212725 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:51.636752 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:51.636669 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd"] Nov 02 18:27:51.748203 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:51.748168 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:51.883296 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:51.883266 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:52.037245 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:52.037205 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:52.176246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:52.176199 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:52.423629 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:52.423530 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:52.553619 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:52.553585 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:52.860187 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:52.860140 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:52.999358 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:52.999282 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:53.698305 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:53.698210 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:54.344974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:54.343114 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:54.416225 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:54.416188 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz"] Nov 02 18:27:54.416415 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:54.416393 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" podUID="c9b89a1f-2e31-4335-9f88-acacb8b00000" containerName="istio-proxy" containerID="cri-o://b7b9b7a2f9f1af600e96042257949118f8c2bcfe7ad78863a30780f878323e2f" gracePeriod=30 Nov 02 18:27:55.118381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:55.117932 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:55.706606 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:55.706563 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:55.911608 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:55.911570 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw"] Nov 02 18:27:55.911857 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:55.911828 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" podUID="e56ccdbd-6737-4a84-9d80-59018a6b77ff" containerName="istio-proxy" containerID="cri-o://56a3e72ae6988fda9241dbdff978cbce31c14d0f3d38c950c58fb642dfb66e29" gracePeriod=30 Nov 02 18:27:56.499708 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:56.499657 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:56.637725 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:56.637645 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" podUID="f2b6e563-a46e-42e2-bc91-5a0277272b56" containerName="payload-pre-processing" containerID="cri-o://8d86ac4a6797893de697669125fbd7b745dd11e963b40cefff833832b52f4d14" gracePeriod=25 Nov 02 18:27:57.231007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:57.229290 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:57.873755 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:57.873724 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:58.651978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:58.651743 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:59.415555 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.415516 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:59.549770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.549732 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:27:59.676359 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.676324 5549 generic.go:356] "Generic (PLEG): container finished" podID="c9b89a1f-2e31-4335-9f88-acacb8b00000" containerID="b7b9b7a2f9f1af600e96042257949118f8c2bcfe7ad78863a30780f878323e2f" exitCode=0 Nov 02 18:27:59.676536 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.676397 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" event={"ID":"c9b89a1f-2e31-4335-9f88-acacb8b00000","Type":"ContainerDied","Data":"b7b9b7a2f9f1af600e96042257949118f8c2bcfe7ad78863a30780f878323e2f"} Nov 02 18:27:59.705750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.705721 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 18:27:59.776516 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.776476 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-workload-socket\") pod \"c9b89a1f-2e31-4335-9f88-acacb8b00000\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " Nov 02 18:27:59.776745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.776707 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-podinfo\") pod \"c9b89a1f-2e31-4335-9f88-acacb8b00000\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " Nov 02 18:27:59.776745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.776724 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-workload-socket" pod "c9b89a1f-2e31-4335-9f88-acacb8b00000" (UID: "c9b89a1f-2e31-4335-9f88-acacb8b00000"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:59.776892 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.776815 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-credential-socket\") pod \"c9b89a1f-2e31-4335-9f88-acacb8b00000\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " Nov 02 18:27:59.776976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.776927 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-token\" (UniqueName: \"kubernetes.io/projected/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-token\") pod \"c9b89a1f-2e31-4335-9f88-acacb8b00000\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " Nov 02 18:27:59.777043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.777002 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b89a1f-2e31-4335-9f88-acacb8b00000-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b89a1f-2e31-4335-9f88-acacb8b00000-istiod-ca-cert\") pod \"c9b89a1f-2e31-4335-9f88-acacb8b00000\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " Nov 02 18:27:59.777105 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.777093 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-workload-certs\") pod \"c9b89a1f-2e31-4335-9f88-acacb8b00000\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " Nov 02 18:27:59.777166 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.777145 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-data\") pod \"c9b89a1f-2e31-4335-9f88-acacb8b00000\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " Nov 02 18:27:59.777239 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.777212 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b89a1f-2e31-4335-9f88-acacb8b00000-kube-api-access-4h2t9\" (UniqueName: \"kubernetes.io/projected/c9b89a1f-2e31-4335-9f88-acacb8b00000-kube-api-access-4h2t9\") pod \"c9b89a1f-2e31-4335-9f88-acacb8b00000\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " Nov 02 18:27:59.777295 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.777265 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-envoy\") pod \"c9b89a1f-2e31-4335-9f88-acacb8b00000\" (UID: \"c9b89a1f-2e31-4335-9f88-acacb8b00000\") " Nov 02 18:27:59.777637 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.777141 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-credential-socket" pod "c9b89a1f-2e31-4335-9f88-acacb8b00000" (UID: "c9b89a1f-2e31-4335-9f88-acacb8b00000"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:59.777637 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.777468 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b89a1f-2e31-4335-9f88-acacb8b00000-istiod-ca-cert" pod "c9b89a1f-2e31-4335-9f88-acacb8b00000" (UID: "c9b89a1f-2e31-4335-9f88-acacb8b00000"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:27:59.777637 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.777551 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-data" pod "c9b89a1f-2e31-4335-9f88-acacb8b00000" (UID: "c9b89a1f-2e31-4335-9f88-acacb8b00000"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:59.777892 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.777736 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-workload-certs" pod "c9b89a1f-2e31-4335-9f88-acacb8b00000" (UID: "c9b89a1f-2e31-4335-9f88-acacb8b00000"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:59.778833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.778810 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:27:59.779068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.779051 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b89a1f-2e31-4335-9f88-acacb8b00000-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:27:59.779182 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.779170 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:27:59.779272 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.779259 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:27:59.779379 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.779367 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:27:59.779788 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.779750 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-podinfo" pod "c9b89a1f-2e31-4335-9f88-acacb8b00000" (UID: "c9b89a1f-2e31-4335-9f88-acacb8b00000"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:27:59.781495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.781471 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b89a1f-2e31-4335-9f88-acacb8b00000-kube-api-access-4h2t9" pod "c9b89a1f-2e31-4335-9f88-acacb8b00000" (UID: "c9b89a1f-2e31-4335-9f88-acacb8b00000"). InnerVolumeSpecName "kube-api-access-4h2t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:27:59.781853 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.781821 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-token" pod "c9b89a1f-2e31-4335-9f88-acacb8b00000" (UID: "c9b89a1f-2e31-4335-9f88-acacb8b00000"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:27:59.782055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.782024 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-envoy" pod "c9b89a1f-2e31-4335-9f88-acacb8b00000" (UID: "c9b89a1f-2e31-4335-9f88-acacb8b00000"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:59.881884 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.881850 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:27:59.881884 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.881884 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h2t9\" (UniqueName: \"kubernetes.io/projected/c9b89a1f-2e31-4335-9f88-acacb8b00000-kube-api-access-4h2t9\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:27:59.882207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.881898 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:27:59.882207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.881912 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b89a1f-2e31-4335-9f88-acacb8b00000-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:27:59.882645 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.882607 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" podUID="bc5febcd-72c8-415a-888a-edf57777464c" containerName="main" containerID="cri-o://296dc13cadebbed52320ae5dafe032e8ae0deb647e2bafd70a59da90472c5aac" gracePeriod=45 Nov 02 18:27:59.897485 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:27:59.897443 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:28:00.129545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.129444 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:00.171862 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.171834 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:28:00.207160 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.207086 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww"] Nov 02 18:28:00.231574 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.230728 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b"] Nov 02 18:28:00.287754 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.287673 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-model-cache\") pod \"bc5febcd-72c8-415a-888a-edf57777464c\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " Nov 02 18:28:00.287903 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.287843 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-home\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-home\") pod \"bc5febcd-72c8-415a-888a-edf57777464c\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " Nov 02 18:28:00.287903 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.287874 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-tmp-dir\") pod \"bc5febcd-72c8-415a-888a-edf57777464c\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " Nov 02 18:28:00.288200 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.287975 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc5febcd-72c8-415a-888a-edf57777464c-kube-api-access-dvdzr\" (UniqueName: \"kubernetes.io/projected/bc5febcd-72c8-415a-888a-edf57777464c-kube-api-access-dvdzr\") pod \"bc5febcd-72c8-415a-888a-edf57777464c\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " Nov 02 18:28:00.288200 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.288035 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-model-cache" pod "bc5febcd-72c8-415a-888a-edf57777464c" (UID: "bc5febcd-72c8-415a-888a-edf57777464c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:00.288200 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.288193 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc5febcd-72c8-415a-888a-edf57777464c-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5febcd-72c8-415a-888a-edf57777464c-tls-certs\") pod \"bc5febcd-72c8-415a-888a-edf57777464c\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " Nov 02 18:28:00.288375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.288211 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-home" pod "bc5febcd-72c8-415a-888a-edf57777464c" (UID: "bc5febcd-72c8-415a-888a-edf57777464c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:00.288375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.288224 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-tmp-dir" pod "bc5febcd-72c8-415a-888a-edf57777464c" (UID: "bc5febcd-72c8-415a-888a-edf57777464c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:00.288375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.288230 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-dshm\") pod \"bc5febcd-72c8-415a-888a-edf57777464c\" (UID: \"bc5febcd-72c8-415a-888a-edf57777464c\") " Nov 02 18:28:00.288914 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.288892 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:00.289062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.288917 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:00.289062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.288930 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:00.290803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.290776 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc5febcd-72c8-415a-888a-edf57777464c-kube-api-access-dvdzr" pod "bc5febcd-72c8-415a-888a-edf57777464c" (UID: "bc5febcd-72c8-415a-888a-edf57777464c"). InnerVolumeSpecName "kube-api-access-dvdzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:00.290897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.290776 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-dshm" pod "bc5febcd-72c8-415a-888a-edf57777464c" (UID: "bc5febcd-72c8-415a-888a-edf57777464c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:00.290897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.290817 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc5febcd-72c8-415a-888a-edf57777464c-tls-certs" pod "bc5febcd-72c8-415a-888a-edf57777464c" (UID: "bc5febcd-72c8-415a-888a-edf57777464c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:28:00.330474 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.330440 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:00.390090 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.390049 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvdzr\" (UniqueName: \"kubernetes.io/projected/bc5febcd-72c8-415a-888a-edf57777464c-kube-api-access-dvdzr\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:00.390090 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.390087 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5febcd-72c8-415a-888a-edf57777464c-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:00.390326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.390102 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5febcd-72c8-415a-888a-edf57777464c-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:00.476629 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.475891 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:00.687494 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.687414 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" Nov 02 18:28:00.687655 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.687422 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz" event={"ID":"c9b89a1f-2e31-4335-9f88-acacb8b00000","Type":"ContainerDied","Data":"29970c00eec28221373824820509ab134e3fcc7fabb85930794f6aba7fc98975"} Nov 02 18:28:00.687655 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.687556 5549 scope.go:122] "RemoveContainer" containerID="b7b9b7a2f9f1af600e96042257949118f8c2bcfe7ad78863a30780f878323e2f" Nov 02 18:28:00.689270 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.689201 5549 generic.go:356] "Generic (PLEG): container finished" podID="bc5febcd-72c8-415a-888a-edf57777464c" containerID="296dc13cadebbed52320ae5dafe032e8ae0deb647e2bafd70a59da90472c5aac" exitCode=0 Nov 02 18:28:00.689270 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.689262 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" Nov 02 18:28:00.689480 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.689329 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" event={"ID":"bc5febcd-72c8-415a-888a-edf57777464c","Type":"ContainerDied","Data":"296dc13cadebbed52320ae5dafe032e8ae0deb647e2bafd70a59da90472c5aac"} Nov 02 18:28:00.689480 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.689389 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc" event={"ID":"bc5febcd-72c8-415a-888a-edf57777464c","Type":"ContainerDied","Data":"c82d95f15c10dfe900863553205b37dd545af29131e6b43511ff4ea984863d37"} Nov 02 18:28:00.699982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.699936 5549 scope.go:122] "RemoveContainer" containerID="296dc13cadebbed52320ae5dafe032e8ae0deb647e2bafd70a59da90472c5aac" Nov 02 18:28:00.710281 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.710257 5549 scope.go:122] "RemoveContainer" containerID="296dc13cadebbed52320ae5dafe032e8ae0deb647e2bafd70a59da90472c5aac" Nov 02 18:28:00.710545 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:28:00.710521 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296dc13cadebbed52320ae5dafe032e8ae0deb647e2bafd70a59da90472c5aac\": container with ID starting with 296dc13cadebbed52320ae5dafe032e8ae0deb647e2bafd70a59da90472c5aac not found: ID does not exist" containerID="296dc13cadebbed52320ae5dafe032e8ae0deb647e2bafd70a59da90472c5aac" Nov 02 18:28:00.710603 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.710552 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296dc13cadebbed52320ae5dafe032e8ae0deb647e2bafd70a59da90472c5aac"} err="failed to get container status \"296dc13cadebbed52320ae5dafe032e8ae0deb647e2bafd70a59da90472c5aac\": rpc error: code = NotFound desc = could not find container \"296dc13cadebbed52320ae5dafe032e8ae0deb647e2bafd70a59da90472c5aac\": container with ID starting with 296dc13cadebbed52320ae5dafe032e8ae0deb647e2bafd70a59da90472c5aac not found: ID does not exist" Nov 02 18:28:00.710770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.710752 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz"] Nov 02 18:28:00.716870 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.716826 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-z6bsz"] Nov 02 18:28:00.729099 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.729070 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc"] Nov 02 18:28:00.732937 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:00.732913 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cc0b41/test-model-cc0b41-kserve-677864c7b5-27rcc"] Nov 02 18:28:01.202062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.198468 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:01.227448 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.227411 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 18:28:01.246398 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.246342 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" podUID="f2b6e563-a46e-42e2-bc91-5a0277272b56" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:28:01.313374 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.313340 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-workload-certs\") pod \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " Nov 02 18:28:01.313552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.313409 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-token\" (UniqueName: \"kubernetes.io/projected/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-token\") pod \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " Nov 02 18:28:01.313552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.313463 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-podinfo\") pod \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " Nov 02 18:28:01.313665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.313562 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-envoy\") pod \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " Nov 02 18:28:01.313665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.313599 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-workload-socket\") pod \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " Nov 02 18:28:01.313665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.313624 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-credential-socket\") pod \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " Nov 02 18:28:01.313834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.313669 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-data\") pod \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " Nov 02 18:28:01.313834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.313669 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-workload-certs" pod "e56ccdbd-6737-4a84-9d80-59018a6b77ff" (UID: "e56ccdbd-6737-4a84-9d80-59018a6b77ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:01.313834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.313729 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e56ccdbd-6737-4a84-9d80-59018a6b77ff-kube-api-access-dsgb8\" (UniqueName: \"kubernetes.io/projected/e56ccdbd-6737-4a84-9d80-59018a6b77ff-kube-api-access-dsgb8\") pod \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " Nov 02 18:28:01.314055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.313861 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istiod-ca-cert\") pod \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\" (UID: \"e56ccdbd-6737-4a84-9d80-59018a6b77ff\") " Nov 02 18:28:01.314055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.313918 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-credential-socket" pod "e56ccdbd-6737-4a84-9d80-59018a6b77ff" (UID: "e56ccdbd-6737-4a84-9d80-59018a6b77ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:01.314055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.313988 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-data" pod "e56ccdbd-6737-4a84-9d80-59018a6b77ff" (UID: "e56ccdbd-6737-4a84-9d80-59018a6b77ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:01.314192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.314084 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-workload-socket" pod "e56ccdbd-6737-4a84-9d80-59018a6b77ff" (UID: "e56ccdbd-6737-4a84-9d80-59018a6b77ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:01.314345 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.314323 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istiod-ca-cert" pod "e56ccdbd-6737-4a84-9d80-59018a6b77ff" (UID: "e56ccdbd-6737-4a84-9d80-59018a6b77ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:01.314558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.314539 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:01.314601 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.314565 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:01.314601 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.314580 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:01.314601 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.314596 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:01.314703 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.314610 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:01.316018 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.315995 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-token" pod "e56ccdbd-6737-4a84-9d80-59018a6b77ff" (UID: "e56ccdbd-6737-4a84-9d80-59018a6b77ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:01.316288 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.316259 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-podinfo" pod "e56ccdbd-6737-4a84-9d80-59018a6b77ff" (UID: "e56ccdbd-6737-4a84-9d80-59018a6b77ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:28:01.316288 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.316273 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56ccdbd-6737-4a84-9d80-59018a6b77ff-kube-api-access-dsgb8" pod "e56ccdbd-6737-4a84-9d80-59018a6b77ff" (UID: "e56ccdbd-6737-4a84-9d80-59018a6b77ff"). InnerVolumeSpecName "kube-api-access-dsgb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:01.316432 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.316322 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-envoy" pod "e56ccdbd-6737-4a84-9d80-59018a6b77ff" (UID: "e56ccdbd-6737-4a84-9d80-59018a6b77ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:01.427976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.427287 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:01.427976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.427329 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:01.427976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.427345 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e56ccdbd-6737-4a84-9d80-59018a6b77ff-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:01.427976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.427359 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsgb8\" (UniqueName: \"kubernetes.io/projected/e56ccdbd-6737-4a84-9d80-59018a6b77ff-kube-api-access-dsgb8\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:01.591040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.590940 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:01.705975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.703082 5549 generic.go:356] "Generic (PLEG): container finished" podID="e56ccdbd-6737-4a84-9d80-59018a6b77ff" containerID="56a3e72ae6988fda9241dbdff978cbce31c14d0f3d38c950c58fb642dfb66e29" exitCode=0 Nov 02 18:28:01.705975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.703130 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" Nov 02 18:28:01.705975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.703262 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" event={"ID":"e56ccdbd-6737-4a84-9d80-59018a6b77ff","Type":"ContainerDied","Data":"56a3e72ae6988fda9241dbdff978cbce31c14d0f3d38c950c58fb642dfb66e29"} Nov 02 18:28:01.705975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.703305 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw" event={"ID":"e56ccdbd-6737-4a84-9d80-59018a6b77ff","Type":"ContainerDied","Data":"c316867731343119c1f44bbe0539bee32e9e9f2845f36cf228a935b85a20304d"} Nov 02 18:28:01.705975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.703331 5549 scope.go:122] "RemoveContainer" containerID="56a3e72ae6988fda9241dbdff978cbce31c14d0f3d38c950c58fb642dfb66e29" Nov 02 18:28:01.754102 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.754066 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw"] Nov 02 18:28:01.759967 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.759906 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-r67jw"] Nov 02 18:28:01.764364 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.764335 5549 scope.go:122] "RemoveContainer" containerID="56a3e72ae6988fda9241dbdff978cbce31c14d0f3d38c950c58fb642dfb66e29" Nov 02 18:28:01.764594 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:28:01.764570 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56a3e72ae6988fda9241dbdff978cbce31c14d0f3d38c950c58fb642dfb66e29\": container with ID starting with 56a3e72ae6988fda9241dbdff978cbce31c14d0f3d38c950c58fb642dfb66e29 not found: ID does not exist" containerID="56a3e72ae6988fda9241dbdff978cbce31c14d0f3d38c950c58fb642dfb66e29" Nov 02 18:28:01.764674 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:01.764597 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56a3e72ae6988fda9241dbdff978cbce31c14d0f3d38c950c58fb642dfb66e29"} err="failed to get container status \"56a3e72ae6988fda9241dbdff978cbce31c14d0f3d38c950c58fb642dfb66e29\": rpc error: code = NotFound desc = could not find container \"56a3e72ae6988fda9241dbdff978cbce31c14d0f3d38c950c58fb642dfb66e29\": container with ID starting with 56a3e72ae6988fda9241dbdff978cbce31c14d0f3d38c950c58fb642dfb66e29 not found: ID does not exist" Nov 02 18:28:02.042847 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:02.042810 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:02.323778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:02.323688 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc5febcd-72c8-415a-888a-edf57777464c" path="/var/lib/kubelet/pods/bc5febcd-72c8-415a-888a-edf57777464c/volumes" Nov 02 18:28:02.324398 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:02.324374 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9b89a1f-2e31-4335-9f88-acacb8b00000" path="/var/lib/kubelet/pods/c9b89a1f-2e31-4335-9f88-acacb8b00000/volumes" Nov 02 18:28:02.324926 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:02.324906 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e56ccdbd-6737-4a84-9d80-59018a6b77ff" path="/var/lib/kubelet/pods/e56ccdbd-6737-4a84-9d80-59018a6b77ff/volumes" Nov 02 18:28:02.663692 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:02.663566 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:02.880710 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:02.880675 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:03.409699 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:03.409656 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k"] Nov 02 18:28:03.410027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:03.409991 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" podUID="216112c3-41c7-45f6-9553-628fa92c38cb" containerName="maas-api" containerID="cri-o://6ee759cc3da469533c1d78af08aa050398e4e010bc98d350d56afbd46d633dc5" gracePeriod=30 Nov 02 18:28:03.497808 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:03.497770 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:03.921589 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:03.921558 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" Nov 02 18:28:03.964336 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:03.964294 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:04.007266 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.007232 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/216112c3-41c7-45f6-9553-628fa92c38cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/216112c3-41c7-45f6-9553-628fa92c38cb-maas-api-tls\") pod \"216112c3-41c7-45f6-9553-628fa92c38cb\" (UID: \"216112c3-41c7-45f6-9553-628fa92c38cb\") " Nov 02 18:28:04.007445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.007279 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/216112c3-41c7-45f6-9553-628fa92c38cb-kube-api-access-x2ghz\" (UniqueName: \"kubernetes.io/projected/216112c3-41c7-45f6-9553-628fa92c38cb-kube-api-access-x2ghz\") pod \"216112c3-41c7-45f6-9553-628fa92c38cb\" (UID: \"216112c3-41c7-45f6-9553-628fa92c38cb\") " Nov 02 18:28:04.007445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.007393 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/216112c3-41c7-45f6-9553-628fa92c38cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/216112c3-41c7-45f6-9553-628fa92c38cb-openshift-service-ca\") pod \"216112c3-41c7-45f6-9553-628fa92c38cb\" (UID: \"216112c3-41c7-45f6-9553-628fa92c38cb\") " Nov 02 18:28:04.007445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.007436 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/216112c3-41c7-45f6-9553-628fa92c38cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/216112c3-41c7-45f6-9553-628fa92c38cb-metrics-certs\") pod \"216112c3-41c7-45f6-9553-628fa92c38cb\" (UID: \"216112c3-41c7-45f6-9553-628fa92c38cb\") " Nov 02 18:28:04.007820 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.007795 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216112c3-41c7-45f6-9553-628fa92c38cb-openshift-service-ca" pod "216112c3-41c7-45f6-9553-628fa92c38cb" (UID: "216112c3-41c7-45f6-9553-628fa92c38cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:04.008120 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.008100 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/216112c3-41c7-45f6-9553-628fa92c38cb-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:04.009539 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.009515 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/216112c3-41c7-45f6-9553-628fa92c38cb-maas-api-tls" pod "216112c3-41c7-45f6-9553-628fa92c38cb" (UID: "216112c3-41c7-45f6-9553-628fa92c38cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:28:04.009640 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.009534 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/216112c3-41c7-45f6-9553-628fa92c38cb-metrics-certs" pod "216112c3-41c7-45f6-9553-628fa92c38cb" (UID: "216112c3-41c7-45f6-9553-628fa92c38cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:28:04.009640 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.009523 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216112c3-41c7-45f6-9553-628fa92c38cb-kube-api-access-x2ghz" pod "216112c3-41c7-45f6-9553-628fa92c38cb" (UID: "216112c3-41c7-45f6-9553-628fa92c38cb"). InnerVolumeSpecName "kube-api-access-x2ghz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:04.110000 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.109964 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/216112c3-41c7-45f6-9553-628fa92c38cb-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:04.110000 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.109998 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/216112c3-41c7-45f6-9553-628fa92c38cb-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:04.110223 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.110010 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2ghz\" (UniqueName: \"kubernetes.io/projected/216112c3-41c7-45f6-9553-628fa92c38cb-kube-api-access-x2ghz\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:04.640624 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.640509 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:04.740552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.740516 5549 generic.go:356] "Generic (PLEG): container finished" podID="216112c3-41c7-45f6-9553-628fa92c38cb" containerID="6ee759cc3da469533c1d78af08aa050398e4e010bc98d350d56afbd46d633dc5" exitCode=0 Nov 02 18:28:04.740740 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.740565 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" Nov 02 18:28:04.740740 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.740605 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" event={"ID":"216112c3-41c7-45f6-9553-628fa92c38cb","Type":"ContainerDied","Data":"6ee759cc3da469533c1d78af08aa050398e4e010bc98d350d56afbd46d633dc5"} Nov 02 18:28:04.740740 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.740664 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k" event={"ID":"216112c3-41c7-45f6-9553-628fa92c38cb","Type":"ContainerDied","Data":"181d7b0f68826fe2d4165fe8188aa7b8a9e72c156b0af689818db98059c13f66"} Nov 02 18:28:04.740740 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.740691 5549 scope.go:122] "RemoveContainer" containerID="6ee759cc3da469533c1d78af08aa050398e4e010bc98d350d56afbd46d633dc5" Nov 02 18:28:04.752529 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.752501 5549 scope.go:122] "RemoveContainer" containerID="6ee759cc3da469533c1d78af08aa050398e4e010bc98d350d56afbd46d633dc5" Nov 02 18:28:04.752832 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:28:04.752806 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee759cc3da469533c1d78af08aa050398e4e010bc98d350d56afbd46d633dc5\": container with ID starting with 6ee759cc3da469533c1d78af08aa050398e4e010bc98d350d56afbd46d633dc5 not found: ID does not exist" containerID="6ee759cc3da469533c1d78af08aa050398e4e010bc98d350d56afbd46d633dc5" Nov 02 18:28:04.752923 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.752842 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee759cc3da469533c1d78af08aa050398e4e010bc98d350d56afbd46d633dc5"} err="failed to get container status \"6ee759cc3da469533c1d78af08aa050398e4e010bc98d350d56afbd46d633dc5\": rpc error: code = NotFound desc = could not find container \"6ee759cc3da469533c1d78af08aa050398e4e010bc98d350d56afbd46d633dc5\": container with ID starting with 6ee759cc3da469533c1d78af08aa050398e4e010bc98d350d56afbd46d633dc5 not found: ID does not exist" Nov 02 18:28:04.761597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.761569 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k"] Nov 02 18:28:04.767057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:04.767030 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2966e7-7579cc967f-h8r7k"] Nov 02 18:28:05.208034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:05.207934 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" podUID="078222c5-8861-4bb3-9969-5aca397e9195" containerName="payload-processing" containerID="cri-o://ec208a09f5b6d8644a1a2bcaf1a63fddd42b8d7f0d32833f5eaf7a2bd81c8ce5" gracePeriod=25 Nov 02 18:28:05.231327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:05.231278 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" podUID="52c33649-7da9-4772-817d-d924c1013123" containerName="payload-pre-processing" containerID="cri-o://52ba1de1ba738949b873f2471d44302a45f5b8df42d1a5f619cde9d1a1cd7870" gracePeriod=25 Nov 02 18:28:05.318040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:05.318000 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:05.777184 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:05.777148 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:06.322518 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:06.322480 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="216112c3-41c7-45f6-9553-628fa92c38cb" path="/var/lib/kubelet/pods/216112c3-41c7-45f6-9553-628fa92c38cb/volumes" Nov 02 18:28:09.853677 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:09.853638 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:10.109240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:10.109138 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:10.806107 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:10.806060 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:11.246096 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:11.246057 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" podUID="f2b6e563-a46e-42e2-bc91-5a0277272b56" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:28:11.419117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:11.419079 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:11.538490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:11.538391 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" podUID="52c33649-7da9-4772-817d-d924c1013123" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:28:11.538490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:11.538438 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" podUID="078222c5-8861-4bb3-9969-5aca397e9195" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:28:12.130748 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:12.130691 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:12.746091 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:12.746051 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb"] Nov 02 18:28:12.762045 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:12.762006 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl"] Nov 02 18:28:12.867185 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:12.867139 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:13.036128 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:13.031404 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:13.820180 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:13.817252 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:13.952094 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:13.952063 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:14.528495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:14.528455 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:15.236444 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:15.236402 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:15.811193 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:15.811154 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:16.595015 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:16.594979 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:16.859921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:16.859839 5549 generic.go:356] "Generic (PLEG): container finished" podID="f2b6e563-a46e-42e2-bc91-5a0277272b56" containerID="8d86ac4a6797893de697669125fbd7b745dd11e963b40cefff833832b52f4d14" exitCode=0 Nov 02 18:28:16.860088 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:16.859917 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" event={"ID":"f2b6e563-a46e-42e2-bc91-5a0277272b56","Type":"ContainerDied","Data":"8d86ac4a6797893de697669125fbd7b745dd11e963b40cefff833832b52f4d14"} Nov 02 18:28:16.940510 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:16.940483 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" Nov 02 18:28:17.045547 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:17.045507 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b6e563-a46e-42e2-bc91-5a0277272b56-kube-api-access-6flmd\" (UniqueName: \"kubernetes.io/projected/f2b6e563-a46e-42e2-bc91-5a0277272b56-kube-api-access-6flmd\") pod \"f2b6e563-a46e-42e2-bc91-5a0277272b56\" (UID: \"f2b6e563-a46e-42e2-bc91-5a0277272b56\") " Nov 02 18:28:17.045750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:17.045691 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b6e563-a46e-42e2-bc91-5a0277272b56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b6e563-a46e-42e2-bc91-5a0277272b56-plugins-config-volume\") pod \"f2b6e563-a46e-42e2-bc91-5a0277272b56\" (UID: \"f2b6e563-a46e-42e2-bc91-5a0277272b56\") " Nov 02 18:28:17.046072 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:17.046048 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b6e563-a46e-42e2-bc91-5a0277272b56-plugins-config-volume" pod "f2b6e563-a46e-42e2-bc91-5a0277272b56" (UID: "f2b6e563-a46e-42e2-bc91-5a0277272b56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:17.046206 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:17.046190 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b6e563-a46e-42e2-bc91-5a0277272b56-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:17.047800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:17.047776 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b6e563-a46e-42e2-bc91-5a0277272b56-kube-api-access-6flmd" pod "f2b6e563-a46e-42e2-bc91-5a0277272b56" (UID: "f2b6e563-a46e-42e2-bc91-5a0277272b56"). InnerVolumeSpecName "kube-api-access-6flmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:17.147648 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:17.147553 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6flmd\" (UniqueName: \"kubernetes.io/projected/f2b6e563-a46e-42e2-bc91-5a0277272b56-kube-api-access-6flmd\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:17.230396 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:17.230357 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:17.592352 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:28:17.592309 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:28:17.746690 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:17.746641 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb" podUID="ff3423b0-fb1a-4082-8227-bd154ed372d7" containerName="payload-processing" containerID="cri-o://2c48b63b9dcff7b42388b9cca89f4d46bf85d5c8f98a27b0ad6269f75e873faa" gracePeriod=25 Nov 02 18:28:17.763012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:17.762972 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" podUID="5b44de0c-f730-47cd-80e0-4f6b88f5b1fc" containerName="payload-pre-processing" containerID="cri-o://68d3c4ec5094b9c8b1a25acc1dbe99052485c54143fb006ab7218b596d0a7ca6" gracePeriod=25 Nov 02 18:28:17.869666 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:17.869630 5549 generic.go:356] "Generic (PLEG): container finished" podID="ff3423b0-fb1a-4082-8227-bd154ed372d7" containerID="2c48b63b9dcff7b42388b9cca89f4d46bf85d5c8f98a27b0ad6269f75e873faa" exitCode=0 Nov 02 18:28:17.869809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:17.869701 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb" event={"ID":"ff3423b0-fb1a-4082-8227-bd154ed372d7","Type":"ContainerDied","Data":"2c48b63b9dcff7b42388b9cca89f4d46bf85d5c8f98a27b0ad6269f75e873faa"} Nov 02 18:28:17.871489 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:17.871470 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" Nov 02 18:28:17.871628 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:17.871512 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd" event={"ID":"f2b6e563-a46e-42e2-bc91-5a0277272b56","Type":"ContainerDied","Data":"39730ba6e585f13b26eba42e3e419111389329161d2852de7d0893b823a78c1a"} Nov 02 18:28:17.871628 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:17.871572 5549 scope.go:122] "RemoveContainer" containerID="8d86ac4a6797893de697669125fbd7b745dd11e963b40cefff833832b52f4d14" Nov 02 18:28:17.896451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:17.896372 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:17.936879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:17.936824 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd"] Nov 02 18:28:17.940715 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:17.940686 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2966e7-c4477dcd7-jq4qd"] Nov 02 18:28:18.056479 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:18.056449 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb" Nov 02 18:28:18.162743 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:18.162646 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff3423b0-fb1a-4082-8227-bd154ed372d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3423b0-fb1a-4082-8227-bd154ed372d7-plugins-config-volume\") pod \"ff3423b0-fb1a-4082-8227-bd154ed372d7\" (UID: \"ff3423b0-fb1a-4082-8227-bd154ed372d7\") " Nov 02 18:28:18.162936 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:18.162844 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3423b0-fb1a-4082-8227-bd154ed372d7-kube-api-access-9kfl5\" (UniqueName: \"kubernetes.io/projected/ff3423b0-fb1a-4082-8227-bd154ed372d7-kube-api-access-9kfl5\") pod \"ff3423b0-fb1a-4082-8227-bd154ed372d7\" (UID: \"ff3423b0-fb1a-4082-8227-bd154ed372d7\") " Nov 02 18:28:18.163058 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:18.163020 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff3423b0-fb1a-4082-8227-bd154ed372d7-plugins-config-volume" pod "ff3423b0-fb1a-4082-8227-bd154ed372d7" (UID: "ff3423b0-fb1a-4082-8227-bd154ed372d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:18.163430 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:18.163407 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3423b0-fb1a-4082-8227-bd154ed372d7-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:18.164965 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:18.164933 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3423b0-fb1a-4082-8227-bd154ed372d7-kube-api-access-9kfl5" pod "ff3423b0-fb1a-4082-8227-bd154ed372d7" (UID: "ff3423b0-fb1a-4082-8227-bd154ed372d7"). InnerVolumeSpecName "kube-api-access-9kfl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:18.264523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:18.264484 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kfl5\" (UniqueName: \"kubernetes.io/projected/ff3423b0-fb1a-4082-8227-bd154ed372d7-kube-api-access-9kfl5\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:18.328289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:18.328248 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b6e563-a46e-42e2-bc91-5a0277272b56" path="/var/lib/kubelet/pods/f2b6e563-a46e-42e2-bc91-5a0277272b56/volumes" Nov 02 18:28:18.410116 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:18.410081 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:18.766015 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:18.760361 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:18.882765 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:18.882731 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb" Nov 02 18:28:18.883006 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:18.882733 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb" event={"ID":"ff3423b0-fb1a-4082-8227-bd154ed372d7","Type":"ContainerDied","Data":"901ea22623f8600608414d0324595c93b6caf05c8438cef9a250d7ab69d38d38"} Nov 02 18:28:18.883006 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:18.882861 5549 scope.go:122] "RemoveContainer" containerID="2c48b63b9dcff7b42388b9cca89f4d46bf85d5c8f98a27b0ad6269f75e873faa" Nov 02 18:28:18.905827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:18.905794 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb"] Nov 02 18:28:18.912289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:18.912261 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cc0b41-79dd8fbb7c-gfrgb"] Nov 02 18:28:19.688767 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:19.685569 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:20.140255 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:20.140218 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:20.322299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:20.322265 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff3423b0-fb1a-4082-8227-bd154ed372d7" path="/var/lib/kubelet/pods/ff3423b0-fb1a-4082-8227-bd154ed372d7/volumes" Nov 02 18:28:20.394588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:20.394494 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:20.994026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:20.993972 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:21.538383 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:21.538345 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" podUID="52c33649-7da9-4772-817d-d924c1013123" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:28:21.538781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:21.538345 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" podUID="078222c5-8861-4bb3-9969-5aca397e9195" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:28:21.628699 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:21.628617 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:22.177342 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:22.177309 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:22.314166 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:22.314116 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:22.905802 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:22.905768 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:23.752227 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:23.752181 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:24.067165 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:24.067038 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:24.222315 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:24.222280 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:24.382563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:24.382489 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:24.595520 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:24.595475 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:24.838209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:24.838168 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh"] Nov 02 18:28:24.838448 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:24.838420 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" podUID="d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" containerName="istio-proxy" containerID="cri-o://a33d40a5260c77ad0c8ff115e6beeb25f6faec27c35092cb0509ac125985e055" gracePeriod=30 Nov 02 18:28:24.908513 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:24.908451 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:25.414116 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:25.400238 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:25.554473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:25.554438 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" podUID="5b44de0c-f730-47cd-80e0-4f6b88f5b1fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:28:25.597288 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:25.597260 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" Nov 02 18:28:25.600804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:25.600778 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" Nov 02 18:28:25.722969 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:25.722929 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c33649-7da9-4772-817d-d924c1013123-kube-api-access-58xw6\" (UniqueName: \"kubernetes.io/projected/52c33649-7da9-4772-817d-d924c1013123-kube-api-access-58xw6\") pod \"52c33649-7da9-4772-817d-d924c1013123\" (UID: \"52c33649-7da9-4772-817d-d924c1013123\") " Nov 02 18:28:25.723155 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:25.723122 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52c33649-7da9-4772-817d-d924c1013123-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c33649-7da9-4772-817d-d924c1013123-plugins-config-volume\") pod \"52c33649-7da9-4772-817d-d924c1013123\" (UID: \"52c33649-7da9-4772-817d-d924c1013123\") " Nov 02 18:28:25.723218 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:25.723157 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/078222c5-8861-4bb3-9969-5aca397e9195-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078222c5-8861-4bb3-9969-5aca397e9195-plugins-config-volume\") pod \"078222c5-8861-4bb3-9969-5aca397e9195\" (UID: \"078222c5-8861-4bb3-9969-5aca397e9195\") " Nov 02 18:28:25.723218 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:25.723191 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/078222c5-8861-4bb3-9969-5aca397e9195-kube-api-access-k76xr\" (UniqueName: \"kubernetes.io/projected/078222c5-8861-4bb3-9969-5aca397e9195-kube-api-access-k76xr\") pod \"078222c5-8861-4bb3-9969-5aca397e9195\" (UID: \"078222c5-8861-4bb3-9969-5aca397e9195\") " Nov 02 18:28:25.723551 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:25.723523 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078222c5-8861-4bb3-9969-5aca397e9195-plugins-config-volume" pod "078222c5-8861-4bb3-9969-5aca397e9195" (UID: "078222c5-8861-4bb3-9969-5aca397e9195"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:25.723641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:25.723565 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c33649-7da9-4772-817d-d924c1013123-plugins-config-volume" pod "52c33649-7da9-4772-817d-d924c1013123" (UID: "52c33649-7da9-4772-817d-d924c1013123"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:25.723789 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:25.723772 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c33649-7da9-4772-817d-d924c1013123-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:25.723829 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:25.723791 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078222c5-8861-4bb3-9969-5aca397e9195-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:25.725174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:25.725150 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c33649-7da9-4772-817d-d924c1013123-kube-api-access-58xw6" pod "52c33649-7da9-4772-817d-d924c1013123" (UID: "52c33649-7da9-4772-817d-d924c1013123"). InnerVolumeSpecName "kube-api-access-58xw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:25.725290 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:25.725270 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078222c5-8861-4bb3-9969-5aca397e9195-kube-api-access-k76xr" pod "078222c5-8861-4bb3-9969-5aca397e9195" (UID: "078222c5-8861-4bb3-9969-5aca397e9195"). InnerVolumeSpecName "kube-api-access-k76xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:25.767719 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:25.767640 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:25.824706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:25.824663 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k76xr\" (UniqueName: \"kubernetes.io/projected/078222c5-8861-4bb3-9969-5aca397e9195-kube-api-access-k76xr\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:25.824706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:25.824701 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58xw6\" (UniqueName: \"kubernetes.io/projected/52c33649-7da9-4772-817d-d924c1013123-kube-api-access-58xw6\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:26.037522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.037430 5549 generic.go:356] "Generic (PLEG): container finished" podID="078222c5-8861-4bb3-9969-5aca397e9195" containerID="ec208a09f5b6d8644a1a2bcaf1a63fddd42b8d7f0d32833f5eaf7a2bd81c8ce5" exitCode=0 Nov 02 18:28:26.037522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.037482 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" Nov 02 18:28:26.037522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.037509 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" event={"ID":"078222c5-8861-4bb3-9969-5aca397e9195","Type":"ContainerDied","Data":"ec208a09f5b6d8644a1a2bcaf1a63fddd42b8d7f0d32833f5eaf7a2bd81c8ce5"} Nov 02 18:28:26.037833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.037588 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww" event={"ID":"078222c5-8861-4bb3-9969-5aca397e9195","Type":"ContainerDied","Data":"12ce20a61cb9c475952a2539c4e1dce0bc5f8fd57ede5ef260276e1ecfadf171"} Nov 02 18:28:26.037833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.037611 5549 scope.go:122] "RemoveContainer" containerID="ec208a09f5b6d8644a1a2bcaf1a63fddd42b8d7f0d32833f5eaf7a2bd81c8ce5" Nov 02 18:28:26.039333 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.039312 5549 generic.go:356] "Generic (PLEG): container finished" podID="52c33649-7da9-4772-817d-d924c1013123" containerID="52ba1de1ba738949b873f2471d44302a45f5b8df42d1a5f619cde9d1a1cd7870" exitCode=0 Nov 02 18:28:26.039448 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.039362 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" Nov 02 18:28:26.039495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.039367 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" event={"ID":"52c33649-7da9-4772-817d-d924c1013123","Type":"ContainerDied","Data":"52ba1de1ba738949b873f2471d44302a45f5b8df42d1a5f619cde9d1a1cd7870"} Nov 02 18:28:26.039552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.039489 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b" event={"ID":"52c33649-7da9-4772-817d-d924c1013123","Type":"ContainerDied","Data":"800c2e7fc21a0d5e320a19bd7a24451a839407ddd89e1062f83298606428899f"} Nov 02 18:28:26.051141 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.051119 5549 scope.go:122] "RemoveContainer" containerID="ec208a09f5b6d8644a1a2bcaf1a63fddd42b8d7f0d32833f5eaf7a2bd81c8ce5" Nov 02 18:28:26.051425 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:28:26.051404 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec208a09f5b6d8644a1a2bcaf1a63fddd42b8d7f0d32833f5eaf7a2bd81c8ce5\": container with ID starting with ec208a09f5b6d8644a1a2bcaf1a63fddd42b8d7f0d32833f5eaf7a2bd81c8ce5 not found: ID does not exist" containerID="ec208a09f5b6d8644a1a2bcaf1a63fddd42b8d7f0d32833f5eaf7a2bd81c8ce5" Nov 02 18:28:26.051496 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.051431 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec208a09f5b6d8644a1a2bcaf1a63fddd42b8d7f0d32833f5eaf7a2bd81c8ce5"} err="failed to get container status \"ec208a09f5b6d8644a1a2bcaf1a63fddd42b8d7f0d32833f5eaf7a2bd81c8ce5\": rpc error: code = NotFound desc = could not find container \"ec208a09f5b6d8644a1a2bcaf1a63fddd42b8d7f0d32833f5eaf7a2bd81c8ce5\": container with ID starting with ec208a09f5b6d8644a1a2bcaf1a63fddd42b8d7f0d32833f5eaf7a2bd81c8ce5 not found: ID does not exist" Nov 02 18:28:26.051496 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.051448 5549 scope.go:122] "RemoveContainer" containerID="52ba1de1ba738949b873f2471d44302a45f5b8df42d1a5f619cde9d1a1cd7870" Nov 02 18:28:26.079645 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.074770 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:26.100683 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.090284 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww"] Nov 02 18:28:26.104049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.103025 5549 scope.go:122] "RemoveContainer" containerID="52ba1de1ba738949b873f2471d44302a45f5b8df42d1a5f619cde9d1a1cd7870" Nov 02 18:28:26.104049 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:28:26.103588 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52ba1de1ba738949b873f2471d44302a45f5b8df42d1a5f619cde9d1a1cd7870\": container with ID starting with 52ba1de1ba738949b873f2471d44302a45f5b8df42d1a5f619cde9d1a1cd7870 not found: ID does not exist" containerID="52ba1de1ba738949b873f2471d44302a45f5b8df42d1a5f619cde9d1a1cd7870" Nov 02 18:28:26.104049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.103634 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ba1de1ba738949b873f2471d44302a45f5b8df42d1a5f619cde9d1a1cd7870"} err="failed to get container status \"52ba1de1ba738949b873f2471d44302a45f5b8df42d1a5f619cde9d1a1cd7870\": rpc error: code = NotFound desc = could not find container \"52ba1de1ba738949b873f2471d44302a45f5b8df42d1a5f619cde9d1a1cd7870\": container with ID starting with 52ba1de1ba738949b873f2471d44302a45f5b8df42d1a5f619cde9d1a1cd7870 not found: ID does not exist" Nov 02 18:28:26.120012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.110367 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-807c08-56f5765679-h2lww"] Nov 02 18:28:26.120012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.117613 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b"] Nov 02 18:28:26.127090 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.127058 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-807c08-64d48fd4f9-xh75b"] Nov 02 18:28:26.322558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.322483 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="078222c5-8861-4bb3-9969-5aca397e9195" path="/var/lib/kubelet/pods/078222c5-8861-4bb3-9969-5aca397e9195/volumes" Nov 02 18:28:26.322976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.322961 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52c33649-7da9-4772-817d-d924c1013123" path="/var/lib/kubelet/pods/52c33649-7da9-4772-817d-d924c1013123/volumes" Nov 02 18:28:26.661404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.661313 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:26.829001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:26.824689 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:27.614297 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:27.613115 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:28.345349 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:28.345263 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:29.011755 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:29.011692 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:29.622036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:29.621995 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:30.085339 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.084768 5549 generic.go:356] "Generic (PLEG): container finished" podID="d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" containerID="a33d40a5260c77ad0c8ff115e6beeb25f6faec27c35092cb0509ac125985e055" exitCode=0 Nov 02 18:28:30.085339 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.085103 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" event={"ID":"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23","Type":"ContainerDied","Data":"a33d40a5260c77ad0c8ff115e6beeb25f6faec27c35092cb0509ac125985e055"} Nov 02 18:28:30.085339 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.085204 5549 scope.go:122] "RemoveContainer" containerID="f4829e725c79e563c30f2e2294bd3d0ffc65d6cdb1e34e4bebd3c1da6cf10254" Nov 02 18:28:30.126446 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.126426 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:28:30.200813 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.200775 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-podinfo\") pod \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " Nov 02 18:28:30.200997 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.200867 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-credential-socket\") pod \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " Nov 02 18:28:30.201049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.200998 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-data\") pod \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " Nov 02 18:28:30.201049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.201032 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-workload-certs\") pod \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " Nov 02 18:28:30.201163 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.201151 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-kube-api-access-mp4xv\" (UniqueName: \"kubernetes.io/projected/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-kube-api-access-mp4xv\") pod \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " Nov 02 18:28:30.201243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.201176 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-envoy\") pod \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " Nov 02 18:28:30.201243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.201183 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-credential-socket" pod "d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" (UID: "d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:30.201362 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.201219 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-token\" (UniqueName: \"kubernetes.io/projected/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-token\") pod \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " Nov 02 18:28:30.201362 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.201269 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-data" pod "d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" (UID: "d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:30.201362 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.201320 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-workload-socket\") pod \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " Nov 02 18:28:30.201575 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.201370 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istiod-ca-cert\") pod \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\" (UID: \"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23\") " Nov 02 18:28:30.201575 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.201374 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-workload-certs" pod "d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" (UID: "d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:30.201708 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.201588 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-workload-socket" pod "d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" (UID: "d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:30.201809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.201789 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istiod-ca-cert" pod "d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" (UID: "d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:30.202189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.202169 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:30.202277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.202192 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:30.202277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.202207 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:30.202277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.202220 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:30.202277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.202232 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:30.203174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.203144 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-podinfo" pod "d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" (UID: "d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:28:30.203393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.203366 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-kube-api-access-mp4xv" pod "d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" (UID: "d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23"). InnerVolumeSpecName "kube-api-access-mp4xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:30.203763 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.203747 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-token" pod "d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" (UID: "d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:30.203811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.203757 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-envoy" pod "d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" (UID: "d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:30.304114 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.304075 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp4xv\" (UniqueName: \"kubernetes.io/projected/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-kube-api-access-mp4xv\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:30.304114 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.304106 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:30.304114 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.304116 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:30.304358 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.304126 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:30.354037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.349938 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:30.781429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.781385 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726"] Nov 02 18:28:30.797056 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.797022 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z"] Nov 02 18:28:30.887359 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:30.887313 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:31.021356 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:31.021323 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:31.094974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:31.094867 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" Nov 02 18:28:31.094974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:31.094914 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh" event={"ID":"d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23","Type":"ContainerDied","Data":"7dd145f8c0c5150416d72679e898ac57fa0b56f64e29f96f2bb0d50aa80d4eb2"} Nov 02 18:28:31.095219 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:31.095011 5549 scope.go:122] "RemoveContainer" containerID="a33d40a5260c77ad0c8ff115e6beeb25f6faec27c35092cb0509ac125985e055" Nov 02 18:28:31.157468 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:31.157391 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:31.169486 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:31.169416 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh"] Nov 02 18:28:31.176738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:31.176698 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-807c08-openshift-default-56bd9457-mp8xh"] Nov 02 18:28:31.925621 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:31.925578 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:32.211326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:32.211288 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:32.322063 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:32.322017 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" path="/var/lib/kubelet/pods/d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23/volumes" Nov 02 18:28:32.591521 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:32.591425 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:33.192186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:33.192152 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:33.815145 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:33.815108 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:34.535402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:34.535370 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:35.206817 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:35.206775 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:35.554361 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:35.554280 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" podUID="5b44de0c-f730-47cd-80e0-4f6b88f5b1fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:28:35.753332 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:35.753293 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:35.782376 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:35.782333 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726" podUID="bbffe25b-3628-4d4a-a7a8-af2eb246aaba" containerName="payload-processing" containerID="cri-o://f07c81a75f3d62189fc5639d54c901615ddab87c4bf91f2bb6b4ad2807336e0a" gracePeriod=25 Nov 02 18:28:35.798221 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:35.798185 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" podUID="afdfbd7a-1637-47c2-b84e-abfb6319483a" containerName="payload-pre-processing" containerID="cri-o://108926dafb2221310167071da2e527eb7c2a3da5e04508c301b365f808f802a3" gracePeriod=25 Nov 02 18:28:35.896298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:35.896220 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:36.028299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.028257 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:36.108039 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.108010 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726" Nov 02 18:28:36.137678 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.137642 5549 generic.go:356] "Generic (PLEG): container finished" podID="bbffe25b-3628-4d4a-a7a8-af2eb246aaba" containerID="f07c81a75f3d62189fc5639d54c901615ddab87c4bf91f2bb6b4ad2807336e0a" exitCode=0 Nov 02 18:28:36.137841 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.137700 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726" Nov 02 18:28:36.137841 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.137726 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726" event={"ID":"bbffe25b-3628-4d4a-a7a8-af2eb246aaba","Type":"ContainerDied","Data":"f07c81a75f3d62189fc5639d54c901615ddab87c4bf91f2bb6b4ad2807336e0a"} Nov 02 18:28:36.137841 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.137812 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726" event={"ID":"bbffe25b-3628-4d4a-a7a8-af2eb246aaba","Type":"ContainerDied","Data":"c4072f4dd453dff1eb8bde0b680c1a4088a88231768c5ea488fabc1e613e9371"} Nov 02 18:28:36.137841 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.137838 5549 scope.go:122] "RemoveContainer" containerID="f07c81a75f3d62189fc5639d54c901615ddab87c4bf91f2bb6b4ad2807336e0a" Nov 02 18:28:36.152321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.152289 5549 scope.go:122] "RemoveContainer" containerID="f07c81a75f3d62189fc5639d54c901615ddab87c4bf91f2bb6b4ad2807336e0a" Nov 02 18:28:36.152682 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:28:36.152654 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f07c81a75f3d62189fc5639d54c901615ddab87c4bf91f2bb6b4ad2807336e0a\": container with ID starting with f07c81a75f3d62189fc5639d54c901615ddab87c4bf91f2bb6b4ad2807336e0a not found: ID does not exist" containerID="f07c81a75f3d62189fc5639d54c901615ddab87c4bf91f2bb6b4ad2807336e0a" Nov 02 18:28:36.152774 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.152696 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f07c81a75f3d62189fc5639d54c901615ddab87c4bf91f2bb6b4ad2807336e0a"} err="failed to get container status \"f07c81a75f3d62189fc5639d54c901615ddab87c4bf91f2bb6b4ad2807336e0a\": rpc error: code = NotFound desc = could not find container \"f07c81a75f3d62189fc5639d54c901615ddab87c4bf91f2bb6b4ad2807336e0a\": container with ID starting with f07c81a75f3d62189fc5639d54c901615ddab87c4bf91f2bb6b4ad2807336e0a not found: ID does not exist" Nov 02 18:28:36.235981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.235145 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbffe25b-3628-4d4a-a7a8-af2eb246aaba-kube-api-access-7kshg\" (UniqueName: \"kubernetes.io/projected/bbffe25b-3628-4d4a-a7a8-af2eb246aaba-kube-api-access-7kshg\") pod \"bbffe25b-3628-4d4a-a7a8-af2eb246aaba\" (UID: \"bbffe25b-3628-4d4a-a7a8-af2eb246aaba\") " Nov 02 18:28:36.235981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.235468 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbffe25b-3628-4d4a-a7a8-af2eb246aaba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbffe25b-3628-4d4a-a7a8-af2eb246aaba-plugins-config-volume\") pod \"bbffe25b-3628-4d4a-a7a8-af2eb246aaba\" (UID: \"bbffe25b-3628-4d4a-a7a8-af2eb246aaba\") " Nov 02 18:28:36.238111 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.238047 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbffe25b-3628-4d4a-a7a8-af2eb246aaba-plugins-config-volume" pod "bbffe25b-3628-4d4a-a7a8-af2eb246aaba" (UID: "bbffe25b-3628-4d4a-a7a8-af2eb246aaba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:36.238416 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.238395 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbffe25b-3628-4d4a-a7a8-af2eb246aaba-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:36.239990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.239689 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:36.239990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.239869 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbffe25b-3628-4d4a-a7a8-af2eb246aaba-kube-api-access-7kshg" pod "bbffe25b-3628-4d4a-a7a8-af2eb246aaba" (UID: "bbffe25b-3628-4d4a-a7a8-af2eb246aaba"). InnerVolumeSpecName "kube-api-access-7kshg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:36.340451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.340417 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kshg\" (UniqueName: \"kubernetes.io/projected/bbffe25b-3628-4d4a-a7a8-af2eb246aaba-kube-api-access-7kshg\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:36.474970 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.474725 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726"] Nov 02 18:28:36.478574 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.478542 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bee8f6-c879f6577-9l726"] Nov 02 18:28:36.548820 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.548785 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:36.745504 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:36.745419 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" podUID="afdfbd7a-1637-47c2-b84e-abfb6319483a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:28:37.014743 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:37.014027 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:37.279178 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:37.279100 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:37.629813 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:37.629714 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:38.054568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.054540 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" Nov 02 18:28:38.065334 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.065308 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b44de0c-f730-47cd-80e0-4f6b88f5b1fc-kube-api-access-d87vw\" (UniqueName: \"kubernetes.io/projected/5b44de0c-f730-47cd-80e0-4f6b88f5b1fc-kube-api-access-d87vw\") pod \"5b44de0c-f730-47cd-80e0-4f6b88f5b1fc\" (UID: \"5b44de0c-f730-47cd-80e0-4f6b88f5b1fc\") " Nov 02 18:28:38.065463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.065400 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b44de0c-f730-47cd-80e0-4f6b88f5b1fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b44de0c-f730-47cd-80e0-4f6b88f5b1fc-plugins-config-volume\") pod \"5b44de0c-f730-47cd-80e0-4f6b88f5b1fc\" (UID: \"5b44de0c-f730-47cd-80e0-4f6b88f5b1fc\") " Nov 02 18:28:38.065725 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.065703 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b44de0c-f730-47cd-80e0-4f6b88f5b1fc-plugins-config-volume" pod "5b44de0c-f730-47cd-80e0-4f6b88f5b1fc" (UID: "5b44de0c-f730-47cd-80e0-4f6b88f5b1fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:38.065996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.065973 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b44de0c-f730-47cd-80e0-4f6b88f5b1fc-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:38.067461 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.067442 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b44de0c-f730-47cd-80e0-4f6b88f5b1fc-kube-api-access-d87vw" pod "5b44de0c-f730-47cd-80e0-4f6b88f5b1fc" (UID: "5b44de0c-f730-47cd-80e0-4f6b88f5b1fc"). InnerVolumeSpecName "kube-api-access-d87vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:38.155227 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.155186 5549 generic.go:356] "Generic (PLEG): container finished" podID="5b44de0c-f730-47cd-80e0-4f6b88f5b1fc" containerID="68d3c4ec5094b9c8b1a25acc1dbe99052485c54143fb006ab7218b596d0a7ca6" exitCode=0 Nov 02 18:28:38.155401 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.155234 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" Nov 02 18:28:38.155401 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.155263 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" event={"ID":"5b44de0c-f730-47cd-80e0-4f6b88f5b1fc","Type":"ContainerDied","Data":"68d3c4ec5094b9c8b1a25acc1dbe99052485c54143fb006ab7218b596d0a7ca6"} Nov 02 18:28:38.155401 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.155323 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl" event={"ID":"5b44de0c-f730-47cd-80e0-4f6b88f5b1fc","Type":"ContainerDied","Data":"9c95c6620c2dfa33badb0201a5d3a8d15bf461cb0da88f7ac68613c3291c062b"} Nov 02 18:28:38.155401 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.155347 5549 scope.go:122] "RemoveContainer" containerID="68d3c4ec5094b9c8b1a25acc1dbe99052485c54143fb006ab7218b596d0a7ca6" Nov 02 18:28:38.166748 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.166724 5549 scope.go:122] "RemoveContainer" containerID="68d3c4ec5094b9c8b1a25acc1dbe99052485c54143fb006ab7218b596d0a7ca6" Nov 02 18:28:38.167077 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:28:38.167048 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d3c4ec5094b9c8b1a25acc1dbe99052485c54143fb006ab7218b596d0a7ca6\": container with ID starting with 68d3c4ec5094b9c8b1a25acc1dbe99052485c54143fb006ab7218b596d0a7ca6 not found: ID does not exist" containerID="68d3c4ec5094b9c8b1a25acc1dbe99052485c54143fb006ab7218b596d0a7ca6" Nov 02 18:28:38.167171 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.167089 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d3c4ec5094b9c8b1a25acc1dbe99052485c54143fb006ab7218b596d0a7ca6"} err="failed to get container status \"68d3c4ec5094b9c8b1a25acc1dbe99052485c54143fb006ab7218b596d0a7ca6\": rpc error: code = NotFound desc = could not find container \"68d3c4ec5094b9c8b1a25acc1dbe99052485c54143fb006ab7218b596d0a7ca6\": container with ID starting with 68d3c4ec5094b9c8b1a25acc1dbe99052485c54143fb006ab7218b596d0a7ca6 not found: ID does not exist" Nov 02 18:28:38.167329 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.167296 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d87vw\" (UniqueName: \"kubernetes.io/projected/5b44de0c-f730-47cd-80e0-4f6b88f5b1fc-kube-api-access-d87vw\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:38.180437 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.180412 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl"] Nov 02 18:28:38.183865 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.183839 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cc0b41-d965d7fc5-qxkzl"] Nov 02 18:28:38.340441 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.339361 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b44de0c-f730-47cd-80e0-4f6b88f5b1fc" path="/var/lib/kubelet/pods/5b44de0c-f730-47cd-80e0-4f6b88f5b1fc/volumes" Nov 02 18:28:38.340441 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.340152 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbffe25b-3628-4d4a-a7a8-af2eb246aaba" path="/var/lib/kubelet/pods/bbffe25b-3628-4d4a-a7a8-af2eb246aaba/volumes" Nov 02 18:28:38.359493 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:38.359455 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:39.215405 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:39.215350 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:39.979976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:39.977908 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:41.745713 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:41.745681 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:41.880017 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:41.879980 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:42.301985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:42.301929 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:43.133458 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:43.124742 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:43.362682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:43.362650 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:43.500556 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:43.500520 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:44.122812 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:44.122777 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:44.257149 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:44.257106 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:44.927036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:44.926996 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:45.723701 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:45.723648 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:46.232544 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:46.232500 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:46.683216 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:46.683135 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:46.745716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:46.745671 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" podUID="afdfbd7a-1637-47c2-b84e-abfb6319483a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:28:46.833653 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:46.833600 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:47.135409 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:47.135374 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:47.400472 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:47.400354 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:47.589713 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:28:47.589669 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:28:47.874181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:47.874136 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:48.163541 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:48.163458 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:48.516158 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:48.516041 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:49.247300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:49.247266 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:49.940310 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:49.940263 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:49.990273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:49.990246 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:28:50.207381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:50.207349 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:28:50.614119 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:50.612945 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:51.201296 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:51.201243 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:51.326648 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:51.326615 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:51.941329 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:51.941290 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:52.444794 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:52.444752 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m"] Nov 02 18:28:52.445107 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:52.445037 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" podUID="07a5568c-0ad3-4713-92ab-1730ef3e0607" containerName="maas-api" containerID="cri-o://0717b3d52f98f93f88b0183801d55d19284ff5a9dcf62b2ce9e4f869fe42400d" gracePeriod=30 Nov 02 18:28:52.534499 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:52.534460 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:52.856543 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:52.856513 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:28:52.915797 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:52.915761 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07a5568c-0ad3-4713-92ab-1730ef3e0607-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a5568c-0ad3-4713-92ab-1730ef3e0607-maas-api-tls\") pod \"07a5568c-0ad3-4713-92ab-1730ef3e0607\" (UID: \"07a5568c-0ad3-4713-92ab-1730ef3e0607\") " Nov 02 18:28:52.916001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:52.915856 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a5568c-0ad3-4713-92ab-1730ef3e0607-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a5568c-0ad3-4713-92ab-1730ef3e0607-openshift-service-ca\") pod \"07a5568c-0ad3-4713-92ab-1730ef3e0607\" (UID: \"07a5568c-0ad3-4713-92ab-1730ef3e0607\") " Nov 02 18:28:52.916001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:52.915916 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07a5568c-0ad3-4713-92ab-1730ef3e0607-metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a5568c-0ad3-4713-92ab-1730ef3e0607-metrics-certs\") pod \"07a5568c-0ad3-4713-92ab-1730ef3e0607\" (UID: \"07a5568c-0ad3-4713-92ab-1730ef3e0607\") " Nov 02 18:28:52.916096 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:52.916027 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a5568c-0ad3-4713-92ab-1730ef3e0607-kube-api-access-nhcgz\" (UniqueName: \"kubernetes.io/projected/07a5568c-0ad3-4713-92ab-1730ef3e0607-kube-api-access-nhcgz\") pod \"07a5568c-0ad3-4713-92ab-1730ef3e0607\" (UID: \"07a5568c-0ad3-4713-92ab-1730ef3e0607\") " Nov 02 18:28:52.917250 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:52.917200 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a5568c-0ad3-4713-92ab-1730ef3e0607-openshift-service-ca" pod "07a5568c-0ad3-4713-92ab-1730ef3e0607" (UID: "07a5568c-0ad3-4713-92ab-1730ef3e0607"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:52.936461 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:52.936405 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a5568c-0ad3-4713-92ab-1730ef3e0607-maas-api-tls" pod "07a5568c-0ad3-4713-92ab-1730ef3e0607" (UID: "07a5568c-0ad3-4713-92ab-1730ef3e0607"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:28:52.936630 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:52.936434 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a5568c-0ad3-4713-92ab-1730ef3e0607-kube-api-access-nhcgz" pod "07a5568c-0ad3-4713-92ab-1730ef3e0607" (UID: "07a5568c-0ad3-4713-92ab-1730ef3e0607"). InnerVolumeSpecName "kube-api-access-nhcgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:52.936734 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:52.936450 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a5568c-0ad3-4713-92ab-1730ef3e0607-metrics-certs" pod "07a5568c-0ad3-4713-92ab-1730ef3e0607" (UID: "07a5568c-0ad3-4713-92ab-1730ef3e0607"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:28:53.017825 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:53.017781 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhcgz\" (UniqueName: \"kubernetes.io/projected/07a5568c-0ad3-4713-92ab-1730ef3e0607-kube-api-access-nhcgz\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:53.017825 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:53.017818 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a5568c-0ad3-4713-92ab-1730ef3e0607-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:53.018309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:53.017833 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a5568c-0ad3-4713-92ab-1730ef3e0607-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:53.018309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:53.017846 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a5568c-0ad3-4713-92ab-1730ef3e0607-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:53.028441 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:53.028409 5549 scope.go:122] "RemoveContainer" containerID="0717b3d52f98f93f88b0183801d55d19284ff5a9dcf62b2ce9e4f869fe42400d" Nov 02 18:28:53.132719 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:53.132694 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:53.301070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:53.301015 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" event={"ID":"07a5568c-0ad3-4713-92ab-1730ef3e0607","Type":"ContainerDied","Data":"0717b3d52f98f93f88b0183801d55d19284ff5a9dcf62b2ce9e4f869fe42400d"} Nov 02 18:28:53.301248 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:53.301096 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" event={"ID":"07a5568c-0ad3-4713-92ab-1730ef3e0607","Type":"ContainerDied","Data":"b7f2c4b7fad846dd4fe12735078cb6e5ce4b86e25df04d03753f21925bb89c91"} Nov 02 18:28:53.301248 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:53.301174 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m" Nov 02 18:28:53.334892 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:53.334856 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m"] Nov 02 18:28:53.338341 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:53.338311 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-42d786-67689c66ff-hkz5m"] Nov 02 18:28:53.686633 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:53.686598 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:53.892614 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:53.892574 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7"] Nov 02 18:28:53.899688 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:53.899648 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn"] Nov 02 18:28:54.039701 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:54.039667 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:54.166973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:54.166250 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:54.322165 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:54.322093 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a5568c-0ad3-4713-92ab-1730ef3e0607" path="/var/lib/kubelet/pods/07a5568c-0ad3-4713-92ab-1730ef3e0607/volumes" Nov 02 18:28:54.622408 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:54.620374 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:54.766764 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:54.766727 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:55.426240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:55.426197 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:56.116141 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:56.110337 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:56.124495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:56.124109 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" Nov 02 18:28:56.282707 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:56.282673 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afdfbd7a-1637-47c2-b84e-abfb6319483a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdfbd7a-1637-47c2-b84e-abfb6319483a-plugins-config-volume\") pod \"afdfbd7a-1637-47c2-b84e-abfb6319483a\" (UID: \"afdfbd7a-1637-47c2-b84e-abfb6319483a\") " Nov 02 18:28:56.282894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:56.282762 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afdfbd7a-1637-47c2-b84e-abfb6319483a-kube-api-access-f85qc\" (UniqueName: \"kubernetes.io/projected/afdfbd7a-1637-47c2-b84e-abfb6319483a-kube-api-access-f85qc\") pod \"afdfbd7a-1637-47c2-b84e-abfb6319483a\" (UID: \"afdfbd7a-1637-47c2-b84e-abfb6319483a\") " Nov 02 18:28:56.283109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:56.283084 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afdfbd7a-1637-47c2-b84e-abfb6319483a-plugins-config-volume" pod "afdfbd7a-1637-47c2-b84e-abfb6319483a" (UID: "afdfbd7a-1637-47c2-b84e-abfb6319483a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:56.283314 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:56.283299 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdfbd7a-1637-47c2-b84e-abfb6319483a-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:56.284998 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:56.284971 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afdfbd7a-1637-47c2-b84e-abfb6319483a-kube-api-access-f85qc" pod "afdfbd7a-1637-47c2-b84e-abfb6319483a" (UID: "afdfbd7a-1637-47c2-b84e-abfb6319483a"). InnerVolumeSpecName "kube-api-access-f85qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:56.326367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:56.326283 5549 generic.go:356] "Generic (PLEG): container finished" podID="afdfbd7a-1637-47c2-b84e-abfb6319483a" containerID="108926dafb2221310167071da2e527eb7c2a3da5e04508c301b365f808f802a3" exitCode=0 Nov 02 18:28:56.326367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:56.326325 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" Nov 02 18:28:56.326568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:56.326363 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" event={"ID":"afdfbd7a-1637-47c2-b84e-abfb6319483a","Type":"ContainerDied","Data":"108926dafb2221310167071da2e527eb7c2a3da5e04508c301b365f808f802a3"} Nov 02 18:28:56.326568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:56.326418 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z" event={"ID":"afdfbd7a-1637-47c2-b84e-abfb6319483a","Type":"ContainerDied","Data":"00d55ac3e660ca056b7bab2e32db99952598fe414fdb89fa66ff7c1ff786a866"} Nov 02 18:28:56.326568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:56.326449 5549 scope.go:122] "RemoveContainer" containerID="108926dafb2221310167071da2e527eb7c2a3da5e04508c301b365f808f802a3" Nov 02 18:28:56.338924 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:56.338895 5549 scope.go:122] "RemoveContainer" containerID="108926dafb2221310167071da2e527eb7c2a3da5e04508c301b365f808f802a3" Nov 02 18:28:56.339266 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:28:56.339225 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"108926dafb2221310167071da2e527eb7c2a3da5e04508c301b365f808f802a3\": container with ID starting with 108926dafb2221310167071da2e527eb7c2a3da5e04508c301b365f808f802a3 not found: ID does not exist" containerID="108926dafb2221310167071da2e527eb7c2a3da5e04508c301b365f808f802a3" Nov 02 18:28:56.339359 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:56.339272 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"108926dafb2221310167071da2e527eb7c2a3da5e04508c301b365f808f802a3"} err="failed to get container status \"108926dafb2221310167071da2e527eb7c2a3da5e04508c301b365f808f802a3\": rpc error: code = NotFound desc = could not find container \"108926dafb2221310167071da2e527eb7c2a3da5e04508c301b365f808f802a3\": container with ID starting with 108926dafb2221310167071da2e527eb7c2a3da5e04508c301b365f808f802a3 not found: ID does not exist" Nov 02 18:28:56.354241 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:56.354210 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z"] Nov 02 18:28:56.358263 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:56.358214 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bee8f6-74cc6bdfc5-7q79z"] Nov 02 18:28:56.385147 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:56.385110 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f85qc\" (UniqueName: \"kubernetes.io/projected/afdfbd7a-1637-47c2-b84e-abfb6319483a-kube-api-access-f85qc\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:56.838851 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:56.836542 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:57.353402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:57.353360 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:57.521708 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:57.521669 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:57.873255 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:57.873218 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:58.005942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:58.005906 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:58.229172 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:58.229136 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz"] Nov 02 18:28:58.229424 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:58.229394 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" podUID="d725b366-553a-416e-bfc1-90a005778f02" containerName="istio-proxy" containerID="cri-o://0b7b8745dc15112e555471c60eb8afb0e618953964aa0d1cc7f4179f12e0f145" gracePeriod=30 Nov 02 18:28:58.323984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:58.323931 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afdfbd7a-1637-47c2-b84e-abfb6319483a" path="/var/lib/kubelet/pods/afdfbd7a-1637-47c2-b84e-abfb6319483a/volumes" Nov 02 18:28:58.389976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:58.373754 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:58.812279 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:58.812242 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:58.893392 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:58.893351 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7" podUID="8f88db9e-ffae-47f6-ad0e-4d2c834af420" containerName="payload-processing" containerID="cri-o://3368036274b55e6cc59be424b8924f5fa956befa214198d391e0fe41909971f2" gracePeriod=25 Nov 02 18:28:58.900786 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:58.900755 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn" podUID="bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5" containerName="payload-pre-processing" containerID="cri-o://a61c948d81a6741ec472c067da6a27d2abbb00a6b1eebeb08b5b2555a86eddce" gracePeriod=25 Nov 02 18:28:59.025488 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.025453 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:28:59.188519 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.188486 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7" Nov 02 18:28:59.220206 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.220183 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn" Nov 02 18:28:59.354225 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.354148 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f88db9e-ffae-47f6-ad0e-4d2c834af420-kube-api-access-pt5t2\" (UniqueName: \"kubernetes.io/projected/8f88db9e-ffae-47f6-ad0e-4d2c834af420-kube-api-access-pt5t2\") pod \"8f88db9e-ffae-47f6-ad0e-4d2c834af420\" (UID: \"8f88db9e-ffae-47f6-ad0e-4d2c834af420\") " Nov 02 18:28:59.354225 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.354196 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f88db9e-ffae-47f6-ad0e-4d2c834af420-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f88db9e-ffae-47f6-ad0e-4d2c834af420-plugins-config-volume\") pod \"8f88db9e-ffae-47f6-ad0e-4d2c834af420\" (UID: \"8f88db9e-ffae-47f6-ad0e-4d2c834af420\") " Nov 02 18:28:59.354432 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.354364 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5-plugins-config-volume\") pod \"bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5\" (UID: \"bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5\") " Nov 02 18:28:59.354478 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.354464 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5-kube-api-access-4slq8\" (UniqueName: \"kubernetes.io/projected/bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5-kube-api-access-4slq8\") pod \"bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5\" (UID: \"bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5\") " Nov 02 18:28:59.354707 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.354560 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f88db9e-ffae-47f6-ad0e-4d2c834af420-plugins-config-volume" pod "8f88db9e-ffae-47f6-ad0e-4d2c834af420" (UID: "8f88db9e-ffae-47f6-ad0e-4d2c834af420"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:59.354783 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.354710 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5-plugins-config-volume" pod "bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5" (UID: "bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:59.355038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.355019 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f88db9e-ffae-47f6-ad0e-4d2c834af420-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:59.355038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.355038 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:59.356449 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.356426 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f88db9e-ffae-47f6-ad0e-4d2c834af420-kube-api-access-pt5t2" pod "8f88db9e-ffae-47f6-ad0e-4d2c834af420" (UID: "8f88db9e-ffae-47f6-ad0e-4d2c834af420"). InnerVolumeSpecName "kube-api-access-pt5t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:59.356534 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.356457 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5-kube-api-access-4slq8" pod "bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5" (UID: "bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5"). InnerVolumeSpecName "kube-api-access-4slq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:59.421875 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.421846 5549 generic.go:356] "Generic (PLEG): container finished" podID="bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5" containerID="a61c948d81a6741ec472c067da6a27d2abbb00a6b1eebeb08b5b2555a86eddce" exitCode=0 Nov 02 18:28:59.422028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.421895 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn" Nov 02 18:28:59.422028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.421936 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn" event={"ID":"bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5","Type":"ContainerDied","Data":"a61c948d81a6741ec472c067da6a27d2abbb00a6b1eebeb08b5b2555a86eddce"} Nov 02 18:28:59.422028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.422005 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn" event={"ID":"bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5","Type":"ContainerDied","Data":"597051a6ab881520e3aefe089930df41e3cf9dc2ee16c6c12dc0f8d7c095fd12"} Nov 02 18:28:59.422209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.422030 5549 scope.go:122] "RemoveContainer" containerID="a61c948d81a6741ec472c067da6a27d2abbb00a6b1eebeb08b5b2555a86eddce" Nov 02 18:28:59.423603 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.423581 5549 generic.go:356] "Generic (PLEG): container finished" podID="8f88db9e-ffae-47f6-ad0e-4d2c834af420" containerID="3368036274b55e6cc59be424b8924f5fa956befa214198d391e0fe41909971f2" exitCode=0 Nov 02 18:28:59.423686 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.423642 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7" Nov 02 18:28:59.423686 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.423656 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7" event={"ID":"8f88db9e-ffae-47f6-ad0e-4d2c834af420","Type":"ContainerDied","Data":"3368036274b55e6cc59be424b8924f5fa956befa214198d391e0fe41909971f2"} Nov 02 18:28:59.423770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.423694 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7" event={"ID":"8f88db9e-ffae-47f6-ad0e-4d2c834af420","Type":"ContainerDied","Data":"5f46d4991502e8034ba1ee59f3ace518eeb74a85ce8cbbbc98eba5c1cf6c8ed6"} Nov 02 18:28:59.434237 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.434181 5549 scope.go:122] "RemoveContainer" containerID="a61c948d81a6741ec472c067da6a27d2abbb00a6b1eebeb08b5b2555a86eddce" Nov 02 18:28:59.434531 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:28:59.434507 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61c948d81a6741ec472c067da6a27d2abbb00a6b1eebeb08b5b2555a86eddce\": container with ID starting with a61c948d81a6741ec472c067da6a27d2abbb00a6b1eebeb08b5b2555a86eddce not found: ID does not exist" containerID="a61c948d81a6741ec472c067da6a27d2abbb00a6b1eebeb08b5b2555a86eddce" Nov 02 18:28:59.434602 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.434538 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61c948d81a6741ec472c067da6a27d2abbb00a6b1eebeb08b5b2555a86eddce"} err="failed to get container status \"a61c948d81a6741ec472c067da6a27d2abbb00a6b1eebeb08b5b2555a86eddce\": rpc error: code = NotFound desc = could not find container \"a61c948d81a6741ec472c067da6a27d2abbb00a6b1eebeb08b5b2555a86eddce\": container with ID starting with a61c948d81a6741ec472c067da6a27d2abbb00a6b1eebeb08b5b2555a86eddce not found: ID does not exist" Nov 02 18:28:59.434602 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.434556 5549 scope.go:122] "RemoveContainer" containerID="3368036274b55e6cc59be424b8924f5fa956befa214198d391e0fe41909971f2" Nov 02 18:28:59.443871 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.443848 5549 scope.go:122] "RemoveContainer" containerID="3368036274b55e6cc59be424b8924f5fa956befa214198d391e0fe41909971f2" Nov 02 18:28:59.444148 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:28:59.444130 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3368036274b55e6cc59be424b8924f5fa956befa214198d391e0fe41909971f2\": container with ID starting with 3368036274b55e6cc59be424b8924f5fa956befa214198d391e0fe41909971f2 not found: ID does not exist" containerID="3368036274b55e6cc59be424b8924f5fa956befa214198d391e0fe41909971f2" Nov 02 18:28:59.444205 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.444154 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3368036274b55e6cc59be424b8924f5fa956befa214198d391e0fe41909971f2"} err="failed to get container status \"3368036274b55e6cc59be424b8924f5fa956befa214198d391e0fe41909971f2\": rpc error: code = NotFound desc = could not find container \"3368036274b55e6cc59be424b8924f5fa956befa214198d391e0fe41909971f2\": container with ID starting with 3368036274b55e6cc59be424b8924f5fa956befa214198d391e0fe41909971f2 not found: ID does not exist" Nov 02 18:28:59.450071 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.450048 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7"] Nov 02 18:28:59.453384 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.453362 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b6b6f4-85c49d9b75-6szp7"] Nov 02 18:28:59.456161 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.456136 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4slq8\" (UniqueName: \"kubernetes.io/projected/bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5-kube-api-access-4slq8\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:59.456242 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.456166 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt5t2\" (UniqueName: \"kubernetes.io/projected/8f88db9e-ffae-47f6-ad0e-4d2c834af420-kube-api-access-pt5t2\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:59.461648 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.461628 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn"] Nov 02 18:28:59.465701 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.465679 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b6b6f4-74548ccbd7-hflxn"] Nov 02 18:28:59.771101 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:28:59.771066 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:00.295266 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:00.294287 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:00.328576 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:00.328527 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f88db9e-ffae-47f6-ad0e-4d2c834af420" path="/var/lib/kubelet/pods/8f88db9e-ffae-47f6-ad0e-4d2c834af420/volumes" Nov 02 18:29:00.329121 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:00.329103 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5" path="/var/lib/kubelet/pods/bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5/volumes" Nov 02 18:29:00.918938 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:00.918893 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:01.197961 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:01.197916 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:01.330069 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:01.330038 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:01.798300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:01.798273 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:01.930617 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:01.930581 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:02.164187 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:02.164099 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:02.631879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:02.631838 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:02.769278 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:02.769246 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:02.931212 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:02.931136 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:03.335717 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.335565 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:03.462092 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.462055 5549 generic.go:356] "Generic (PLEG): container finished" podID="d725b366-553a-416e-bfc1-90a005778f02" containerID="0b7b8745dc15112e555471c60eb8afb0e618953964aa0d1cc7f4179f12e0f145" exitCode=0 Nov 02 18:29:03.462261 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.462128 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" event={"ID":"d725b366-553a-416e-bfc1-90a005778f02","Type":"ContainerDied","Data":"0b7b8745dc15112e555471c60eb8afb0e618953964aa0d1cc7f4179f12e0f145"} Nov 02 18:29:03.516064 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.516032 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:29:03.537180 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.537152 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d725b366-553a-416e-bfc1-90a005778f02-kube-api-access-vtptf\" (UniqueName: \"kubernetes.io/projected/d725b366-553a-416e-bfc1-90a005778f02-kube-api-access-vtptf\") pod \"d725b366-553a-416e-bfc1-90a005778f02\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " Nov 02 18:29:03.537365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.537196 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d725b366-553a-416e-bfc1-90a005778f02-istio-token\" (UniqueName: \"kubernetes.io/projected/d725b366-553a-416e-bfc1-90a005778f02-istio-token\") pod \"d725b366-553a-416e-bfc1-90a005778f02\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " Nov 02 18:29:03.537365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.537235 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-workload-certs\") pod \"d725b366-553a-416e-bfc1-90a005778f02\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " Nov 02 18:29:03.537365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.537308 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-istio-envoy\") pod \"d725b366-553a-416e-bfc1-90a005778f02\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " Nov 02 18:29:03.537365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.537347 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d725b366-553a-416e-bfc1-90a005778f02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d725b366-553a-416e-bfc1-90a005778f02-istiod-ca-cert\") pod \"d725b366-553a-416e-bfc1-90a005778f02\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " Nov 02 18:29:03.537603 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.537389 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d725b366-553a-416e-bfc1-90a005778f02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d725b366-553a-416e-bfc1-90a005778f02-istio-podinfo\") pod \"d725b366-553a-416e-bfc1-90a005778f02\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " Nov 02 18:29:03.537603 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.537518 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-istio-data\") pod \"d725b366-553a-416e-bfc1-90a005778f02\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " Nov 02 18:29:03.537603 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.537574 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-credential-socket\") pod \"d725b366-553a-416e-bfc1-90a005778f02\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " Nov 02 18:29:03.537750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.537665 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-workload-socket\") pod \"d725b366-553a-416e-bfc1-90a005778f02\" (UID: \"d725b366-553a-416e-bfc1-90a005778f02\") " Nov 02 18:29:03.537750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.537693 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-workload-certs" pod "d725b366-553a-416e-bfc1-90a005778f02" (UID: "d725b366-553a-416e-bfc1-90a005778f02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:03.537922 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.537902 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-istio-data" pod "d725b366-553a-416e-bfc1-90a005778f02" (UID: "d725b366-553a-416e-bfc1-90a005778f02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:03.538185 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.538141 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-workload-socket" pod "d725b366-553a-416e-bfc1-90a005778f02" (UID: "d725b366-553a-416e-bfc1-90a005778f02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:03.538269 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.538220 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-credential-socket" pod "d725b366-553a-416e-bfc1-90a005778f02" (UID: "d725b366-553a-416e-bfc1-90a005778f02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:03.538349 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.538328 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:03.538404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.538356 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:03.538404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.538372 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:03.538791 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.538687 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d725b366-553a-416e-bfc1-90a005778f02-istiod-ca-cert" pod "d725b366-553a-416e-bfc1-90a005778f02" (UID: "d725b366-553a-416e-bfc1-90a005778f02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:03.541829 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.540665 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d725b366-553a-416e-bfc1-90a005778f02-istio-podinfo" pod "d725b366-553a-416e-bfc1-90a005778f02" (UID: "d725b366-553a-416e-bfc1-90a005778f02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:29:03.542573 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.542549 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d725b366-553a-416e-bfc1-90a005778f02-istio-token" pod "d725b366-553a-416e-bfc1-90a005778f02" (UID: "d725b366-553a-416e-bfc1-90a005778f02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:03.547173 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.547146 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-istio-envoy" pod "d725b366-553a-416e-bfc1-90a005778f02" (UID: "d725b366-553a-416e-bfc1-90a005778f02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:03.553601 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.553575 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d725b366-553a-416e-bfc1-90a005778f02-kube-api-access-vtptf" pod "d725b366-553a-416e-bfc1-90a005778f02" (UID: "d725b366-553a-416e-bfc1-90a005778f02"). InnerVolumeSpecName "kube-api-access-vtptf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:03.639414 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.639366 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:03.639414 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.639407 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d725b366-553a-416e-bfc1-90a005778f02-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:03.639414 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.639419 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d725b366-553a-416e-bfc1-90a005778f02-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:03.639414 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.639428 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d725b366-553a-416e-bfc1-90a005778f02-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:03.639904 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.639439 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtptf\" (UniqueName: \"kubernetes.io/projected/d725b366-553a-416e-bfc1-90a005778f02-kube-api-access-vtptf\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:03.639904 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.639450 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d725b366-553a-416e-bfc1-90a005778f02-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:03.855972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.855880 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:29:03.924427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:03.924284 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:04.470783 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:04.470742 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" event={"ID":"d725b366-553a-416e-bfc1-90a005778f02","Type":"ContainerDied","Data":"316060e70f6c4bdf01c8293c93b49598a69e04d3b83bd5f189e92ada22b99a68"} Nov 02 18:29:04.471014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:04.470813 5549 scope.go:122] "RemoveContainer" containerID="0b7b8745dc15112e555471c60eb8afb0e618953964aa0d1cc7f4179f12e0f145" Nov 02 18:29:04.471014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:04.470750 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz" Nov 02 18:29:04.492741 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:04.492715 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz"] Nov 02 18:29:04.498268 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:04.498242 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-j7jzz"] Nov 02 18:29:04.747550 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:04.747517 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:05.265431 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:05.265394 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27"] Nov 02 18:29:05.281597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:05.281558 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh"] Nov 02 18:29:05.398097 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:05.397577 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:05.562073 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:05.560170 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:05.701014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:05.700569 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:06.321879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:06.321834 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d725b366-553a-416e-bfc1-90a005778f02" path="/var/lib/kubelet/pods/d725b366-553a-416e-bfc1-90a005778f02/volumes" Nov 02 18:29:06.557522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:06.557491 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:06.685207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:06.685108 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:07.255843 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:07.255754 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:08.022258 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:08.022214 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:08.170439 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:08.170399 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm"] Nov 02 18:29:08.181787 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:08.181752 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4"] Nov 02 18:29:08.289402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:08.289322 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:08.506312 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:08.506275 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:08.990558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:08.987553 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:09.619200 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:09.619159 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:10.266547 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:10.266491 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" podUID="88ee8230-6e65-4a89-b9bd-1f7100b03218" containerName="payload-processing" containerID="cri-o://2c8fcd8bcc9d19d7465aa82957f7b2a66d3db5690c79ca44f8442d2cf9a57ab7" gracePeriod=25 Nov 02 18:29:10.282436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:10.282397 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" podUID="9baba3f3-ce78-4167-8f82-0e24b38a5cda" containerName="payload-pre-processing" containerID="cri-o://d2eb79253c87269fc3bb921f14602077d89fedc597159de40def37e2d7a0978f" gracePeriod=25 Nov 02 18:29:10.378219 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:10.378169 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:10.514218 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:10.514179 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:11.037373 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:11.037333 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:11.727239 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:11.727201 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:12.422023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:12.421971 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:12.832171 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:12.832138 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:12.980981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:12.980918 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:13.111374 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:13.111284 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:13.171402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:13.171346 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" podUID="fd061465-fa5b-4565-a280-f546c45b9f0e" containerName="payload-processing" containerID="cri-o://01c128338df9ff573678190f5bda667e1e8df6353d8588f5f8d3a793e2c6016c" gracePeriod=25 Nov 02 18:29:13.182584 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:13.182551 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" podUID="e14cd218-72ea-4e71-bd6d-daad87430119" containerName="payload-pre-processing" containerID="cri-o://cc6f8a40384ca3fd6280c093d95e645b13dd2f74a1315606da71eb3d81808398" gracePeriod=25 Nov 02 18:29:13.639173 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:13.639134 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:14.131011 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:14.130973 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:14.682350 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:14.677303 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:15.292401 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:15.292363 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:15.639415 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:15.639308 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" podUID="9baba3f3-ce78-4167-8f82-0e24b38a5cda" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:29:15.995021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:15.994980 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:16.483022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:16.482983 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:16.647270 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:16.647235 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:16.682174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:16.682137 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" podUID="88ee8230-6e65-4a89-b9bd-1f7100b03218" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:29:17.303875 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:17.303838 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:17.598920 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:29:17.598831 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:29:17.903667 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:17.903570 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:18.622471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:18.622427 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:18.929472 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:18.929378 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx"] Nov 02 18:29:19.051014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.050972 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:19.267686 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.267645 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:19.351318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.351278 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx"] Nov 02 18:29:19.351572 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.351539 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" podUID="a8a52d50-ddee-466d-9630-50d84b3eac4a" containerName="maas-api" containerID="cri-o://0788742d42feff2df6fdbcb778274787c326e49c0d2aef6e5ca3fb32e26120e2" gracePeriod=30 Nov 02 18:29:19.428229 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.428196 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:19.622048 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.621989 5549 generic.go:356] "Generic (PLEG): container finished" podID="a8a52d50-ddee-466d-9630-50d84b3eac4a" containerID="0788742d42feff2df6fdbcb778274787c326e49c0d2aef6e5ca3fb32e26120e2" exitCode=0 Nov 02 18:29:19.622464 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.622421 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" event={"ID":"a8a52d50-ddee-466d-9630-50d84b3eac4a","Type":"ContainerDied","Data":"0788742d42feff2df6fdbcb778274787c326e49c0d2aef6e5ca3fb32e26120e2"} Nov 02 18:29:19.725244 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.725215 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 18:29:19.860544 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.860422 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a52d50-ddee-466d-9630-50d84b3eac4a-kube-api-access-x6p6n\" (UniqueName: \"kubernetes.io/projected/a8a52d50-ddee-466d-9630-50d84b3eac4a-kube-api-access-x6p6n\") pod \"a8a52d50-ddee-466d-9630-50d84b3eac4a\" (UID: \"a8a52d50-ddee-466d-9630-50d84b3eac4a\") " Nov 02 18:29:19.860544 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.860537 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8a52d50-ddee-466d-9630-50d84b3eac4a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a52d50-ddee-466d-9630-50d84b3eac4a-metrics-certs\") pod \"a8a52d50-ddee-466d-9630-50d84b3eac4a\" (UID: \"a8a52d50-ddee-466d-9630-50d84b3eac4a\") " Nov 02 18:29:19.860942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.860611 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8a52d50-ddee-466d-9630-50d84b3eac4a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a52d50-ddee-466d-9630-50d84b3eac4a-maas-api-tls\") pod \"a8a52d50-ddee-466d-9630-50d84b3eac4a\" (UID: \"a8a52d50-ddee-466d-9630-50d84b3eac4a\") " Nov 02 18:29:19.860942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.860790 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8a52d50-ddee-466d-9630-50d84b3eac4a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a52d50-ddee-466d-9630-50d84b3eac4a-openshift-service-ca\") pod \"a8a52d50-ddee-466d-9630-50d84b3eac4a\" (UID: \"a8a52d50-ddee-466d-9630-50d84b3eac4a\") " Nov 02 18:29:19.864098 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.864066 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a52d50-ddee-466d-9630-50d84b3eac4a-openshift-service-ca" pod "a8a52d50-ddee-466d-9630-50d84b3eac4a" (UID: "a8a52d50-ddee-466d-9630-50d84b3eac4a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:19.865380 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.865347 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a52d50-ddee-466d-9630-50d84b3eac4a-kube-api-access-x6p6n" pod "a8a52d50-ddee-466d-9630-50d84b3eac4a" (UID: "a8a52d50-ddee-466d-9630-50d84b3eac4a"). InnerVolumeSpecName "kube-api-access-x6p6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:19.866570 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.866535 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a52d50-ddee-466d-9630-50d84b3eac4a-metrics-certs" pod "a8a52d50-ddee-466d-9630-50d84b3eac4a" (UID: "a8a52d50-ddee-466d-9630-50d84b3eac4a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:19.866671 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.866546 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a52d50-ddee-466d-9630-50d84b3eac4a-maas-api-tls" pod "a8a52d50-ddee-466d-9630-50d84b3eac4a" (UID: "a8a52d50-ddee-466d-9630-50d84b3eac4a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:19.964049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.964010 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a52d50-ddee-466d-9630-50d84b3eac4a-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:19.964049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.964045 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a52d50-ddee-466d-9630-50d84b3eac4a-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:19.964345 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.964062 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a52d50-ddee-466d-9630-50d84b3eac4a-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:19.964345 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.964078 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6p6n\" (UniqueName: \"kubernetes.io/projected/a8a52d50-ddee-466d-9630-50d84b3eac4a-kube-api-access-x6p6n\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:19.964900 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:19.964870 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:20.098263 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:20.098231 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:20.436024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:20.428908 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:20.585014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:20.584825 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:20.632733 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:20.632700 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" event={"ID":"a8a52d50-ddee-466d-9630-50d84b3eac4a","Type":"ContainerDied","Data":"fbb9bb9642429f5c959b608a9ea07757e8744c98ba9e42f9b78080dfe0014f51"} Nov 02 18:29:20.632910 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:20.632764 5549 scope.go:122] "RemoveContainer" containerID="0788742d42feff2df6fdbcb778274787c326e49c0d2aef6e5ca3fb32e26120e2" Nov 02 18:29:20.633017 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:20.632911 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx" Nov 02 18:29:20.658019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:20.657947 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx"] Nov 02 18:29:20.662470 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:20.662441 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2961d5-656655f48f-k9shx"] Nov 02 18:29:20.793285 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:20.793244 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:21.043146 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:21.043102 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:21.465875 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:21.465837 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:21.529627 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:21.529588 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" podUID="fd061465-fa5b-4565-a280-f546c45b9f0e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:29:21.529879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:21.529596 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" podUID="e14cd218-72ea-4e71-bd6d-daad87430119" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:29:21.828970 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:21.828820 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:22.321570 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:22.321532 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a52d50-ddee-466d-9630-50d84b3eac4a" path="/var/lib/kubelet/pods/a8a52d50-ddee-466d-9630-50d84b3eac4a/volumes" Nov 02 18:29:22.598045 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:22.597922 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:23.046352 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:23.046291 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:23.182063 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:23.182029 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:23.318845 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:23.318754 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:23.469874 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:23.469839 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:23.792586 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:23.792552 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:23.930047 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:23.930000 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" podUID="766b693c-dd34-4892-9eeb-3528e08a3147" containerName="payload-processing" containerID="cri-o://e4a7ba3036f8bebbbf8f6efae196aab43c58866bc33d4584b0c246ad16abb892" gracePeriod=25 Nov 02 18:29:24.224258 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.224187 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:24.361235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.361200 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:24.456501 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.456468 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7"] Nov 02 18:29:24.457273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457253 5549 cpu_manager.go:414] "containerMap: removing container" podUID="bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5" containerName="payload-pre-processing" Nov 02 18:29:24.457327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457274 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5" containerName="payload-pre-processing" Nov 02 18:29:24.457327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457290 5549 cpu_manager.go:414] "containerMap: removing container" podUID="d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" containerName="istio-proxy" Nov 02 18:29:24.457327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457295 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" containerName="istio-proxy" Nov 02 18:29:24.457327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457303 5549 cpu_manager.go:414] "containerMap: removing container" podUID="52c33649-7da9-4772-817d-d924c1013123" containerName="payload-pre-processing" Nov 02 18:29:24.457327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457308 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c33649-7da9-4772-817d-d924c1013123" containerName="payload-pre-processing" Nov 02 18:29:24.457327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457325 5549 cpu_manager.go:414] "containerMap: removing container" podUID="ff3423b0-fb1a-4082-8227-bd154ed372d7" containerName="payload-processing" Nov 02 18:29:24.457327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457330 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3423b0-fb1a-4082-8227-bd154ed372d7" containerName="payload-processing" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457339 5549 cpu_manager.go:414] "containerMap: removing container" podUID="8f88db9e-ffae-47f6-ad0e-4d2c834af420" containerName="payload-processing" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457344 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f88db9e-ffae-47f6-ad0e-4d2c834af420" containerName="payload-processing" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457350 5549 cpu_manager.go:414] "containerMap: removing container" podUID="d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" containerName="istio-proxy" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457355 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" containerName="istio-proxy" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457367 5549 cpu_manager.go:414] "containerMap: removing container" podUID="216112c3-41c7-45f6-9553-628fa92c38cb" containerName="maas-api" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457372 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216112c3-41c7-45f6-9553-628fa92c38cb" containerName="maas-api" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457379 5549 cpu_manager.go:414] "containerMap: removing container" podUID="078222c5-8861-4bb3-9969-5aca397e9195" containerName="payload-processing" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457384 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078222c5-8861-4bb3-9969-5aca397e9195" containerName="payload-processing" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457390 5549 cpu_manager.go:414] "containerMap: removing container" podUID="bbffe25b-3628-4d4a-a7a8-af2eb246aaba" containerName="payload-processing" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457394 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbffe25b-3628-4d4a-a7a8-af2eb246aaba" containerName="payload-processing" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457407 5549 cpu_manager.go:414] "containerMap: removing container" podUID="d725b366-553a-416e-bfc1-90a005778f02" containerName="istio-proxy" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457412 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d725b366-553a-416e-bfc1-90a005778f02" containerName="istio-proxy" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457424 5549 cpu_manager.go:414] "containerMap: removing container" podUID="5b44de0c-f730-47cd-80e0-4f6b88f5b1fc" containerName="payload-pre-processing" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457429 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b44de0c-f730-47cd-80e0-4f6b88f5b1fc" containerName="payload-pre-processing" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457437 5549 cpu_manager.go:414] "containerMap: removing container" podUID="bc5febcd-72c8-415a-888a-edf57777464c" containerName="main" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457443 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc5febcd-72c8-415a-888a-edf57777464c" containerName="main" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457449 5549 cpu_manager.go:414] "containerMap: removing container" podUID="07a5568c-0ad3-4713-92ab-1730ef3e0607" containerName="maas-api" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457454 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a5568c-0ad3-4713-92ab-1730ef3e0607" containerName="maas-api" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457462 5549 cpu_manager.go:414] "containerMap: removing container" podUID="f2b6e563-a46e-42e2-bc91-5a0277272b56" containerName="payload-pre-processing" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457466 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b6e563-a46e-42e2-bc91-5a0277272b56" containerName="payload-pre-processing" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457473 5549 cpu_manager.go:414] "containerMap: removing container" podUID="a8a52d50-ddee-466d-9630-50d84b3eac4a" containerName="maas-api" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457478 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a52d50-ddee-466d-9630-50d84b3eac4a" containerName="maas-api" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457490 5549 cpu_manager.go:414] "containerMap: removing container" podUID="afdfbd7a-1637-47c2-b84e-abfb6319483a" containerName="payload-pre-processing" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457495 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afdfbd7a-1637-47c2-b84e-abfb6319483a" containerName="payload-pre-processing" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457500 5549 cpu_manager.go:414] "containerMap: removing container" podUID="e56ccdbd-6737-4a84-9d80-59018a6b77ff" containerName="istio-proxy" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457505 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56ccdbd-6737-4a84-9d80-59018a6b77ff" containerName="istio-proxy" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457513 5549 cpu_manager.go:414] "containerMap: removing container" podUID="c9b89a1f-2e31-4335-9f88-acacb8b00000" containerName="istio-proxy" Nov 02 18:29:24.457560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457517 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b89a1f-2e31-4335-9f88-acacb8b00000" containerName="istio-proxy" Nov 02 18:29:24.458564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457612 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc5febcd-72c8-415a-888a-edf57777464c" containerName="main" Nov 02 18:29:24.458564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457621 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a5568c-0ad3-4713-92ab-1730ef3e0607" containerName="maas-api" Nov 02 18:29:24.458564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457630 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b6e563-a46e-42e2-bc91-5a0277272b56" containerName="payload-pre-processing" Nov 02 18:29:24.458564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457641 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a52d50-ddee-466d-9630-50d84b3eac4a" containerName="maas-api" Nov 02 18:29:24.458564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457651 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="afdfbd7a-1637-47c2-b84e-abfb6319483a" containerName="payload-pre-processing" Nov 02 18:29:24.458564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457661 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56ccdbd-6737-4a84-9d80-59018a6b77ff" containerName="istio-proxy" Nov 02 18:29:24.458564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457668 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b89a1f-2e31-4335-9f88-acacb8b00000" containerName="istio-proxy" Nov 02 18:29:24.458564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457678 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf572a0-b48e-4b0b-adf5-e1e8b8a70bb5" containerName="payload-pre-processing" Nov 02 18:29:24.458564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457687 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" containerName="istio-proxy" Nov 02 18:29:24.458564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457695 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c33649-7da9-4772-817d-d924c1013123" containerName="payload-pre-processing" Nov 02 18:29:24.458564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457707 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff3423b0-fb1a-4082-8227-bd154ed372d7" containerName="payload-processing" Nov 02 18:29:24.458564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457719 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f88db9e-ffae-47f6-ad0e-4d2c834af420" containerName="payload-processing" Nov 02 18:29:24.458564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457729 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e4e1d9-7229-43d7-8e25-50d6dc6d1b23" containerName="istio-proxy" Nov 02 18:29:24.458564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457742 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="216112c3-41c7-45f6-9553-628fa92c38cb" containerName="maas-api" Nov 02 18:29:24.458564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457751 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="078222c5-8861-4bb3-9969-5aca397e9195" containerName="payload-processing" Nov 02 18:29:24.458564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457760 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbffe25b-3628-4d4a-a7a8-af2eb246aaba" containerName="payload-processing" Nov 02 18:29:24.458564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457772 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d725b366-553a-416e-bfc1-90a005778f02" containerName="istio-proxy" Nov 02 18:29:24.458564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.457782 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b44de0c-f730-47cd-80e0-4f6b88f5b1fc" containerName="payload-pre-processing" Nov 02 18:29:24.461761 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.461743 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.466045 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.466021 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-7771bf-openshift-default-dockercfg-k8z2z\"" Nov 02 18:29:24.490012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.481196 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7"] Nov 02 18:29:24.553707 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.553674 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-workload-socket\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.553887 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.553721 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istiod-ca-cert\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.553887 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.553877 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-data\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.554005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.553899 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-envoy\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.554005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.553928 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-podinfo\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.554082 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.554029 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z22t2\" (UniqueName: \"kubernetes.io/projected/4603b3a2-b9a8-4a06-97c0-a97d08f96470-kube-api-access-z22t2\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.554122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.554092 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-credential-socket\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.554161 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.554125 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-token\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.554230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.554210 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-workload-certs\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.655287 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.655255 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-workload-certs\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.655478 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.655311 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-workload-socket\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.655478 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.655358 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istiod-ca-cert\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.655579 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.655486 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-data\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.655579 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.655507 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-envoy\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.655579 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.655540 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-podinfo\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.655579 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.655570 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z22t2\" (UniqueName: \"kubernetes.io/projected/4603b3a2-b9a8-4a06-97c0-a97d08f96470-kube-api-access-z22t2\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.655858 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.655667 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-credential-socket\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.655858 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.655689 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-workload-certs\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.655858 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.655707 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-workload-socket\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.655858 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.655719 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-token\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.656145 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.655998 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-data\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.656145 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.656018 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-credential-socket\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.656286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.656261 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istiod-ca-cert\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.658244 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.658218 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-envoy\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.658359 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.658315 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-podinfo\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.663576 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.663549 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-token\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.663729 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.663707 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z22t2\" (UniqueName: \"kubernetes.io/projected/4603b3a2-b9a8-4a06-97c0-a97d08f96470-kube-api-access-z22t2\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.779492 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.779407 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:24.917029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:24.916923 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:25.067420 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:25.067385 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7"] Nov 02 18:29:25.069412 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:29:25.069389 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4603b3a2_b9a8_4a06_97c0_a97d08f96470.slice/crio-7fea6645e315b8cee268a52b49e4c84b6bb686453c0e408b7c53b12469b61576 WatchSource:0}: Error finding container 7fea6645e315b8cee268a52b49e4c84b6bb686453c0e408b7c53b12469b61576: Status 404 returned error can't find the container with id 7fea6645e315b8cee268a52b49e4c84b6bb686453c0e408b7c53b12469b61576 Nov 02 18:29:25.072840 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:25.072798 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:29:25.072930 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:25.072888 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:29:25.072995 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:25.072975 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:29:25.636877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:25.636438 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:25.644051 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:25.644012 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" podUID="9baba3f3-ce78-4167-8f82-0e24b38a5cda" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:29:25.671892 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:25.671854 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" event={"ID":"4603b3a2-b9a8-4a06-97c0-a97d08f96470","Type":"ContainerStarted","Data":"3bf1c88325b40080cb32b45a91a28d059648effddbc58871b506879cedb7f30d"} Nov 02 18:29:25.672061 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:25.671980 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" event={"ID":"4603b3a2-b9a8-4a06-97c0-a97d08f96470","Type":"ContainerStarted","Data":"7fea6645e315b8cee268a52b49e4c84b6bb686453c0e408b7c53b12469b61576"} Nov 02 18:29:25.804094 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:25.804050 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:26.242004 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:26.241947 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:26.704420 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:26.704377 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" podUID="88ee8230-6e65-4a89-b9bd-1f7100b03218" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:29:26.807308 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:26.807275 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body= Nov 02 18:29:26.807524 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:26.807375 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" podUID="4603b3a2-b9a8-4a06-97c0-a97d08f96470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" Nov 02 18:29:26.916943 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:26.916901 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:27.064723 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:27.064644 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:27.654471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:27.654440 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:27.804127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:27.804085 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body= Nov 02 18:29:27.804738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:27.804697 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" podUID="4603b3a2-b9a8-4a06-97c0-a97d08f96470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" Nov 02 18:29:28.207212 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:28.207171 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:28.573656 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:28.573572 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:28.774343 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:28.774306 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:28.784397 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:28.784358 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body= Nov 02 18:29:28.784698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:28.784665 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" podUID="4603b3a2-b9a8-4a06-97c0-a97d08f96470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" Nov 02 18:29:29.202400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:29.202364 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:29.361730 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:29.361692 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:29.795845 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:29.795356 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body= Nov 02 18:29:29.795845 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:29.795432 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" podUID="4603b3a2-b9a8-4a06-97c0-a97d08f96470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" Nov 02 18:29:29.841343 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:29.841306 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:30.003738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.003683 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:30.623351 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.623319 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" Nov 02 18:29:30.643580 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.643525 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" podStartSLOduration=6.64350638 podStartE2EDuration="6.64350638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:29:25.704442761 +0000 UTC m=+3337.492948290" watchObservedRunningTime="2026-11-02 18:29:30.64350638 +0000 UTC m=+3342.432011911" Nov 02 18:29:30.674764 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.674737 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88ee8230-6e65-4a89-b9bd-1f7100b03218-kube-api-access-jcmst\" (UniqueName: \"kubernetes.io/projected/88ee8230-6e65-4a89-b9bd-1f7100b03218-kube-api-access-jcmst\") pod \"88ee8230-6e65-4a89-b9bd-1f7100b03218\" (UID: \"88ee8230-6e65-4a89-b9bd-1f7100b03218\") " Nov 02 18:29:30.674922 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.674824 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88ee8230-6e65-4a89-b9bd-1f7100b03218-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88ee8230-6e65-4a89-b9bd-1f7100b03218-plugins-config-volume\") pod \"88ee8230-6e65-4a89-b9bd-1f7100b03218\" (UID: \"88ee8230-6e65-4a89-b9bd-1f7100b03218\") " Nov 02 18:29:30.675345 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.675316 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88ee8230-6e65-4a89-b9bd-1f7100b03218-plugins-config-volume" pod "88ee8230-6e65-4a89-b9bd-1f7100b03218" (UID: "88ee8230-6e65-4a89-b9bd-1f7100b03218"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:30.677185 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.677161 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ee8230-6e65-4a89-b9bd-1f7100b03218-kube-api-access-jcmst" pod "88ee8230-6e65-4a89-b9bd-1f7100b03218" (UID: "88ee8230-6e65-4a89-b9bd-1f7100b03218"). InnerVolumeSpecName "kube-api-access-jcmst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:30.700186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.700146 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" Nov 02 18:29:30.721986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.718305 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:30.767120 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.767079 5549 generic.go:356] "Generic (PLEG): container finished" podID="88ee8230-6e65-4a89-b9bd-1f7100b03218" containerID="2c8fcd8bcc9d19d7465aa82957f7b2a66d3db5690c79ca44f8442d2cf9a57ab7" exitCode=0 Nov 02 18:29:30.767321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.767145 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" Nov 02 18:29:30.767321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.767255 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" event={"ID":"88ee8230-6e65-4a89-b9bd-1f7100b03218","Type":"ContainerDied","Data":"2c8fcd8bcc9d19d7465aa82957f7b2a66d3db5690c79ca44f8442d2cf9a57ab7"} Nov 02 18:29:30.767321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.767309 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27" event={"ID":"88ee8230-6e65-4a89-b9bd-1f7100b03218","Type":"ContainerDied","Data":"3fa58c038d6298ff73d01b46ea499fb0a73d34ceaedd87ac2d9108024713a23d"} Nov 02 18:29:30.767474 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.767341 5549 scope.go:122] "RemoveContainer" containerID="2c8fcd8bcc9d19d7465aa82957f7b2a66d3db5690c79ca44f8442d2cf9a57ab7" Nov 02 18:29:30.769617 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.769576 5549 generic.go:356] "Generic (PLEG): container finished" podID="9baba3f3-ce78-4167-8f82-0e24b38a5cda" containerID="d2eb79253c87269fc3bb921f14602077d89fedc597159de40def37e2d7a0978f" exitCode=0 Nov 02 18:29:30.769772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.769685 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" Nov 02 18:29:30.769772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.769706 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" event={"ID":"9baba3f3-ce78-4167-8f82-0e24b38a5cda","Type":"ContainerDied","Data":"d2eb79253c87269fc3bb921f14602077d89fedc597159de40def37e2d7a0978f"} Nov 02 18:29:30.769772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.769763 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh" event={"ID":"9baba3f3-ce78-4167-8f82-0e24b38a5cda","Type":"ContainerDied","Data":"8728de1d36073acfeddd1c2dfbe0ae3d2ee9a4d97756ef564d5442354d16e1b7"} Nov 02 18:29:30.776069 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.776045 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9baba3f3-ce78-4167-8f82-0e24b38a5cda-kube-api-access-nwfkl\" (UniqueName: \"kubernetes.io/projected/9baba3f3-ce78-4167-8f82-0e24b38a5cda-kube-api-access-nwfkl\") pod \"9baba3f3-ce78-4167-8f82-0e24b38a5cda\" (UID: \"9baba3f3-ce78-4167-8f82-0e24b38a5cda\") " Nov 02 18:29:30.776171 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.776097 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9baba3f3-ce78-4167-8f82-0e24b38a5cda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9baba3f3-ce78-4167-8f82-0e24b38a5cda-plugins-config-volume\") pod \"9baba3f3-ce78-4167-8f82-0e24b38a5cda\" (UID: \"9baba3f3-ce78-4167-8f82-0e24b38a5cda\") " Nov 02 18:29:30.776946 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.776919 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcmst\" (UniqueName: \"kubernetes.io/projected/88ee8230-6e65-4a89-b9bd-1f7100b03218-kube-api-access-jcmst\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:30.777076 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.777021 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88ee8230-6e65-4a89-b9bd-1f7100b03218-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:30.777355 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.777329 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9baba3f3-ce78-4167-8f82-0e24b38a5cda-plugins-config-volume" pod "9baba3f3-ce78-4167-8f82-0e24b38a5cda" (UID: "9baba3f3-ce78-4167-8f82-0e24b38a5cda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:30.789225 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.789196 5549 scope.go:122] "RemoveContainer" containerID="2c8fcd8bcc9d19d7465aa82957f7b2a66d3db5690c79ca44f8442d2cf9a57ab7" Nov 02 18:29:30.805878 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.803073 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9baba3f3-ce78-4167-8f82-0e24b38a5cda-kube-api-access-nwfkl" pod "9baba3f3-ce78-4167-8f82-0e24b38a5cda" (UID: "9baba3f3-ce78-4167-8f82-0e24b38a5cda"). InnerVolumeSpecName "kube-api-access-nwfkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:30.805878 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.804308 5549 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body= Nov 02 18:29:30.805878 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.804361 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" podUID="4603b3a2-b9a8-4a06-97c0-a97d08f96470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" Nov 02 18:29:30.808417 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:29:30.806915 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c8fcd8bcc9d19d7465aa82957f7b2a66d3db5690c79ca44f8442d2cf9a57ab7\": container with ID starting with 2c8fcd8bcc9d19d7465aa82957f7b2a66d3db5690c79ca44f8442d2cf9a57ab7 not found: ID does not exist" containerID="2c8fcd8bcc9d19d7465aa82957f7b2a66d3db5690c79ca44f8442d2cf9a57ab7" Nov 02 18:29:30.808417 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.806983 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8fcd8bcc9d19d7465aa82957f7b2a66d3db5690c79ca44f8442d2cf9a57ab7"} err="failed to get container status \"2c8fcd8bcc9d19d7465aa82957f7b2a66d3db5690c79ca44f8442d2cf9a57ab7\": rpc error: code = NotFound desc = could not find container \"2c8fcd8bcc9d19d7465aa82957f7b2a66d3db5690c79ca44f8442d2cf9a57ab7\": container with ID starting with 2c8fcd8bcc9d19d7465aa82957f7b2a66d3db5690c79ca44f8442d2cf9a57ab7 not found: ID does not exist" Nov 02 18:29:30.808417 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.807010 5549 scope.go:122] "RemoveContainer" containerID="d2eb79253c87269fc3bb921f14602077d89fedc597159de40def37e2d7a0978f" Nov 02 18:29:30.824843 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.824819 5549 scope.go:122] "RemoveContainer" containerID="d2eb79253c87269fc3bb921f14602077d89fedc597159de40def37e2d7a0978f" Nov 02 18:29:30.825540 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:29:30.825498 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2eb79253c87269fc3bb921f14602077d89fedc597159de40def37e2d7a0978f\": container with ID starting with d2eb79253c87269fc3bb921f14602077d89fedc597159de40def37e2d7a0978f not found: ID does not exist" containerID="d2eb79253c87269fc3bb921f14602077d89fedc597159de40def37e2d7a0978f" Nov 02 18:29:30.825666 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.825536 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2eb79253c87269fc3bb921f14602077d89fedc597159de40def37e2d7a0978f"} err="failed to get container status \"d2eb79253c87269fc3bb921f14602077d89fedc597159de40def37e2d7a0978f\": rpc error: code = NotFound desc = could not find container \"d2eb79253c87269fc3bb921f14602077d89fedc597159de40def37e2d7a0978f\": container with ID starting with d2eb79253c87269fc3bb921f14602077d89fedc597159de40def37e2d7a0978f not found: ID does not exist" Nov 02 18:29:30.829688 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.829660 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27"] Nov 02 18:29:30.835439 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.835417 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-327608-64955f5f6f-t8d27"] Nov 02 18:29:30.878580 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.878551 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwfkl\" (UniqueName: \"kubernetes.io/projected/9baba3f3-ce78-4167-8f82-0e24b38a5cda-kube-api-access-nwfkl\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:30.878580 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:30.878578 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9baba3f3-ce78-4167-8f82-0e24b38a5cda-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:31.094664 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:31.094625 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh"] Nov 02 18:29:31.097816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:31.097789 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-327608-895868c7f-7m2kh"] Nov 02 18:29:31.146008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:31.145965 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" podUID="766b693c-dd34-4892-9eeb-3528e08a3147" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:29:31.232786 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:31.232738 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:31.529698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:31.529653 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" podUID="e14cd218-72ea-4e71-bd6d-daad87430119" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:29:31.529856 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:31.529739 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" podUID="fd061465-fa5b-4565-a280-f546c45b9f0e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:29:31.583761 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:31.583127 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:31.784649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:31.784571 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:31.785103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:31.785081 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:31.785886 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:31.785866 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:29:32.313678 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:32.313645 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:32.321613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:32.321585 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88ee8230-6e65-4a89-b9bd-1f7100b03218" path="/var/lib/kubelet/pods/88ee8230-6e65-4a89-b9bd-1f7100b03218/volumes" Nov 02 18:29:32.322079 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:32.322064 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9baba3f3-ce78-4167-8f82-0e24b38a5cda" path="/var/lib/kubelet/pods/9baba3f3-ce78-4167-8f82-0e24b38a5cda/volumes" Nov 02 18:29:32.382894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:32.382855 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg"] Nov 02 18:29:32.488786 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:32.487464 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:33.014388 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.014238 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:33.469495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.469461 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" Nov 02 18:29:33.502597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.502558 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" Nov 02 18:29:33.552233 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.552147 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd061465-fa5b-4565-a280-f546c45b9f0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd061465-fa5b-4565-a280-f546c45b9f0e-plugins-config-volume\") pod \"fd061465-fa5b-4565-a280-f546c45b9f0e\" (UID: \"fd061465-fa5b-4565-a280-f546c45b9f0e\") " Nov 02 18:29:33.552436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.552414 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd061465-fa5b-4565-a280-f546c45b9f0e-kube-api-access-729sg\" (UniqueName: \"kubernetes.io/projected/fd061465-fa5b-4565-a280-f546c45b9f0e-kube-api-access-729sg\") pod \"fd061465-fa5b-4565-a280-f546c45b9f0e\" (UID: \"fd061465-fa5b-4565-a280-f546c45b9f0e\") " Nov 02 18:29:33.554070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.553925 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd061465-fa5b-4565-a280-f546c45b9f0e-plugins-config-volume" pod "fd061465-fa5b-4565-a280-f546c45b9f0e" (UID: "fd061465-fa5b-4565-a280-f546c45b9f0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:33.555872 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.555843 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd061465-fa5b-4565-a280-f546c45b9f0e-kube-api-access-729sg" pod "fd061465-fa5b-4565-a280-f546c45b9f0e" (UID: "fd061465-fa5b-4565-a280-f546c45b9f0e"). InnerVolumeSpecName "kube-api-access-729sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:33.562343 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.562313 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:33.654462 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.654427 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e14cd218-72ea-4e71-bd6d-daad87430119-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14cd218-72ea-4e71-bd6d-daad87430119-plugins-config-volume\") pod \"e14cd218-72ea-4e71-bd6d-daad87430119\" (UID: \"e14cd218-72ea-4e71-bd6d-daad87430119\") " Nov 02 18:29:33.654656 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.654615 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e14cd218-72ea-4e71-bd6d-daad87430119-kube-api-access-jw5n5\" (UniqueName: \"kubernetes.io/projected/e14cd218-72ea-4e71-bd6d-daad87430119-kube-api-access-jw5n5\") pod \"e14cd218-72ea-4e71-bd6d-daad87430119\" (UID: \"e14cd218-72ea-4e71-bd6d-daad87430119\") " Nov 02 18:29:33.654914 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.654882 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e14cd218-72ea-4e71-bd6d-daad87430119-plugins-config-volume" pod "e14cd218-72ea-4e71-bd6d-daad87430119" (UID: "e14cd218-72ea-4e71-bd6d-daad87430119"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:33.655337 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.655313 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd061465-fa5b-4565-a280-f546c45b9f0e-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:33.655337 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.655335 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-729sg\" (UniqueName: \"kubernetes.io/projected/fd061465-fa5b-4565-a280-f546c45b9f0e-kube-api-access-729sg\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:33.655504 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.655346 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14cd218-72ea-4e71-bd6d-daad87430119-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:33.656724 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.656699 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14cd218-72ea-4e71-bd6d-daad87430119-kube-api-access-jw5n5" pod "e14cd218-72ea-4e71-bd6d-daad87430119" (UID: "e14cd218-72ea-4e71-bd6d-daad87430119"). InnerVolumeSpecName "kube-api-access-jw5n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:33.757304 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.756572 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw5n5\" (UniqueName: \"kubernetes.io/projected/e14cd218-72ea-4e71-bd6d-daad87430119-kube-api-access-jw5n5\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:33.801528 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.801495 5549 generic.go:356] "Generic (PLEG): container finished" podID="e14cd218-72ea-4e71-bd6d-daad87430119" containerID="cc6f8a40384ca3fd6280c093d95e645b13dd2f74a1315606da71eb3d81808398" exitCode=0 Nov 02 18:29:33.801713 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.801541 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" Nov 02 18:29:33.801713 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.801575 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" event={"ID":"e14cd218-72ea-4e71-bd6d-daad87430119","Type":"ContainerDied","Data":"cc6f8a40384ca3fd6280c093d95e645b13dd2f74a1315606da71eb3d81808398"} Nov 02 18:29:33.801713 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.801632 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4" event={"ID":"e14cd218-72ea-4e71-bd6d-daad87430119","Type":"ContainerDied","Data":"5f92dd46d2cd1d0629f6213f12b39ad320602916f2c1a99fbec91ed6d8011f96"} Nov 02 18:29:33.801713 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.801661 5549 scope.go:122] "RemoveContainer" containerID="cc6f8a40384ca3fd6280c093d95e645b13dd2f74a1315606da71eb3d81808398" Nov 02 18:29:33.803276 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.803255 5549 generic.go:356] "Generic (PLEG): container finished" podID="fd061465-fa5b-4565-a280-f546c45b9f0e" containerID="01c128338df9ff573678190f5bda667e1e8df6353d8588f5f8d3a793e2c6016c" exitCode=0 Nov 02 18:29:33.804014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.803987 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" Nov 02 18:29:33.804158 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.804033 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" event={"ID":"fd061465-fa5b-4565-a280-f546c45b9f0e","Type":"ContainerDied","Data":"01c128338df9ff573678190f5bda667e1e8df6353d8588f5f8d3a793e2c6016c"} Nov 02 18:29:33.804158 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.804091 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm" event={"ID":"fd061465-fa5b-4565-a280-f546c45b9f0e","Type":"ContainerDied","Data":"2df4f47e31b13aefd9e946f285f38f8341a33e2d190e54be0be6d5e1ceba0061"} Nov 02 18:29:33.814643 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.814621 5549 scope.go:122] "RemoveContainer" containerID="cc6f8a40384ca3fd6280c093d95e645b13dd2f74a1315606da71eb3d81808398" Nov 02 18:29:33.815026 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:29:33.815000 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc6f8a40384ca3fd6280c093d95e645b13dd2f74a1315606da71eb3d81808398\": container with ID starting with cc6f8a40384ca3fd6280c093d95e645b13dd2f74a1315606da71eb3d81808398 not found: ID does not exist" containerID="cc6f8a40384ca3fd6280c093d95e645b13dd2f74a1315606da71eb3d81808398" Nov 02 18:29:33.815146 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.815034 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc6f8a40384ca3fd6280c093d95e645b13dd2f74a1315606da71eb3d81808398"} err="failed to get container status \"cc6f8a40384ca3fd6280c093d95e645b13dd2f74a1315606da71eb3d81808398\": rpc error: code = NotFound desc = could not find container \"cc6f8a40384ca3fd6280c093d95e645b13dd2f74a1315606da71eb3d81808398\": container with ID starting with cc6f8a40384ca3fd6280c093d95e645b13dd2f74a1315606da71eb3d81808398 not found: ID does not exist" Nov 02 18:29:33.815146 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.815056 5549 scope.go:122] "RemoveContainer" containerID="01c128338df9ff573678190f5bda667e1e8df6353d8588f5f8d3a793e2c6016c" Nov 02 18:29:33.827357 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.827329 5549 scope.go:122] "RemoveContainer" containerID="01c128338df9ff573678190f5bda667e1e8df6353d8588f5f8d3a793e2c6016c" Nov 02 18:29:33.827673 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:29:33.827651 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c128338df9ff573678190f5bda667e1e8df6353d8588f5f8d3a793e2c6016c\": container with ID starting with 01c128338df9ff573678190f5bda667e1e8df6353d8588f5f8d3a793e2c6016c not found: ID does not exist" containerID="01c128338df9ff573678190f5bda667e1e8df6353d8588f5f8d3a793e2c6016c" Nov 02 18:29:33.827745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.827684 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c128338df9ff573678190f5bda667e1e8df6353d8588f5f8d3a793e2c6016c"} err="failed to get container status \"01c128338df9ff573678190f5bda667e1e8df6353d8588f5f8d3a793e2c6016c\": rpc error: code = NotFound desc = could not find container \"01c128338df9ff573678190f5bda667e1e8df6353d8588f5f8d3a793e2c6016c\": container with ID starting with 01c128338df9ff573678190f5bda667e1e8df6353d8588f5f8d3a793e2c6016c not found: ID does not exist" Nov 02 18:29:33.833916 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.833891 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm"] Nov 02 18:29:33.838513 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.838488 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2961d5-98554786c-b5hmm"] Nov 02 18:29:33.848758 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.848722 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4"] Nov 02 18:29:33.851683 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:33.851660 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2961d5-6b6f67c495-5kbn4"] Nov 02 18:29:34.007500 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:34.003973 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:34.322331 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:34.322296 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e14cd218-72ea-4e71-bd6d-daad87430119" path="/var/lib/kubelet/pods/e14cd218-72ea-4e71-bd6d-daad87430119/volumes" Nov 02 18:29:34.323472 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:34.323440 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd061465-fa5b-4565-a280-f546c45b9f0e" path="/var/lib/kubelet/pods/fd061465-fa5b-4565-a280-f546c45b9f0e/volumes" Nov 02 18:29:34.627716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:34.627674 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:34.960315 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:34.960257 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:35.128128 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:35.128091 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:35.163461 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:35.163415 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f"] Nov 02 18:29:35.177343 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:35.177312 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth"] Nov 02 18:29:35.296480 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:35.296384 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:35.454022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:35.445899 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:35.692160 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:35.692073 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:35.909565 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:35.909528 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:36.056735 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:36.056703 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:36.824317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:36.824271 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:36.961558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:36.961522 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:37.376799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:37.376765 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:37.628720 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:37.628608 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:38.088866 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:38.088827 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn"] Nov 02 18:29:38.206973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:38.200911 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:38.460927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:38.459497 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:39.202275 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:39.202238 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:39.933575 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:39.933536 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:40.163965 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:40.163918 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" podUID="3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0" containerName="payload-processing" containerID="cri-o://75cf09299eed3f253d2b5e94a79365624903e38044b52689a4cf57ec4fc0368e" gracePeriod=25 Nov 02 18:29:40.177617 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:40.177578 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" podUID="060e829c-0a72-48a9-b9db-13db919d875f" containerName="payload-pre-processing" containerID="cri-o://a068c66873b39a611762edeeb9ed2ffda2d3a620fca0303fd24466d3e5cb89b9" gracePeriod=25 Nov 02 18:29:40.712449 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:40.710247 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:41.146859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:41.146778 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" podUID="766b693c-dd34-4892-9eeb-3528e08a3147" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:29:41.594745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:41.594711 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:42.097337 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:42.097305 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:29:42.288080 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:42.288041 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:43.141007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:43.140917 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:43.777598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:43.777560 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" podUID="3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:29:43.777598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:43.777579 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" podUID="060e829c-0a72-48a9-b9db-13db919d875f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:29:44.049505 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.049423 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:44.205974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.205447 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:44.299702 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.299627 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" Nov 02 18:29:44.461818 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.461785 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/766b693c-dd34-4892-9eeb-3528e08a3147-kube-api-access-rq6gm\" (UniqueName: \"kubernetes.io/projected/766b693c-dd34-4892-9eeb-3528e08a3147-kube-api-access-rq6gm\") pod \"766b693c-dd34-4892-9eeb-3528e08a3147\" (UID: \"766b693c-dd34-4892-9eeb-3528e08a3147\") " Nov 02 18:29:44.462016 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.461853 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/766b693c-dd34-4892-9eeb-3528e08a3147-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766b693c-dd34-4892-9eeb-3528e08a3147-plugins-config-volume\") pod \"766b693c-dd34-4892-9eeb-3528e08a3147\" (UID: \"766b693c-dd34-4892-9eeb-3528e08a3147\") " Nov 02 18:29:44.462325 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.462299 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766b693c-dd34-4892-9eeb-3528e08a3147-plugins-config-volume" pod "766b693c-dd34-4892-9eeb-3528e08a3147" (UID: "766b693c-dd34-4892-9eeb-3528e08a3147"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:44.463126 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.463101 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766b693c-dd34-4892-9eeb-3528e08a3147-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.464359 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.464336 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766b693c-dd34-4892-9eeb-3528e08a3147-kube-api-access-rq6gm" pod "766b693c-dd34-4892-9eeb-3528e08a3147" (UID: "766b693c-dd34-4892-9eeb-3528e08a3147"). InnerVolumeSpecName "kube-api-access-rq6gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:44.565008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.564892 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq6gm\" (UniqueName: \"kubernetes.io/projected/766b693c-dd34-4892-9eeb-3528e08a3147-kube-api-access-rq6gm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.631984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.631908 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g"] Nov 02 18:29:44.633160 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.633127 5549 cpu_manager.go:414] "containerMap: removing container" podUID="88ee8230-6e65-4a89-b9bd-1f7100b03218" containerName="payload-processing" Nov 02 18:29:44.633160 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.633158 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88ee8230-6e65-4a89-b9bd-1f7100b03218" containerName="payload-processing" Nov 02 18:29:44.633371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.633178 5549 cpu_manager.go:414] "containerMap: removing container" podUID="fd061465-fa5b-4565-a280-f546c45b9f0e" containerName="payload-processing" Nov 02 18:29:44.633371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.633187 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd061465-fa5b-4565-a280-f546c45b9f0e" containerName="payload-processing" Nov 02 18:29:44.633371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.633210 5549 cpu_manager.go:414] "containerMap: removing container" podUID="e14cd218-72ea-4e71-bd6d-daad87430119" containerName="payload-pre-processing" Nov 02 18:29:44.633371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.633217 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e14cd218-72ea-4e71-bd6d-daad87430119" containerName="payload-pre-processing" Nov 02 18:29:44.633371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.633238 5549 cpu_manager.go:414] "containerMap: removing container" podUID="766b693c-dd34-4892-9eeb-3528e08a3147" containerName="payload-processing" Nov 02 18:29:44.633371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.633245 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766b693c-dd34-4892-9eeb-3528e08a3147" containerName="payload-processing" Nov 02 18:29:44.633371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.633293 5549 cpu_manager.go:414] "containerMap: removing container" podUID="9baba3f3-ce78-4167-8f82-0e24b38a5cda" containerName="payload-pre-processing" Nov 02 18:29:44.633371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.633300 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9baba3f3-ce78-4167-8f82-0e24b38a5cda" containerName="payload-pre-processing" Nov 02 18:29:44.633677 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.633471 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="e14cd218-72ea-4e71-bd6d-daad87430119" containerName="payload-pre-processing" Nov 02 18:29:44.633677 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.633490 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="766b693c-dd34-4892-9eeb-3528e08a3147" containerName="payload-processing" Nov 02 18:29:44.633677 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.633508 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="9baba3f3-ce78-4167-8f82-0e24b38a5cda" containerName="payload-pre-processing" Nov 02 18:29:44.633677 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.633525 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="88ee8230-6e65-4a89-b9bd-1f7100b03218" containerName="payload-processing" Nov 02 18:29:44.633677 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.633535 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd061465-fa5b-4565-a280-f546c45b9f0e" containerName="payload-processing" Nov 02 18:29:44.639866 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.639429 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.650000 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.644272 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g"] Nov 02 18:29:44.650000 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.644317 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-1e2281\"/\"sub-test-model-1e2281-kserve-self-signed-certs\"" Nov 02 18:29:44.650376 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.650246 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:44.666284 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.666248 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-dshm\") pod \"sub-test-model-1e2281-kserve-546ccc6679-svb5g\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.666586 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.666371 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-tmp-dir\") pod \"sub-test-model-1e2281-kserve-546ccc6679-svb5g\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.666586 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.666435 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-model-cache\") pod \"sub-test-model-1e2281-kserve-546ccc6679-svb5g\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.666586 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.666527 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-home\") pod \"sub-test-model-1e2281-kserve-546ccc6679-svb5g\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.666586 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.666561 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6ddfb19-4075-43de-9fb7-395181b36a25-tls-certs\") pod \"sub-test-model-1e2281-kserve-546ccc6679-svb5g\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.667122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.666697 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpx6q\" (UniqueName: \"kubernetes.io/projected/e6ddfb19-4075-43de-9fb7-395181b36a25-kube-api-access-tpx6q\") pod \"sub-test-model-1e2281-kserve-546ccc6679-svb5g\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.768302 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.768262 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpx6q\" (UniqueName: \"kubernetes.io/projected/e6ddfb19-4075-43de-9fb7-395181b36a25-kube-api-access-tpx6q\") pod \"sub-test-model-1e2281-kserve-546ccc6679-svb5g\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.768502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.768391 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-dshm\") pod \"sub-test-model-1e2281-kserve-546ccc6679-svb5g\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.768502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.768495 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-tmp-dir\") pod \"sub-test-model-1e2281-kserve-546ccc6679-svb5g\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.768639 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.768554 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-model-cache\") pod \"sub-test-model-1e2281-kserve-546ccc6679-svb5g\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.769007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.768650 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-home\") pod \"sub-test-model-1e2281-kserve-546ccc6679-svb5g\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.769007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.768680 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6ddfb19-4075-43de-9fb7-395181b36a25-tls-certs\") pod \"sub-test-model-1e2281-kserve-546ccc6679-svb5g\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.769007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.768875 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-tmp-dir\") pod \"sub-test-model-1e2281-kserve-546ccc6679-svb5g\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.769007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.768912 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-model-cache\") pod \"sub-test-model-1e2281-kserve-546ccc6679-svb5g\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.769007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.768966 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-home\") pod \"sub-test-model-1e2281-kserve-546ccc6679-svb5g\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.770980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.770926 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-dshm\") pod \"sub-test-model-1e2281-kserve-546ccc6679-svb5g\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.771378 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.771359 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6ddfb19-4075-43de-9fb7-395181b36a25-tls-certs\") pod \"sub-test-model-1e2281-kserve-546ccc6679-svb5g\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.777510 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.777484 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpx6q\" (UniqueName: \"kubernetes.io/projected/e6ddfb19-4075-43de-9fb7-395181b36a25-kube-api-access-tpx6q\") pod \"sub-test-model-1e2281-kserve-546ccc6679-svb5g\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.806035 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.805998 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:44.970975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.964677 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:44.977801 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.973381 5549 generic.go:356] "Generic (PLEG): container finished" podID="766b693c-dd34-4892-9eeb-3528e08a3147" containerID="e4a7ba3036f8bebbbf8f6efae196aab43c58866bc33d4584b0c246ad16abb892" exitCode=0 Nov 02 18:29:44.977801 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.973442 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" event={"ID":"766b693c-dd34-4892-9eeb-3528e08a3147","Type":"ContainerDied","Data":"e4a7ba3036f8bebbbf8f6efae196aab43c58866bc33d4584b0c246ad16abb892"} Nov 02 18:29:44.977801 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.973492 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" event={"ID":"766b693c-dd34-4892-9eeb-3528e08a3147","Type":"ContainerDied","Data":"e5cf3b25308f2cf8df6b73f9b28a9fdca83ed97ebf7c9747d73e8f9f3c8a89ec"} Nov 02 18:29:44.977801 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.973520 5549 scope.go:122] "RemoveContainer" containerID="e4a7ba3036f8bebbbf8f6efae196aab43c58866bc33d4584b0c246ad16abb892" Nov 02 18:29:44.977801 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:44.973683 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx" Nov 02 18:29:45.025810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:45.025125 5549 scope.go:122] "RemoveContainer" containerID="e4a7ba3036f8bebbbf8f6efae196aab43c58866bc33d4584b0c246ad16abb892" Nov 02 18:29:45.026297 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:29:45.026269 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a7ba3036f8bebbbf8f6efae196aab43c58866bc33d4584b0c246ad16abb892\": container with ID starting with e4a7ba3036f8bebbbf8f6efae196aab43c58866bc33d4584b0c246ad16abb892 not found: ID does not exist" containerID="e4a7ba3036f8bebbbf8f6efae196aab43c58866bc33d4584b0c246ad16abb892" Nov 02 18:29:45.026419 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:45.026303 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a7ba3036f8bebbbf8f6efae196aab43c58866bc33d4584b0c246ad16abb892"} err="failed to get container status \"e4a7ba3036f8bebbbf8f6efae196aab43c58866bc33d4584b0c246ad16abb892\": rpc error: code = NotFound desc = could not find container \"e4a7ba3036f8bebbbf8f6efae196aab43c58866bc33d4584b0c246ad16abb892\": container with ID starting with e4a7ba3036f8bebbbf8f6efae196aab43c58866bc33d4584b0c246ad16abb892 not found: ID does not exist" Nov 02 18:29:45.045162 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:45.045108 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx"] Nov 02 18:29:45.051876 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:45.051633 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a8e574-bfb58f65f-ktqdx"] Nov 02 18:29:45.135252 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:45.135220 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:45.265284 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:45.265252 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g"] Nov 02 18:29:45.266212 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:29:45.266185 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6ddfb19_4075_43de_9fb7_395181b36a25.slice/crio-b0f2655c570c3097401695e7d7d8a28780fe804eec857094c9340da2a441e985 WatchSource:0}: Error finding container b0f2655c570c3097401695e7d7d8a28780fe804eec857094c9340da2a441e985: Status 404 returned error can't find the container with id b0f2655c570c3097401695e7d7d8a28780fe804eec857094c9340da2a441e985 Nov 02 18:29:45.278777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:45.278753 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:45.504100 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:45.504061 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g"] Nov 02 18:29:45.504348 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:45.504323 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" podUID="7669f8e2-f625-4276-9726-d260732ffa7e" containerName="istio-proxy" containerID="cri-o://4517d8051e757f479ac49d66444b0fd5c9ce4d7dec3c5f86ad0a84cf84824ff5" gracePeriod=30 Nov 02 18:29:45.585021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:45.584915 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:45.981920 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:45.981881 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" event={"ID":"e6ddfb19-4075-43de-9fb7-395181b36a25","Type":"ContainerStarted","Data":"373e3969e391cff92c6c52d7523ffecf0aa3ea643c404b5cb5cbe0e5a363744d"} Nov 02 18:29:45.982122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:45.981925 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" event={"ID":"e6ddfb19-4075-43de-9fb7-395181b36a25","Type":"ContainerStarted","Data":"b0f2655c570c3097401695e7d7d8a28780fe804eec857094c9340da2a441e985"} Nov 02 18:29:46.000799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:46.000746 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" podStartSLOduration=2.00073194 podStartE2EDuration="2.00073194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:29:45.997027239 +0000 UTC m=+3357.785532767" watchObservedRunningTime="2026-11-02 18:29:46.00073194 +0000 UTC m=+3357.789237519" Nov 02 18:29:46.180774 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:46.180732 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:46.321682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:46.321582 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="766b693c-dd34-4892-9eeb-3528e08a3147" path="/var/lib/kubelet/pods/766b693c-dd34-4892-9eeb-3528e08a3147/volumes" Nov 02 18:29:46.323190 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:46.323158 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:46.966123 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:46.966090 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:47.104399 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.104366 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:47.411299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.411209 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" podUID="321897e4-9071-443e-a688-8c2aead4d6df" containerName="main" containerID="cri-o://e7b5673d23e63e7981b42c310309db7f2a1c4db4036ffcc9800e819db901c1ad" gracePeriod=45 Nov 02 18:29:47.593089 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:29:47.593008 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:29:47.697501 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.697467 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:29:47.808257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.808221 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321897e4-9071-443e-a688-8c2aead4d6df-kube-api-access-kdnjj\" (UniqueName: \"kubernetes.io/projected/321897e4-9071-443e-a688-8c2aead4d6df-kube-api-access-kdnjj\") pod \"321897e4-9071-443e-a688-8c2aead4d6df\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " Nov 02 18:29:47.808994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.808459 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-model-cache\") pod \"321897e4-9071-443e-a688-8c2aead4d6df\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " Nov 02 18:29:47.808994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.808542 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-tmp-dir\") pod \"321897e4-9071-443e-a688-8c2aead4d6df\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " Nov 02 18:29:47.808994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.808576 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/321897e4-9071-443e-a688-8c2aead4d6df-tls-certs\" (UniqueName: \"kubernetes.io/secret/321897e4-9071-443e-a688-8c2aead4d6df-tls-certs\") pod \"321897e4-9071-443e-a688-8c2aead4d6df\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " Nov 02 18:29:47.808994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.808653 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-home\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-home\") pod \"321897e4-9071-443e-a688-8c2aead4d6df\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " Nov 02 18:29:47.808994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.808715 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-dshm\") pod \"321897e4-9071-443e-a688-8c2aead4d6df\" (UID: \"321897e4-9071-443e-a688-8c2aead4d6df\") " Nov 02 18:29:47.810985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.810924 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-tmp-dir" pod "321897e4-9071-443e-a688-8c2aead4d6df" (UID: "321897e4-9071-443e-a688-8c2aead4d6df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:47.811240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.811212 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-model-cache" pod "321897e4-9071-443e-a688-8c2aead4d6df" (UID: "321897e4-9071-443e-a688-8c2aead4d6df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:47.811473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.811448 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-home" pod "321897e4-9071-443e-a688-8c2aead4d6df" (UID: "321897e4-9071-443e-a688-8c2aead4d6df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:47.820945 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.820912 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-dshm" pod "321897e4-9071-443e-a688-8c2aead4d6df" (UID: "321897e4-9071-443e-a688-8c2aead4d6df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:47.822787 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.822757 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/321897e4-9071-443e-a688-8c2aead4d6df-tls-certs" pod "321897e4-9071-443e-a688-8c2aead4d6df" (UID: "321897e4-9071-443e-a688-8c2aead4d6df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:47.822885 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.822784 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321897e4-9071-443e-a688-8c2aead4d6df-kube-api-access-kdnjj" pod "321897e4-9071-443e-a688-8c2aead4d6df" (UID: "321897e4-9071-443e-a688-8c2aead4d6df"). InnerVolumeSpecName "kube-api-access-kdnjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:47.910877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.910747 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:47.910877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.910790 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:47.910877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.910806 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/321897e4-9071-443e-a688-8c2aead4d6df-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:47.910877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.910819 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:47.910877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.910832 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/321897e4-9071-443e-a688-8c2aead4d6df-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:47.910877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.910847 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdnjj\" (UniqueName: \"kubernetes.io/projected/321897e4-9071-443e-a688-8c2aead4d6df-kube-api-access-kdnjj\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:47.933101 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.924771 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:47.999516 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.999483 5549 generic.go:356] "Generic (PLEG): container finished" podID="321897e4-9071-443e-a688-8c2aead4d6df" containerID="e7b5673d23e63e7981b42c310309db7f2a1c4db4036ffcc9800e819db901c1ad" exitCode=0 Nov 02 18:29:47.999692 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.999547 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" Nov 02 18:29:47.999692 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.999566 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" event={"ID":"321897e4-9071-443e-a688-8c2aead4d6df","Type":"ContainerDied","Data":"e7b5673d23e63e7981b42c310309db7f2a1c4db4036ffcc9800e819db901c1ad"} Nov 02 18:29:47.999692 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.999613 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg" event={"ID":"321897e4-9071-443e-a688-8c2aead4d6df","Type":"ContainerDied","Data":"75dbc888a56e84846f076a854faae26d7fdbcacf1538aff5193b4c3ca4134ba0"} Nov 02 18:29:47.999692 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:47.999635 5549 scope.go:122] "RemoveContainer" containerID="e7b5673d23e63e7981b42c310309db7f2a1c4db4036ffcc9800e819db901c1ad" Nov 02 18:29:48.010498 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:48.010482 5549 scope.go:122] "RemoveContainer" containerID="e7b5673d23e63e7981b42c310309db7f2a1c4db4036ffcc9800e819db901c1ad" Nov 02 18:29:48.010748 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:29:48.010728 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b5673d23e63e7981b42c310309db7f2a1c4db4036ffcc9800e819db901c1ad\": container with ID starting with e7b5673d23e63e7981b42c310309db7f2a1c4db4036ffcc9800e819db901c1ad not found: ID does not exist" containerID="e7b5673d23e63e7981b42c310309db7f2a1c4db4036ffcc9800e819db901c1ad" Nov 02 18:29:48.010801 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:48.010756 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b5673d23e63e7981b42c310309db7f2a1c4db4036ffcc9800e819db901c1ad"} err="failed to get container status \"e7b5673d23e63e7981b42c310309db7f2a1c4db4036ffcc9800e819db901c1ad\": rpc error: code = NotFound desc = could not find container \"e7b5673d23e63e7981b42c310309db7f2a1c4db4036ffcc9800e819db901c1ad\": container with ID starting with e7b5673d23e63e7981b42c310309db7f2a1c4db4036ffcc9800e819db901c1ad not found: ID does not exist" Nov 02 18:29:48.025750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:48.025726 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg"] Nov 02 18:29:48.028653 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:48.028631 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1e2281/rate-test-model-1e2281-kserve-5f6b76d56f-mg2tg"] Nov 02 18:29:48.322188 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:48.322157 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321897e4-9071-443e-a688-8c2aead4d6df" path="/var/lib/kubelet/pods/321897e4-9071-443e-a688-8c2aead4d6df/volumes" Nov 02 18:29:48.603537 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:48.597769 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:49.234749 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:49.234708 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:49.751719 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:49.751681 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj"] Nov 02 18:29:49.752298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:49.751933 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" podUID="2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2" containerName="maas-api" containerID="cri-o://43a6ad10bc23b1232d936448a48df8873e6f47dc326a713d69319564cd374350" gracePeriod=30 Nov 02 18:29:49.841041 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:49.840992 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:49.937969 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:49.929752 5549 patch_prober.go:29] interesting pod/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:29:49.937969 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:49.929837 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" podUID="7669f8e2-f625-4276-9726-d260732ffa7e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:29:50.106849 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.106751 5549 generic.go:356] "Generic (PLEG): container finished" podID="2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2" containerID="43a6ad10bc23b1232d936448a48df8873e6f47dc326a713d69319564cd374350" exitCode=0 Nov 02 18:29:50.107167 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.106809 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" event={"ID":"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2","Type":"ContainerDied","Data":"43a6ad10bc23b1232d936448a48df8873e6f47dc326a713d69319564cd374350"} Nov 02 18:29:50.156859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.156829 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 18:29:50.282656 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.282624 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:50.308540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.307537 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-metrics-certs\") pod \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\" (UID: \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\") " Nov 02 18:29:50.308540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.307898 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-maas-api-tls\") pod \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\" (UID: \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\") " Nov 02 18:29:50.308540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.307985 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-openshift-service-ca\") pod \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\" (UID: \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\") " Nov 02 18:29:50.308540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.308045 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-kube-api-access-5z8v2\" (UniqueName: \"kubernetes.io/projected/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-kube-api-access-5z8v2\") pod \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\" (UID: \"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2\") " Nov 02 18:29:50.314696 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.314649 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-openshift-service-ca" pod "2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2" (UID: "2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:50.314844 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.314772 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-kube-api-access-5z8v2" pod "2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2" (UID: "2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2"). InnerVolumeSpecName "kube-api-access-5z8v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:50.322413 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.322370 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-metrics-certs" pod "2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2" (UID: "2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:50.322413 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.322405 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-maas-api-tls" pod "2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2" (UID: "2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:50.410492 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.410391 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:50.410492 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.410428 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:50.410492 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.410446 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z8v2\" (UniqueName: \"kubernetes.io/projected/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-kube-api-access-5z8v2\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:50.410492 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.410461 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:50.748426 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.748401 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:29:50.919375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.919337 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7669f8e2-f625-4276-9726-d260732ffa7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7669f8e2-f625-4276-9726-d260732ffa7e-istiod-ca-cert\") pod \"7669f8e2-f625-4276-9726-d260732ffa7e\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " Nov 02 18:29:50.919799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.919416 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-workload-socket\") pod \"7669f8e2-f625-4276-9726-d260732ffa7e\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " Nov 02 18:29:50.919799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.919548 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7669f8e2-f625-4276-9726-d260732ffa7e-kube-api-access-gqgbh\" (UniqueName: \"kubernetes.io/projected/7669f8e2-f625-4276-9726-d260732ffa7e-kube-api-access-gqgbh\") pod \"7669f8e2-f625-4276-9726-d260732ffa7e\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " Nov 02 18:29:50.919799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.919604 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-istio-envoy\") pod \"7669f8e2-f625-4276-9726-d260732ffa7e\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " Nov 02 18:29:50.919799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.919715 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7669f8e2-f625-4276-9726-d260732ffa7e-istio-token\" (UniqueName: \"kubernetes.io/projected/7669f8e2-f625-4276-9726-d260732ffa7e-istio-token\") pod \"7669f8e2-f625-4276-9726-d260732ffa7e\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " Nov 02 18:29:50.919799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.919754 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-workload-socket" pod "7669f8e2-f625-4276-9726-d260732ffa7e" (UID: "7669f8e2-f625-4276-9726-d260732ffa7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:50.919799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.919783 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7669f8e2-f625-4276-9726-d260732ffa7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7669f8e2-f625-4276-9726-d260732ffa7e-istio-podinfo\") pod \"7669f8e2-f625-4276-9726-d260732ffa7e\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " Nov 02 18:29:50.920189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.919827 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-workload-certs\") pod \"7669f8e2-f625-4276-9726-d260732ffa7e\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " Nov 02 18:29:50.920189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.919783 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7669f8e2-f625-4276-9726-d260732ffa7e-istiod-ca-cert" pod "7669f8e2-f625-4276-9726-d260732ffa7e" (UID: "7669f8e2-f625-4276-9726-d260732ffa7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:50.920189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.919899 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-credential-socket\") pod \"7669f8e2-f625-4276-9726-d260732ffa7e\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " Nov 02 18:29:50.920189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.919921 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-istio-data\") pod \"7669f8e2-f625-4276-9726-d260732ffa7e\" (UID: \"7669f8e2-f625-4276-9726-d260732ffa7e\") " Nov 02 18:29:50.920427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.920190 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-workload-certs" pod "7669f8e2-f625-4276-9726-d260732ffa7e" (UID: "7669f8e2-f625-4276-9726-d260732ffa7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:50.920427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.920313 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-credential-socket" pod "7669f8e2-f625-4276-9726-d260732ffa7e" (UID: "7669f8e2-f625-4276-9726-d260732ffa7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:50.920427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.920388 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-istio-data" pod "7669f8e2-f625-4276-9726-d260732ffa7e" (UID: "7669f8e2-f625-4276-9726-d260732ffa7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:50.920675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.920654 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:50.920753 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.920682 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:50.920753 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.920698 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:50.920753 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.920713 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7669f8e2-f625-4276-9726-d260732ffa7e-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:50.920753 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.920727 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:50.922816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.922704 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7669f8e2-f625-4276-9726-d260732ffa7e-istio-podinfo" pod "7669f8e2-f625-4276-9726-d260732ffa7e" (UID: "7669f8e2-f625-4276-9726-d260732ffa7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:29:50.922816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.922724 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-istio-envoy" pod "7669f8e2-f625-4276-9726-d260732ffa7e" (UID: "7669f8e2-f625-4276-9726-d260732ffa7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:50.923077 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.922827 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7669f8e2-f625-4276-9726-d260732ffa7e-kube-api-access-gqgbh" pod "7669f8e2-f625-4276-9726-d260732ffa7e" (UID: "7669f8e2-f625-4276-9726-d260732ffa7e"). InnerVolumeSpecName "kube-api-access-gqgbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:50.923987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:50.923927 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7669f8e2-f625-4276-9726-d260732ffa7e-istio-token" pod "7669f8e2-f625-4276-9726-d260732ffa7e" (UID: "7669f8e2-f625-4276-9726-d260732ffa7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:51.012975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.012594 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:51.022773 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.022240 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7669f8e2-f625-4276-9726-d260732ffa7e-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:51.022773 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.022283 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7669f8e2-f625-4276-9726-d260732ffa7e-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:51.022773 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.022298 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7669f8e2-f625-4276-9726-d260732ffa7e-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:51.022773 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.022316 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqgbh\" (UniqueName: \"kubernetes.io/projected/7669f8e2-f625-4276-9726-d260732ffa7e-kube-api-access-gqgbh\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:51.116533 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.116496 5549 generic.go:356] "Generic (PLEG): container finished" podID="7669f8e2-f625-4276-9726-d260732ffa7e" containerID="4517d8051e757f479ac49d66444b0fd5c9ce4d7dec3c5f86ad0a84cf84824ff5" exitCode=0 Nov 02 18:29:51.116701 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.116546 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" Nov 02 18:29:51.116701 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.116579 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" event={"ID":"7669f8e2-f625-4276-9726-d260732ffa7e","Type":"ContainerDied","Data":"4517d8051e757f479ac49d66444b0fd5c9ce4d7dec3c5f86ad0a84cf84824ff5"} Nov 02 18:29:51.116701 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.116637 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g" event={"ID":"7669f8e2-f625-4276-9726-d260732ffa7e","Type":"ContainerDied","Data":"fda3921074babf1f88c494f7399cbf8bdf026a3c524aa8497168427b30286a3c"} Nov 02 18:29:51.116701 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.116663 5549 scope.go:122] "RemoveContainer" containerID="4517d8051e757f479ac49d66444b0fd5c9ce4d7dec3c5f86ad0a84cf84824ff5" Nov 02 18:29:51.118338 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.118310 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" Nov 02 18:29:51.118338 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.118324 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj" event={"ID":"2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2","Type":"ContainerDied","Data":"4465386e31b59fe7bf9068ce5e1e1bfd1b230ef612a3dd052517d33569e3559b"} Nov 02 18:29:51.128624 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.128602 5549 scope.go:122] "RemoveContainer" containerID="a9b74124eb89d95ab2ee6f2cc22707845716c1f2e8ab7abad140ea0ccf9bc514" Nov 02 18:29:51.138565 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.138543 5549 scope.go:122] "RemoveContainer" containerID="4517d8051e757f479ac49d66444b0fd5c9ce4d7dec3c5f86ad0a84cf84824ff5" Nov 02 18:29:51.138846 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:29:51.138818 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4517d8051e757f479ac49d66444b0fd5c9ce4d7dec3c5f86ad0a84cf84824ff5\": container with ID starting with 4517d8051e757f479ac49d66444b0fd5c9ce4d7dec3c5f86ad0a84cf84824ff5 not found: ID does not exist" containerID="4517d8051e757f479ac49d66444b0fd5c9ce4d7dec3c5f86ad0a84cf84824ff5" Nov 02 18:29:51.138975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.138848 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4517d8051e757f479ac49d66444b0fd5c9ce4d7dec3c5f86ad0a84cf84824ff5"} err="failed to get container status \"4517d8051e757f479ac49d66444b0fd5c9ce4d7dec3c5f86ad0a84cf84824ff5\": rpc error: code = NotFound desc = could not find container \"4517d8051e757f479ac49d66444b0fd5c9ce4d7dec3c5f86ad0a84cf84824ff5\": container with ID starting with 4517d8051e757f479ac49d66444b0fd5c9ce4d7dec3c5f86ad0a84cf84824ff5 not found: ID does not exist" Nov 02 18:29:51.138975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.138866 5549 scope.go:122] "RemoveContainer" containerID="a9b74124eb89d95ab2ee6f2cc22707845716c1f2e8ab7abad140ea0ccf9bc514" Nov 02 18:29:51.139140 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:29:51.139123 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b74124eb89d95ab2ee6f2cc22707845716c1f2e8ab7abad140ea0ccf9bc514\": container with ID starting with a9b74124eb89d95ab2ee6f2cc22707845716c1f2e8ab7abad140ea0ccf9bc514 not found: ID does not exist" containerID="a9b74124eb89d95ab2ee6f2cc22707845716c1f2e8ab7abad140ea0ccf9bc514" Nov 02 18:29:51.139186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.139144 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b74124eb89d95ab2ee6f2cc22707845716c1f2e8ab7abad140ea0ccf9bc514"} err="failed to get container status \"a9b74124eb89d95ab2ee6f2cc22707845716c1f2e8ab7abad140ea0ccf9bc514\": rpc error: code = NotFound desc = could not find container \"a9b74124eb89d95ab2ee6f2cc22707845716c1f2e8ab7abad140ea0ccf9bc514\": container with ID starting with a9b74124eb89d95ab2ee6f2cc22707845716c1f2e8ab7abad140ea0ccf9bc514 not found: ID does not exist" Nov 02 18:29:51.139186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.139157 5549 scope.go:122] "RemoveContainer" containerID="43a6ad10bc23b1232d936448a48df8873e6f47dc326a713d69319564cd374350" Nov 02 18:29:51.140265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.140244 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj"] Nov 02 18:29:51.145067 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.145046 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-24cfb6-55f9fd8bcc-vlvcj"] Nov 02 18:29:51.155268 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.155236 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g"] Nov 02 18:29:51.161020 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.160989 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a8e574-openshift-default-658b84d485-kht8g"] Nov 02 18:29:51.571279 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.571233 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:51.712562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.712525 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:51.844038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:51.842252 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:52.325515 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:52.325473 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2" path="/var/lib/kubelet/pods/2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2/volumes" Nov 02 18:29:52.326073 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:52.326056 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7669f8e2-f625-4276-9726-d260732ffa7e" path="/var/lib/kubelet/pods/7669f8e2-f625-4276-9726-d260732ffa7e/volumes" Nov 02 18:29:52.331820 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:52.331797 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:52.549988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:52.542644 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:52.937126 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:52.937083 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:53.121127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.121071 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" podUID="672fc5b2-2136-4d06-bf2f-3f5581500ca3" containerName="main" containerID="cri-o://88be9bebb782401d96b51603267a0fc4155f1d3336bbeae6326baccfeabd6ef6" gracePeriod=45 Nov 02 18:29:53.128843 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.128800 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:53.421723 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.421696 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:29:53.457076 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.457037 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-home\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-home\") pod \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " Nov 02 18:29:53.457345 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.457238 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/672fc5b2-2136-4d06-bf2f-3f5581500ca3-kube-api-access-fs5lf\" (UniqueName: \"kubernetes.io/projected/672fc5b2-2136-4d06-bf2f-3f5581500ca3-kube-api-access-fs5lf\") pod \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " Nov 02 18:29:53.457345 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.457284 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-tmp-dir\") pod \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " Nov 02 18:29:53.457345 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.457305 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-home" pod "672fc5b2-2136-4d06-bf2f-3f5581500ca3" (UID: "672fc5b2-2136-4d06-bf2f-3f5581500ca3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:53.457345 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.457331 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-model-cache\") pod \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " Nov 02 18:29:53.457578 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.457429 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-dshm\") pod \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " Nov 02 18:29:53.457578 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.457493 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/672fc5b2-2136-4d06-bf2f-3f5581500ca3-tls-certs\" (UniqueName: \"kubernetes.io/secret/672fc5b2-2136-4d06-bf2f-3f5581500ca3-tls-certs\") pod \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\" (UID: \"672fc5b2-2136-4d06-bf2f-3f5581500ca3\") " Nov 02 18:29:53.457578 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.457506 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-tmp-dir" pod "672fc5b2-2136-4d06-bf2f-3f5581500ca3" (UID: "672fc5b2-2136-4d06-bf2f-3f5581500ca3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:53.457740 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.457680 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-model-cache" pod "672fc5b2-2136-4d06-bf2f-3f5581500ca3" (UID: "672fc5b2-2136-4d06-bf2f-3f5581500ca3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:53.458154 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.458127 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:53.458154 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.458156 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:53.458380 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.458172 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:53.459827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.459801 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/672fc5b2-2136-4d06-bf2f-3f5581500ca3-tls-certs" pod "672fc5b2-2136-4d06-bf2f-3f5581500ca3" (UID: "672fc5b2-2136-4d06-bf2f-3f5581500ca3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:53.459935 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.459809 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-dshm" pod "672fc5b2-2136-4d06-bf2f-3f5581500ca3" (UID: "672fc5b2-2136-4d06-bf2f-3f5581500ca3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:53.460016 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.459991 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672fc5b2-2136-4d06-bf2f-3f5581500ca3-kube-api-access-fs5lf" pod "672fc5b2-2136-4d06-bf2f-3f5581500ca3" (UID: "672fc5b2-2136-4d06-bf2f-3f5581500ca3"). InnerVolumeSpecName "kube-api-access-fs5lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:53.559989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.559864 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs5lf\" (UniqueName: \"kubernetes.io/projected/672fc5b2-2136-4d06-bf2f-3f5581500ca3-kube-api-access-fs5lf\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:53.559989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.559907 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/672fc5b2-2136-4d06-bf2f-3f5581500ca3-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:53.559989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.559928 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/672fc5b2-2136-4d06-bf2f-3f5581500ca3-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:53.651048 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.651010 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:53.798109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.798061 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" podUID="060e829c-0a72-48a9-b9db-13db919d875f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:29:53.815317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:53.815204 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" podUID="3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:29:54.108135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:54.108044 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:54.149552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:54.149514 5549 generic.go:356] "Generic (PLEG): container finished" podID="672fc5b2-2136-4d06-bf2f-3f5581500ca3" containerID="88be9bebb782401d96b51603267a0fc4155f1d3336bbeae6326baccfeabd6ef6" exitCode=0 Nov 02 18:29:54.149552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:54.149553 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" Nov 02 18:29:54.149766 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:54.149568 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" event={"ID":"672fc5b2-2136-4d06-bf2f-3f5581500ca3","Type":"ContainerDied","Data":"88be9bebb782401d96b51603267a0fc4155f1d3336bbeae6326baccfeabd6ef6"} Nov 02 18:29:54.149766 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:54.149604 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn" event={"ID":"672fc5b2-2136-4d06-bf2f-3f5581500ca3","Type":"ContainerDied","Data":"7611f4558fe3608ad460ddc3e3a2bcae3c8b0399e0d5d965809163fe44c5db78"} Nov 02 18:29:54.149766 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:54.149624 5549 scope.go:122] "RemoveContainer" containerID="88be9bebb782401d96b51603267a0fc4155f1d3336bbeae6326baccfeabd6ef6" Nov 02 18:29:54.168197 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:54.167161 5549 scope.go:122] "RemoveContainer" containerID="88be9bebb782401d96b51603267a0fc4155f1d3336bbeae6326baccfeabd6ef6" Nov 02 18:29:54.168197 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:29:54.168131 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88be9bebb782401d96b51603267a0fc4155f1d3336bbeae6326baccfeabd6ef6\": container with ID starting with 88be9bebb782401d96b51603267a0fc4155f1d3336bbeae6326baccfeabd6ef6 not found: ID does not exist" containerID="88be9bebb782401d96b51603267a0fc4155f1d3336bbeae6326baccfeabd6ef6" Nov 02 18:29:54.168197 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:54.168171 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88be9bebb782401d96b51603267a0fc4155f1d3336bbeae6326baccfeabd6ef6"} err="failed to get container status \"88be9bebb782401d96b51603267a0fc4155f1d3336bbeae6326baccfeabd6ef6\": rpc error: code = NotFound desc = could not find container \"88be9bebb782401d96b51603267a0fc4155f1d3336bbeae6326baccfeabd6ef6\": container with ID starting with 88be9bebb782401d96b51603267a0fc4155f1d3336bbeae6326baccfeabd6ef6 not found: ID does not exist" Nov 02 18:29:54.194711 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:54.194672 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn"] Nov 02 18:29:54.197838 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:54.197805 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7771bf/rate-test-model-7771bf-kserve-5797944949-vltvn"] Nov 02 18:29:54.321324 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:54.321290 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="672fc5b2-2136-4d06-bf2f-3f5581500ca3" path="/var/lib/kubelet/pods/672fc5b2-2136-4d06-bf2f-3f5581500ca3/volumes" Nov 02 18:29:54.727317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:54.727188 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:54.966444 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:54.966036 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:54.979639 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:54.979561 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:55.181151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:55.181110 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:29:55.297583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:55.297413 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:55.803347 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:55.803315 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:56.626420 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:56.626368 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:57.298443 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:57.298405 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:57.932797 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:57.932756 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:58.192478 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:58.192395 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:58.335527 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:58.335493 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:58.572565 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:58.572499 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:58.718443 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:58.718403 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:59.067290 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:59.067253 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:59.431921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:59.431751 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:59.564970 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:59.564918 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:59.732639 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:59.732604 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:29:59.888099 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:29:59.888064 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:00.168473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.167823 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs"] Nov 02 18:30:00.197968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.179477 5549 cpu_manager.go:414] "containerMap: removing container" podUID="321897e4-9071-443e-a688-8c2aead4d6df" containerName="main" Nov 02 18:30:00.197968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.179509 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321897e4-9071-443e-a688-8c2aead4d6df" containerName="main" Nov 02 18:30:00.197968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.179550 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7669f8e2-f625-4276-9726-d260732ffa7e" containerName="istio-proxy" Nov 02 18:30:00.197968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.179558 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7669f8e2-f625-4276-9726-d260732ffa7e" containerName="istio-proxy" Nov 02 18:30:00.197968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.179588 5549 cpu_manager.go:414] "containerMap: removing container" podUID="2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2" containerName="maas-api" Nov 02 18:30:00.197968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.179596 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2" containerName="maas-api" Nov 02 18:30:00.197968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.179621 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7669f8e2-f625-4276-9726-d260732ffa7e" containerName="istio-proxy" Nov 02 18:30:00.197968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.179628 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7669f8e2-f625-4276-9726-d260732ffa7e" containerName="istio-proxy" Nov 02 18:30:00.197968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.179646 5549 cpu_manager.go:414] "containerMap: removing container" podUID="672fc5b2-2136-4d06-bf2f-3f5581500ca3" containerName="main" Nov 02 18:30:00.197968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.179653 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="672fc5b2-2136-4d06-bf2f-3f5581500ca3" containerName="main" Nov 02 18:30:00.197968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.179818 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7669f8e2-f625-4276-9726-d260732ffa7e" containerName="istio-proxy" Nov 02 18:30:00.197968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.179833 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d54ac06-5cc0-40c5-a413-4ab95a5bdbe2" containerName="maas-api" Nov 02 18:30:00.197968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.179846 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="672fc5b2-2136-4d06-bf2f-3f5581500ca3" containerName="main" Nov 02 18:30:00.197968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.179860 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="321897e4-9071-443e-a688-8c2aead4d6df" containerName="main" Nov 02 18:30:00.197968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.193199 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz"] Nov 02 18:30:00.197968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.194417 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs" Nov 02 18:30:00.202811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.200226 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:00.202811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.200624 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7669f8e2-f625-4276-9726-d260732ffa7e" containerName="istio-proxy" Nov 02 18:30:00.206141 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.206116 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs"] Nov 02 18:30:00.206278 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.206223 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz" Nov 02 18:30:00.208110 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.208042 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz"] Nov 02 18:30:00.273947 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.266771 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\"" Nov 02 18:30:00.287858 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.284647 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3ac9a6-4ae1-43f9-acdc-45a36e893708-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz\" (UID: \"5d3ac9a6-4ae1-43f9-acdc-45a36e893708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz" Nov 02 18:30:00.287858 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.284788 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kw7q\" (UniqueName: \"kubernetes.io/projected/5d3ac9a6-4ae1-43f9-acdc-45a36e893708-kube-api-access-8kw7q\") pod \"maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz\" (UID: \"5d3ac9a6-4ae1-43f9-acdc-45a36e893708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz" Nov 02 18:30:00.301812 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.299897 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75"] Nov 02 18:30:00.327001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.323562 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75" Nov 02 18:30:00.345705 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.344324 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75"] Nov 02 18:30:00.349649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.347518 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:00.388240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.388200 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp5wv\" (UniqueName: \"kubernetes.io/projected/f319a273-37c3-41d3-9aa1-14df989794db-kube-api-access-pp5wv\") pod \"maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75\" (UID: \"f319a273-37c3-41d3-9aa1-14df989794db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75" Nov 02 18:30:00.388434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.388373 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f319a273-37c3-41d3-9aa1-14df989794db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75\" (UID: \"f319a273-37c3-41d3-9aa1-14df989794db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75" Nov 02 18:30:00.388519 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.388478 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mz82\" (UniqueName: \"kubernetes.io/projected/ef601c1e-13de-4586-8c17-2847fa6abdbe-kube-api-access-2mz82\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs\" (UID: \"ef601c1e-13de-4586-8c17-2847fa6abdbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs" Nov 02 18:30:00.388598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.388535 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef601c1e-13de-4586-8c17-2847fa6abdbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs\" (UID: \"ef601c1e-13de-4586-8c17-2847fa6abdbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs" Nov 02 18:30:00.388660 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.388599 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3ac9a6-4ae1-43f9-acdc-45a36e893708-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz\" (UID: \"5d3ac9a6-4ae1-43f9-acdc-45a36e893708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz" Nov 02 18:30:00.388847 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.388806 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kw7q\" (UniqueName: \"kubernetes.io/projected/5d3ac9a6-4ae1-43f9-acdc-45a36e893708-kube-api-access-8kw7q\") pod \"maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz\" (UID: \"5d3ac9a6-4ae1-43f9-acdc-45a36e893708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz" Nov 02 18:30:00.389694 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.389670 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3ac9a6-4ae1-43f9-acdc-45a36e893708-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz\" (UID: \"5d3ac9a6-4ae1-43f9-acdc-45a36e893708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz" Nov 02 18:30:00.408526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.408485 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kw7q\" (UniqueName: \"kubernetes.io/projected/5d3ac9a6-4ae1-43f9-acdc-45a36e893708-kube-api-access-8kw7q\") pod \"maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz\" (UID: \"5d3ac9a6-4ae1-43f9-acdc-45a36e893708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz" Nov 02 18:30:00.490920 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.490873 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp5wv\" (UniqueName: \"kubernetes.io/projected/f319a273-37c3-41d3-9aa1-14df989794db-kube-api-access-pp5wv\") pod \"maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75\" (UID: \"f319a273-37c3-41d3-9aa1-14df989794db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75" Nov 02 18:30:00.491419 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.490993 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f319a273-37c3-41d3-9aa1-14df989794db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75\" (UID: \"f319a273-37c3-41d3-9aa1-14df989794db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75" Nov 02 18:30:00.491419 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.491048 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mz82\" (UniqueName: \"kubernetes.io/projected/ef601c1e-13de-4586-8c17-2847fa6abdbe-kube-api-access-2mz82\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs\" (UID: \"ef601c1e-13de-4586-8c17-2847fa6abdbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs" Nov 02 18:30:00.491419 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.491110 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef601c1e-13de-4586-8c17-2847fa6abdbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs\" (UID: \"ef601c1e-13de-4586-8c17-2847fa6abdbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs" Nov 02 18:30:00.492276 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.492252 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f319a273-37c3-41d3-9aa1-14df989794db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75\" (UID: \"f319a273-37c3-41d3-9aa1-14df989794db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75" Nov 02 18:30:00.492596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.492572 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef601c1e-13de-4586-8c17-2847fa6abdbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs\" (UID: \"ef601c1e-13de-4586-8c17-2847fa6abdbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs" Nov 02 18:30:00.498534 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.498488 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:00.507004 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.506945 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mz82\" (UniqueName: \"kubernetes.io/projected/ef601c1e-13de-4586-8c17-2847fa6abdbe-kube-api-access-2mz82\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs\" (UID: \"ef601c1e-13de-4586-8c17-2847fa6abdbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs" Nov 02 18:30:00.507633 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.507609 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp5wv\" (UniqueName: \"kubernetes.io/projected/f319a273-37c3-41d3-9aa1-14df989794db-kube-api-access-pp5wv\") pod \"maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75\" (UID: \"f319a273-37c3-41d3-9aa1-14df989794db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75" Nov 02 18:30:00.554437 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.554408 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" Nov 02 18:30:00.578418 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.578394 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" Nov 02 18:30:00.592984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.592909 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060e829c-0a72-48a9-b9db-13db919d875f-kube-api-access-d9j25\" (UniqueName: \"kubernetes.io/projected/060e829c-0a72-48a9-b9db-13db919d875f-kube-api-access-d9j25\") pod \"060e829c-0a72-48a9-b9db-13db919d875f\" (UID: \"060e829c-0a72-48a9-b9db-13db919d875f\") " Nov 02 18:30:00.593135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.593010 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0-plugins-config-volume\") pod \"3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0\" (UID: \"3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0\") " Nov 02 18:30:00.593135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.593082 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060e829c-0a72-48a9-b9db-13db919d875f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060e829c-0a72-48a9-b9db-13db919d875f-plugins-config-volume\") pod \"060e829c-0a72-48a9-b9db-13db919d875f\" (UID: \"060e829c-0a72-48a9-b9db-13db919d875f\") " Nov 02 18:30:00.593283 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.593212 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0-kube-api-access-rmr76\" (UniqueName: \"kubernetes.io/projected/3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0-kube-api-access-rmr76\") pod \"3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0\" (UID: \"3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0\") " Nov 02 18:30:00.593563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.593528 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060e829c-0a72-48a9-b9db-13db919d875f-plugins-config-volume" pod "060e829c-0a72-48a9-b9db-13db919d875f" (UID: "060e829c-0a72-48a9-b9db-13db919d875f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:00.593751 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.593731 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060e829c-0a72-48a9-b9db-13db919d875f-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:00.594139 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.594112 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0-plugins-config-volume" pod "3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0" (UID: "3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:00.596075 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.596033 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060e829c-0a72-48a9-b9db-13db919d875f-kube-api-access-d9j25" pod "060e829c-0a72-48a9-b9db-13db919d875f" (UID: "060e829c-0a72-48a9-b9db-13db919d875f"). InnerVolumeSpecName "kube-api-access-d9j25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:00.597144 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.597121 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0-kube-api-access-rmr76" pod "3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0" (UID: "3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0"). InnerVolumeSpecName "kube-api-access-rmr76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:00.638775 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.638637 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:00.650457 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.650184 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs" Nov 02 18:30:00.677274 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.677235 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz" Nov 02 18:30:00.692984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.677678 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75" Nov 02 18:30:00.713283 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.696175 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9j25\" (UniqueName: \"kubernetes.io/projected/060e829c-0a72-48a9-b9db-13db919d875f-kube-api-access-d9j25\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:00.713487 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.713464 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:00.713610 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:00.713597 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmr76\" (UniqueName: \"kubernetes.io/projected/3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0-kube-api-access-rmr76\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:01.023235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.023170 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs"] Nov 02 18:30:01.024266 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:30:01.024241 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef601c1e_13de_4586_8c17_2847fa6abdbe.slice/crio-19786c90606fac26ffa7ca09bdb4b6c3f8a53572685c6d727d6d7f2156748879 WatchSource:0}: Error finding container 19786c90606fac26ffa7ca09bdb4b6c3f8a53572685c6d727d6d7f2156748879: Status 404 returned error can't find the container with id 19786c90606fac26ffa7ca09bdb4b6c3f8a53572685c6d727d6d7f2156748879 Nov 02 18:30:01.027883 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:30:01.027861 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf319a273_37c3_41d3_9aa1_14df989794db.slice/crio-8e89d3f6263becae181af8a0a1d9544740882a7171a3527eea7f3c02e875d2eb WatchSource:0}: Error finding container 8e89d3f6263becae181af8a0a1d9544740882a7171a3527eea7f3c02e875d2eb: Status 404 returned error can't find the container with id 8e89d3f6263becae181af8a0a1d9544740882a7171a3527eea7f3c02e875d2eb Nov 02 18:30:01.028122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.028099 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75"] Nov 02 18:30:01.040047 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.040020 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz"] Nov 02 18:30:01.044196 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:30:01.044169 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3ac9a6_4ae1_43f9_acdc_45a36e893708.slice/crio-3a87b15b1bbbf68478f1112f212c7f880b9dc750f722ce36ad0a7807b2796973 WatchSource:0}: Error finding container 3a87b15b1bbbf68478f1112f212c7f880b9dc750f722ce36ad0a7807b2796973: Status 404 returned error can't find the container with id 3a87b15b1bbbf68478f1112f212c7f880b9dc750f722ce36ad0a7807b2796973 Nov 02 18:30:01.275198 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.275119 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:01.290885 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.290850 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs" event={"ID":"ef601c1e-13de-4586-8c17-2847fa6abdbe","Type":"ContainerStarted","Data":"19786c90606fac26ffa7ca09bdb4b6c3f8a53572685c6d727d6d7f2156748879"} Nov 02 18:30:01.292371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.292229 5549 generic.go:356] "Generic (PLEG): container finished" podID="3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0" containerID="75cf09299eed3f253d2b5e94a79365624903e38044b52689a4cf57ec4fc0368e" exitCode=0 Nov 02 18:30:01.292371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.292286 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" Nov 02 18:30:01.292371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.292337 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" event={"ID":"3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0","Type":"ContainerDied","Data":"75cf09299eed3f253d2b5e94a79365624903e38044b52689a4cf57ec4fc0368e"} Nov 02 18:30:01.292565 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.292384 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f" event={"ID":"3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0","Type":"ContainerDied","Data":"889e8bd5fb04401520c19787d4eeb7b8fd12cba426d2a9b2e8b3fdb672e8b85e"} Nov 02 18:30:01.292565 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.292410 5549 scope.go:122] "RemoveContainer" containerID="75cf09299eed3f253d2b5e94a79365624903e38044b52689a4cf57ec4fc0368e" Nov 02 18:30:01.294429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.294403 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75" event={"ID":"f319a273-37c3-41d3-9aa1-14df989794db","Type":"ContainerStarted","Data":"8e89d3f6263becae181af8a0a1d9544740882a7171a3527eea7f3c02e875d2eb"} Nov 02 18:30:01.296465 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.296220 5549 generic.go:356] "Generic (PLEG): container finished" podID="060e829c-0a72-48a9-b9db-13db919d875f" containerID="a068c66873b39a611762edeeb9ed2ffda2d3a620fca0303fd24466d3e5cb89b9" exitCode=0 Nov 02 18:30:01.296465 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.296304 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" Nov 02 18:30:01.296465 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.296315 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" event={"ID":"060e829c-0a72-48a9-b9db-13db919d875f","Type":"ContainerDied","Data":"a068c66873b39a611762edeeb9ed2ffda2d3a620fca0303fd24466d3e5cb89b9"} Nov 02 18:30:01.296465 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.296356 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth" event={"ID":"060e829c-0a72-48a9-b9db-13db919d875f","Type":"ContainerDied","Data":"0cc307e641548f5fd1e4b0c1021166020b8876a3ca5dc01715416e382e134e16"} Nov 02 18:30:01.298430 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.298403 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz" event={"ID":"5d3ac9a6-4ae1-43f9-acdc-45a36e893708","Type":"ContainerStarted","Data":"3a87b15b1bbbf68478f1112f212c7f880b9dc750f722ce36ad0a7807b2796973"} Nov 02 18:30:01.304278 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.304257 5549 scope.go:122] "RemoveContainer" containerID="75cf09299eed3f253d2b5e94a79365624903e38044b52689a4cf57ec4fc0368e" Nov 02 18:30:01.304592 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:01.304570 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75cf09299eed3f253d2b5e94a79365624903e38044b52689a4cf57ec4fc0368e\": container with ID starting with 75cf09299eed3f253d2b5e94a79365624903e38044b52689a4cf57ec4fc0368e not found: ID does not exist" containerID="75cf09299eed3f253d2b5e94a79365624903e38044b52689a4cf57ec4fc0368e" Nov 02 18:30:01.304655 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.304599 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75cf09299eed3f253d2b5e94a79365624903e38044b52689a4cf57ec4fc0368e"} err="failed to get container status \"75cf09299eed3f253d2b5e94a79365624903e38044b52689a4cf57ec4fc0368e\": rpc error: code = NotFound desc = could not find container \"75cf09299eed3f253d2b5e94a79365624903e38044b52689a4cf57ec4fc0368e\": container with ID starting with 75cf09299eed3f253d2b5e94a79365624903e38044b52689a4cf57ec4fc0368e not found: ID does not exist" Nov 02 18:30:01.304655 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.304614 5549 scope.go:122] "RemoveContainer" containerID="a068c66873b39a611762edeeb9ed2ffda2d3a620fca0303fd24466d3e5cb89b9" Nov 02 18:30:01.314381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.314362 5549 scope.go:122] "RemoveContainer" containerID="a068c66873b39a611762edeeb9ed2ffda2d3a620fca0303fd24466d3e5cb89b9" Nov 02 18:30:01.314639 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:01.314620 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a068c66873b39a611762edeeb9ed2ffda2d3a620fca0303fd24466d3e5cb89b9\": container with ID starting with a068c66873b39a611762edeeb9ed2ffda2d3a620fca0303fd24466d3e5cb89b9 not found: ID does not exist" containerID="a068c66873b39a611762edeeb9ed2ffda2d3a620fca0303fd24466d3e5cb89b9" Nov 02 18:30:01.314682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.314648 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a068c66873b39a611762edeeb9ed2ffda2d3a620fca0303fd24466d3e5cb89b9"} err="failed to get container status \"a068c66873b39a611762edeeb9ed2ffda2d3a620fca0303fd24466d3e5cb89b9\": rpc error: code = NotFound desc = could not find container \"a068c66873b39a611762edeeb9ed2ffda2d3a620fca0303fd24466d3e5cb89b9\": container with ID starting with a068c66873b39a611762edeeb9ed2ffda2d3a620fca0303fd24466d3e5cb89b9 not found: ID does not exist" Nov 02 18:30:01.321735 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.321710 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f"] Nov 02 18:30:01.327050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.327017 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-24cfb6-696bd96559-qjk8f"] Nov 02 18:30:01.342012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.341988 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth"] Nov 02 18:30:01.345961 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.345927 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-24cfb6-647b74bd55-ntqth"] Nov 02 18:30:01.433106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:01.433019 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:02.213637 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:02.213535 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:02.307760 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:02.307723 5549 generic.go:356] "Generic (PLEG): container finished" podID="5d3ac9a6-4ae1-43f9-acdc-45a36e893708" containerID="80957e9bcc538f1231dddd9f44144f7141e204774cebbea687ad882058866f11" exitCode=60 Nov 02 18:30:02.307995 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:02.307812 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz" event={"ID":"5d3ac9a6-4ae1-43f9-acdc-45a36e893708","Type":"ContainerDied","Data":"80957e9bcc538f1231dddd9f44144f7141e204774cebbea687ad882058866f11"} Nov 02 18:30:02.308174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:02.308156 5549 scope.go:122] "RemoveContainer" containerID="80957e9bcc538f1231dddd9f44144f7141e204774cebbea687ad882058866f11" Nov 02 18:30:02.309586 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:02.309564 5549 generic.go:356] "Generic (PLEG): container finished" podID="ef601c1e-13de-4586-8c17-2847fa6abdbe" containerID="e635aa61678cf6684fac24849e8e9f669ba3ed3ec7cd425ca9c7ead9b70a940d" exitCode=60 Nov 02 18:30:02.309692 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:02.309651 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs" event={"ID":"ef601c1e-13de-4586-8c17-2847fa6abdbe","Type":"ContainerDied","Data":"e635aa61678cf6684fac24849e8e9f669ba3ed3ec7cd425ca9c7ead9b70a940d"} Nov 02 18:30:02.309932 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:02.309897 5549 scope.go:122] "RemoveContainer" containerID="e635aa61678cf6684fac24849e8e9f669ba3ed3ec7cd425ca9c7ead9b70a940d" Nov 02 18:30:02.312747 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:02.312722 5549 generic.go:356] "Generic (PLEG): container finished" podID="f319a273-37c3-41d3-9aa1-14df989794db" containerID="3fe59e1c9486141a84e7af3d1cffe68e5668d6a82513a37c05724a530e0cacf5" exitCode=60 Nov 02 18:30:02.313067 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:02.313052 5549 scope.go:122] "RemoveContainer" containerID="3fe59e1c9486141a84e7af3d1cffe68e5668d6a82513a37c05724a530e0cacf5" Nov 02 18:30:02.321225 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:02.321195 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060e829c-0a72-48a9-b9db-13db919d875f" path="/var/lib/kubelet/pods/060e829c-0a72-48a9-b9db-13db919d875f/volumes" Nov 02 18:30:02.321787 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:02.321748 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0" path="/var/lib/kubelet/pods/3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0/volumes" Nov 02 18:30:02.322201 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:02.322178 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75" event={"ID":"f319a273-37c3-41d3-9aa1-14df989794db","Type":"ContainerDied","Data":"3fe59e1c9486141a84e7af3d1cffe68e5668d6a82513a37c05724a530e0cacf5"} Nov 02 18:30:02.768421 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:02.768327 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:02.925996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:02.923109 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:03.320977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:03.320923 5549 generic.go:356] "Generic (PLEG): container finished" podID="5d3ac9a6-4ae1-43f9-acdc-45a36e893708" containerID="f5393dbb02b21444a20e76b8e5c6a65e2da7c5b87ea0a750b0648692e96effe7" exitCode=60 Nov 02 18:30:03.321299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:03.320983 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz" event={"ID":"5d3ac9a6-4ae1-43f9-acdc-45a36e893708","Type":"ContainerDied","Data":"f5393dbb02b21444a20e76b8e5c6a65e2da7c5b87ea0a750b0648692e96effe7"} Nov 02 18:30:03.321299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:03.321073 5549 scope.go:122] "RemoveContainer" containerID="80957e9bcc538f1231dddd9f44144f7141e204774cebbea687ad882058866f11" Nov 02 18:30:03.321411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:03.321378 5549 scope.go:122] "RemoveContainer" containerID="f5393dbb02b21444a20e76b8e5c6a65e2da7c5b87ea0a750b0648692e96effe7" Nov 02 18:30:03.321744 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:03.321722 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz_odh-ai-gateway-infra(5d3ac9a6-4ae1-43f9-acdc-45a36e893708)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz" podUID="5d3ac9a6-4ae1-43f9-acdc-45a36e893708" Nov 02 18:30:03.323028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:03.323007 5549 generic.go:356] "Generic (PLEG): container finished" podID="ef601c1e-13de-4586-8c17-2847fa6abdbe" containerID="f83d95195827423688699a53d0a0d8d0860c582bf256c238d1666af3acb8e51a" exitCode=60 Nov 02 18:30:03.323112 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:03.323087 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs" event={"ID":"ef601c1e-13de-4586-8c17-2847fa6abdbe","Type":"ContainerDied","Data":"f83d95195827423688699a53d0a0d8d0860c582bf256c238d1666af3acb8e51a"} Nov 02 18:30:03.323348 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:03.323333 5549 scope.go:122] "RemoveContainer" containerID="f83d95195827423688699a53d0a0d8d0860c582bf256c238d1666af3acb8e51a" Nov 02 18:30:03.323684 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:03.323653 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs_odh-ai-gateway-infra(ef601c1e-13de-4586-8c17-2847fa6abdbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs" podUID="ef601c1e-13de-4586-8c17-2847fa6abdbe" Nov 02 18:30:03.325112 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:03.325089 5549 generic.go:356] "Generic (PLEG): container finished" podID="f319a273-37c3-41d3-9aa1-14df989794db" containerID="b5ea41fb6814412c376cc220ccdc4c790ed654cb91f889c84542d387ec59f0e5" exitCode=60 Nov 02 18:30:03.325205 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:03.325173 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75" event={"ID":"f319a273-37c3-41d3-9aa1-14df989794db","Type":"ContainerDied","Data":"b5ea41fb6814412c376cc220ccdc4c790ed654cb91f889c84542d387ec59f0e5"} Nov 02 18:30:03.325347 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:03.325329 5549 scope.go:122] "RemoveContainer" containerID="b5ea41fb6814412c376cc220ccdc4c790ed654cb91f889c84542d387ec59f0e5" Nov 02 18:30:03.325566 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:03.325546 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75_odh-ai-gateway-infra(f319a273-37c3-41d3-9aa1-14df989794db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75" podUID="f319a273-37c3-41d3-9aa1-14df989794db" Nov 02 18:30:03.409420 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:03.409392 5549 scope.go:122] "RemoveContainer" containerID="e635aa61678cf6684fac24849e8e9f669ba3ed3ec7cd425ca9c7ead9b70a940d" Nov 02 18:30:03.422751 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:03.422728 5549 scope.go:122] "RemoveContainer" containerID="3fe59e1c9486141a84e7af3d1cffe68e5668d6a82513a37c05724a530e0cacf5" Nov 02 18:30:03.692337 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:03.690807 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:04.333188 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:04.333152 5549 scope.go:122] "RemoveContainer" containerID="f83d95195827423688699a53d0a0d8d0860c582bf256c238d1666af3acb8e51a" Nov 02 18:30:04.333607 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:04.333502 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs_odh-ai-gateway-infra(ef601c1e-13de-4586-8c17-2847fa6abdbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs" podUID="ef601c1e-13de-4586-8c17-2847fa6abdbe" Nov 02 18:30:04.334841 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:04.334814 5549 scope.go:122] "RemoveContainer" containerID="b5ea41fb6814412c376cc220ccdc4c790ed654cb91f889c84542d387ec59f0e5" Nov 02 18:30:04.335142 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:04.335121 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75_odh-ai-gateway-infra(f319a273-37c3-41d3-9aa1-14df989794db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75" podUID="f319a273-37c3-41d3-9aa1-14df989794db" Nov 02 18:30:04.336466 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:04.336442 5549 scope.go:122] "RemoveContainer" containerID="f5393dbb02b21444a20e76b8e5c6a65e2da7c5b87ea0a750b0648692e96effe7" Nov 02 18:30:04.336676 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:04.336653 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz_odh-ai-gateway-infra(5d3ac9a6-4ae1-43f9-acdc-45a36e893708)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz" podUID="5d3ac9a6-4ae1-43f9-acdc-45a36e893708" Nov 02 18:30:04.380325 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:04.380193 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq"] Nov 02 18:30:04.389327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:04.389297 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg"] Nov 02 18:30:04.533089 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:04.533053 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:04.667547 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:04.667463 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:05.374243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:05.374206 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:06.125599 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:06.125563 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:06.894898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:06.894851 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:07.048542 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:07.047390 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:07.197094 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:07.197063 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:07.270207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:07.270167 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd"] Nov 02 18:30:07.287748 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:07.286191 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf"] Nov 02 18:30:07.479396 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:07.479297 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:07.680097 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:07.680058 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:07.888056 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:07.887455 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:08.481089 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:08.481051 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk"] Nov 02 18:30:08.482351 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:08.482318 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" podUID="6fb1f539-186f-4f47-9ca1-abf514ef710e" containerName="maas-api" containerID="cri-o://e56edcd13dd770576c4aceea8fc7be6e025aeb20b9781c991deb07809997ebf5" gracePeriod=30 Nov 02 18:30:08.563215 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:08.563179 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:08.875979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:08.874129 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 18:30:08.881208 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:08.876410 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:08.954089 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:08.954055 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fb1f539-186f-4f47-9ca1-abf514ef710e-kube-api-access-k2pz4\" (UniqueName: \"kubernetes.io/projected/6fb1f539-186f-4f47-9ca1-abf514ef710e-kube-api-access-k2pz4\") pod \"6fb1f539-186f-4f47-9ca1-abf514ef710e\" (UID: \"6fb1f539-186f-4f47-9ca1-abf514ef710e\") " Nov 02 18:30:08.954283 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:08.954122 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fb1f539-186f-4f47-9ca1-abf514ef710e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fb1f539-186f-4f47-9ca1-abf514ef710e-maas-api-tls\") pod \"6fb1f539-186f-4f47-9ca1-abf514ef710e\" (UID: \"6fb1f539-186f-4f47-9ca1-abf514ef710e\") " Nov 02 18:30:08.954283 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:08.954195 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fb1f539-186f-4f47-9ca1-abf514ef710e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fb1f539-186f-4f47-9ca1-abf514ef710e-openshift-service-ca\") pod \"6fb1f539-186f-4f47-9ca1-abf514ef710e\" (UID: \"6fb1f539-186f-4f47-9ca1-abf514ef710e\") " Nov 02 18:30:08.954440 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:08.954302 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fb1f539-186f-4f47-9ca1-abf514ef710e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fb1f539-186f-4f47-9ca1-abf514ef710e-metrics-certs\") pod \"6fb1f539-186f-4f47-9ca1-abf514ef710e\" (UID: \"6fb1f539-186f-4f47-9ca1-abf514ef710e\") " Nov 02 18:30:08.954580 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:08.954548 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb1f539-186f-4f47-9ca1-abf514ef710e-openshift-service-ca" pod "6fb1f539-186f-4f47-9ca1-abf514ef710e" (UID: "6fb1f539-186f-4f47-9ca1-abf514ef710e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:08.954870 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:08.954848 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fb1f539-186f-4f47-9ca1-abf514ef710e-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:08.956277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:08.956247 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb1f539-186f-4f47-9ca1-abf514ef710e-kube-api-access-k2pz4" pod "6fb1f539-186f-4f47-9ca1-abf514ef710e" (UID: "6fb1f539-186f-4f47-9ca1-abf514ef710e"). InnerVolumeSpecName "kube-api-access-k2pz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:08.956394 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:08.956369 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fb1f539-186f-4f47-9ca1-abf514ef710e-maas-api-tls" pod "6fb1f539-186f-4f47-9ca1-abf514ef710e" (UID: "6fb1f539-186f-4f47-9ca1-abf514ef710e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:30:08.956479 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:08.956458 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fb1f539-186f-4f47-9ca1-abf514ef710e-metrics-certs" pod "6fb1f539-186f-4f47-9ca1-abf514ef710e" (UID: "6fb1f539-186f-4f47-9ca1-abf514ef710e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:30:09.056318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:09.056284 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fb1f539-186f-4f47-9ca1-abf514ef710e-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:09.056318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:09.056314 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fb1f539-186f-4f47-9ca1-abf514ef710e-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:09.056318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:09.056325 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2pz4\" (UniqueName: \"kubernetes.io/projected/6fb1f539-186f-4f47-9ca1-abf514ef710e-kube-api-access-k2pz4\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:09.182554 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:09.182473 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:09.388453 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:09.388417 5549 generic.go:356] "Generic (PLEG): container finished" podID="6fb1f539-186f-4f47-9ca1-abf514ef710e" containerID="e56edcd13dd770576c4aceea8fc7be6e025aeb20b9781c991deb07809997ebf5" exitCode=0 Nov 02 18:30:09.388633 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:09.388474 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" Nov 02 18:30:09.388633 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:09.388504 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" event={"ID":"6fb1f539-186f-4f47-9ca1-abf514ef710e","Type":"ContainerDied","Data":"e56edcd13dd770576c4aceea8fc7be6e025aeb20b9781c991deb07809997ebf5"} Nov 02 18:30:09.388633 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:09.388563 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk" event={"ID":"6fb1f539-186f-4f47-9ca1-abf514ef710e","Type":"ContainerDied","Data":"8f75a05ebc20a76d93dd46a7dba967d95c57085261b6f9df5c1dba07bc1c6e37"} Nov 02 18:30:09.388633 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:09.388595 5549 scope.go:122] "RemoveContainer" containerID="e56edcd13dd770576c4aceea8fc7be6e025aeb20b9781c991deb07809997ebf5" Nov 02 18:30:09.400112 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:09.400091 5549 scope.go:122] "RemoveContainer" containerID="e56edcd13dd770576c4aceea8fc7be6e025aeb20b9781c991deb07809997ebf5" Nov 02 18:30:09.400392 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:09.400367 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56edcd13dd770576c4aceea8fc7be6e025aeb20b9781c991deb07809997ebf5\": container with ID starting with e56edcd13dd770576c4aceea8fc7be6e025aeb20b9781c991deb07809997ebf5 not found: ID does not exist" containerID="e56edcd13dd770576c4aceea8fc7be6e025aeb20b9781c991deb07809997ebf5" Nov 02 18:30:09.400473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:09.400402 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56edcd13dd770576c4aceea8fc7be6e025aeb20b9781c991deb07809997ebf5"} err="failed to get container status \"e56edcd13dd770576c4aceea8fc7be6e025aeb20b9781c991deb07809997ebf5\": rpc error: code = NotFound desc = could not find container \"e56edcd13dd770576c4aceea8fc7be6e025aeb20b9781c991deb07809997ebf5\": container with ID starting with e56edcd13dd770576c4aceea8fc7be6e025aeb20b9781c991deb07809997ebf5 not found: ID does not exist" Nov 02 18:30:09.412274 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:09.412244 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk"] Nov 02 18:30:09.420808 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:09.420782 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-86f5f0-7d9f5dbf4f-mvrbk"] Nov 02 18:30:09.510873 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:09.510828 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:09.770028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:09.769888 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:09.913071 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:09.913038 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:10.090088 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:10.090003 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd"] Nov 02 18:30:10.090272 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:10.090237 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" podUID="7af43d18-1d68-45cb-839f-bc220435b550" containerName="istio-proxy" containerID="cri-o://3797c44e3da50c979557ed0e15ca8e421b69176f863fb7610b391154b4a50559" gracePeriod=30 Nov 02 18:30:10.177353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:10.177320 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:10.332672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:10.329890 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fb1f539-186f-4f47-9ca1-abf514ef710e" path="/var/lib/kubelet/pods/6fb1f539-186f-4f47-9ca1-abf514ef710e/volumes" Nov 02 18:30:10.706450 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:10.706397 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:11.326985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:11.325718 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:12.073703 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.073658 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:12.271345 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.271281 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd" podUID="5897373f-0b89-4603-9b04-a99f0f23582b" containerName="payload-processing" containerID="cri-o://2b703500ea3d4d25171ca465015fa8dfd6980276469f2b39047fccb83843785f" gracePeriod=25 Nov 02 18:30:12.286697 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.286659 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf" podUID="6d0ffd57-6326-4c50-aea7-36e52d0e40d0" containerName="payload-pre-processing" containerID="cri-o://40fd4a906d1bc9b435dfb6ea17240115f5834ff2eb4d9ad09ac52533d077f430" gracePeriod=25 Nov 02 18:30:12.414771 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.414739 5549 generic.go:356] "Generic (PLEG): container finished" podID="5897373f-0b89-4603-9b04-a99f0f23582b" containerID="2b703500ea3d4d25171ca465015fa8dfd6980276469f2b39047fccb83843785f" exitCode=0 Nov 02 18:30:12.414946 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.414814 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd" event={"ID":"5897373f-0b89-4603-9b04-a99f0f23582b","Type":"ContainerDied","Data":"2b703500ea3d4d25171ca465015fa8dfd6980276469f2b39047fccb83843785f"} Nov 02 18:30:12.416570 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.416541 5549 generic.go:356] "Generic (PLEG): container finished" podID="6d0ffd57-6326-4c50-aea7-36e52d0e40d0" containerID="40fd4a906d1bc9b435dfb6ea17240115f5834ff2eb4d9ad09ac52533d077f430" exitCode=0 Nov 02 18:30:12.416744 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.416653 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf" event={"ID":"6d0ffd57-6326-4c50-aea7-36e52d0e40d0","Type":"ContainerDied","Data":"40fd4a906d1bc9b435dfb6ea17240115f5834ff2eb4d9ad09ac52533d077f430"} Nov 02 18:30:12.563239 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.563210 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd" Nov 02 18:30:12.609438 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.609403 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5897373f-0b89-4603-9b04-a99f0f23582b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5897373f-0b89-4603-9b04-a99f0f23582b-plugins-config-volume\") pod \"5897373f-0b89-4603-9b04-a99f0f23582b\" (UID: \"5897373f-0b89-4603-9b04-a99f0f23582b\") " Nov 02 18:30:12.609665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.609649 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5897373f-0b89-4603-9b04-a99f0f23582b-kube-api-access-9grlw\" (UniqueName: \"kubernetes.io/projected/5897373f-0b89-4603-9b04-a99f0f23582b-kube-api-access-9grlw\") pod \"5897373f-0b89-4603-9b04-a99f0f23582b\" (UID: \"5897373f-0b89-4603-9b04-a99f0f23582b\") " Nov 02 18:30:12.609844 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.609814 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5897373f-0b89-4603-9b04-a99f0f23582b-plugins-config-volume" pod "5897373f-0b89-4603-9b04-a99f0f23582b" (UID: "5897373f-0b89-4603-9b04-a99f0f23582b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:12.610231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.610207 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5897373f-0b89-4603-9b04-a99f0f23582b-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:12.611788 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.611767 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5897373f-0b89-4603-9b04-a99f0f23582b-kube-api-access-9grlw" pod "5897373f-0b89-4603-9b04-a99f0f23582b" (UID: "5897373f-0b89-4603-9b04-a99f0f23582b"). InnerVolumeSpecName "kube-api-access-9grlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:12.617384 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.617358 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf" Nov 02 18:30:12.711262 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.711231 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d0ffd57-6326-4c50-aea7-36e52d0e40d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0ffd57-6326-4c50-aea7-36e52d0e40d0-plugins-config-volume\") pod \"6d0ffd57-6326-4c50-aea7-36e52d0e40d0\" (UID: \"6d0ffd57-6326-4c50-aea7-36e52d0e40d0\") " Nov 02 18:30:12.711510 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.711288 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0ffd57-6326-4c50-aea7-36e52d0e40d0-kube-api-access-ktkr7\" (UniqueName: \"kubernetes.io/projected/6d0ffd57-6326-4c50-aea7-36e52d0e40d0-kube-api-access-ktkr7\") pod \"6d0ffd57-6326-4c50-aea7-36e52d0e40d0\" (UID: \"6d0ffd57-6326-4c50-aea7-36e52d0e40d0\") " Nov 02 18:30:12.711735 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.711706 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0ffd57-6326-4c50-aea7-36e52d0e40d0-plugins-config-volume" pod "6d0ffd57-6326-4c50-aea7-36e52d0e40d0" (UID: "6d0ffd57-6326-4c50-aea7-36e52d0e40d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:12.712026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.711896 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9grlw\" (UniqueName: \"kubernetes.io/projected/5897373f-0b89-4603-9b04-a99f0f23582b-kube-api-access-9grlw\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:12.712026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.711925 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0ffd57-6326-4c50-aea7-36e52d0e40d0-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:12.713660 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.713636 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0ffd57-6326-4c50-aea7-36e52d0e40d0-kube-api-access-ktkr7" pod "6d0ffd57-6326-4c50-aea7-36e52d0e40d0" (UID: "6d0ffd57-6326-4c50-aea7-36e52d0e40d0"). InnerVolumeSpecName "kube-api-access-ktkr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:12.819801 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.818833 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktkr7\" (UniqueName: \"kubernetes.io/projected/6d0ffd57-6326-4c50-aea7-36e52d0e40d0-kube-api-access-ktkr7\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:12.819801 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.819240 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:12.963787 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:12.963749 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk"] Nov 02 18:30:13.058428 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:13.058395 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:13.212198 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:13.212094 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:13.424131 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:13.424099 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd" Nov 02 18:30:13.424332 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:13.424109 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd" event={"ID":"5897373f-0b89-4603-9b04-a99f0f23582b","Type":"ContainerDied","Data":"32437ebd1b82c290a133e34817883c50d1bc8a0181f3d203ca8345b5d0f740d2"} Nov 02 18:30:13.424332 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:13.424267 5549 scope.go:122] "RemoveContainer" containerID="2b703500ea3d4d25171ca465015fa8dfd6980276469f2b39047fccb83843785f" Nov 02 18:30:13.427615 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:13.427592 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf" event={"ID":"6d0ffd57-6326-4c50-aea7-36e52d0e40d0","Type":"ContainerDied","Data":"a9b022d50d33545e72ba5472512a4bdbd5e0d7f51fffd772f07ef89540b2a7b9"} Nov 02 18:30:13.427684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:13.427668 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf" Nov 02 18:30:13.437319 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:13.437296 5549 scope.go:122] "RemoveContainer" containerID="40fd4a906d1bc9b435dfb6ea17240115f5834ff2eb4d9ad09ac52533d077f430" Nov 02 18:30:13.452853 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:13.452829 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd"] Nov 02 18:30:13.462045 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:13.462024 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-86f5f0-7446f48bdc-57npd"] Nov 02 18:30:13.474221 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:13.474182 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf"] Nov 02 18:30:13.477792 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:13.477770 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-86f5f0-6f7f46dccf-rqnxf"] Nov 02 18:30:13.741083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:13.741046 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:14.312868 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:14.312828 5549 scope.go:122] "RemoveContainer" containerID="f5393dbb02b21444a20e76b8e5c6a65e2da7c5b87ea0a750b0648692e96effe7" Nov 02 18:30:14.322030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:14.322004 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5897373f-0b89-4603-9b04-a99f0f23582b" path="/var/lib/kubelet/pods/5897373f-0b89-4603-9b04-a99f0f23582b/volumes" Nov 02 18:30:14.322418 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:14.322405 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d0ffd57-6326-4c50-aea7-36e52d0e40d0" path="/var/lib/kubelet/pods/6d0ffd57-6326-4c50-aea7-36e52d0e40d0/volumes" Nov 02 18:30:14.398636 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:14.398610 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:14.595997 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:14.595946 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:15.248309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.248271 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:15.448908 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.448876 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 18:30:15.459099 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.459060 5549 generic.go:356] "Generic (PLEG): container finished" podID="5d3ac9a6-4ae1-43f9-acdc-45a36e893708" containerID="44cf32ee24caf31a65e794db12eed6452618134c3c6bb86e8ade653164740631" exitCode=60 Nov 02 18:30:15.459289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.459204 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz" event={"ID":"5d3ac9a6-4ae1-43f9-acdc-45a36e893708","Type":"ContainerDied","Data":"44cf32ee24caf31a65e794db12eed6452618134c3c6bb86e8ade653164740631"} Nov 02 18:30:15.459351 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.459299 5549 scope.go:122] "RemoveContainer" containerID="f5393dbb02b21444a20e76b8e5c6a65e2da7c5b87ea0a750b0648692e96effe7" Nov 02 18:30:15.460083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.459893 5549 scope.go:122] "RemoveContainer" containerID="44cf32ee24caf31a65e794db12eed6452618134c3c6bb86e8ade653164740631" Nov 02 18:30:15.460449 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:15.460416 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz_odh-ai-gateway-infra(5d3ac9a6-4ae1-43f9-acdc-45a36e893708)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz" podUID="5d3ac9a6-4ae1-43f9-acdc-45a36e893708" Nov 02 18:30:15.462001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.461966 5549 generic.go:356] "Generic (PLEG): container finished" podID="7af43d18-1d68-45cb-839f-bc220435b550" containerID="3797c44e3da50c979557ed0e15ca8e421b69176f863fb7610b391154b4a50559" exitCode=0 Nov 02 18:30:15.462252 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.462231 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" event={"ID":"7af43d18-1d68-45cb-839f-bc220435b550","Type":"ContainerDied","Data":"3797c44e3da50c979557ed0e15ca8e421b69176f863fb7610b391154b4a50559"} Nov 02 18:30:15.462349 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.462292 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" event={"ID":"7af43d18-1d68-45cb-839f-bc220435b550","Type":"ContainerDied","Data":"e42d5526b1e0c27161222123f5aefa7ca6b45ab1af6fbbddcabe36661fe908fb"} Nov 02 18:30:15.462439 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.462421 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd" Nov 02 18:30:15.466756 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.466729 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-workload-socket\") pod \"7af43d18-1d68-45cb-839f-bc220435b550\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " Nov 02 18:30:15.466885 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.466811 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-workload-certs\") pod \"7af43d18-1d68-45cb-839f-bc220435b550\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " Nov 02 18:30:15.466885 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.466848 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7af43d18-1d68-45cb-839f-bc220435b550-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af43d18-1d68-45cb-839f-bc220435b550-istiod-ca-cert\") pod \"7af43d18-1d68-45cb-839f-bc220435b550\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " Nov 02 18:30:15.467046 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.466928 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af43d18-1d68-45cb-839f-bc220435b550-istio-token\" (UniqueName: \"kubernetes.io/projected/7af43d18-1d68-45cb-839f-bc220435b550-istio-token\") pod \"7af43d18-1d68-45cb-839f-bc220435b550\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " Nov 02 18:30:15.467197 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.467175 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af43d18-1d68-45cb-839f-bc220435b550-kube-api-access-z2pk5\" (UniqueName: \"kubernetes.io/projected/7af43d18-1d68-45cb-839f-bc220435b550-kube-api-access-z2pk5\") pod \"7af43d18-1d68-45cb-839f-bc220435b550\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " Nov 02 18:30:15.467274 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.467257 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-istio-data\") pod \"7af43d18-1d68-45cb-839f-bc220435b550\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " Nov 02 18:30:15.467381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.467357 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-workload-socket" pod "7af43d18-1d68-45cb-839f-bc220435b550" (UID: "7af43d18-1d68-45cb-839f-bc220435b550"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:15.467510 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.467491 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-istio-data" pod "7af43d18-1d68-45cb-839f-bc220435b550" (UID: "7af43d18-1d68-45cb-839f-bc220435b550"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:15.467510 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.467502 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-workload-certs" pod "7af43d18-1d68-45cb-839f-bc220435b550" (UID: "7af43d18-1d68-45cb-839f-bc220435b550"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:15.467646 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.467628 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-istio-envoy\") pod \"7af43d18-1d68-45cb-839f-bc220435b550\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " Nov 02 18:30:15.467714 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.467698 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-credential-socket\") pod \"7af43d18-1d68-45cb-839f-bc220435b550\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " Nov 02 18:30:15.468065 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.468038 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7af43d18-1d68-45cb-839f-bc220435b550-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af43d18-1d68-45cb-839f-bc220435b550-istio-podinfo\") pod \"7af43d18-1d68-45cb-839f-bc220435b550\" (UID: \"7af43d18-1d68-45cb-839f-bc220435b550\") " Nov 02 18:30:15.468397 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.467942 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-credential-socket" pod "7af43d18-1d68-45cb-839f-bc220435b550" (UID: "7af43d18-1d68-45cb-839f-bc220435b550"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:15.468397 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.468013 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af43d18-1d68-45cb-839f-bc220435b550-istiod-ca-cert" pod "7af43d18-1d68-45cb-839f-bc220435b550" (UID: "7af43d18-1d68-45cb-839f-bc220435b550"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:15.470482 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.470451 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af43d18-1d68-45cb-839f-bc220435b550-istio-token" pod "7af43d18-1d68-45cb-839f-bc220435b550" (UID: "7af43d18-1d68-45cb-839f-bc220435b550"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:15.470566 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.470490 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af43d18-1d68-45cb-839f-bc220435b550-kube-api-access-z2pk5" pod "7af43d18-1d68-45cb-839f-bc220435b550" (UID: "7af43d18-1d68-45cb-839f-bc220435b550"). InnerVolumeSpecName "kube-api-access-z2pk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:15.470685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.470665 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7af43d18-1d68-45cb-839f-bc220435b550-istio-podinfo" pod "7af43d18-1d68-45cb-839f-bc220435b550" (UID: "7af43d18-1d68-45cb-839f-bc220435b550"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:30:15.470770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.470684 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-istio-envoy" pod "7af43d18-1d68-45cb-839f-bc220435b550" (UID: "7af43d18-1d68-45cb-839f-bc220435b550"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:15.471437 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.471387 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:15.471437 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.471429 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af43d18-1d68-45cb-839f-bc220435b550-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:15.471704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.471456 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:15.471704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.471475 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:15.471704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.471492 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af43d18-1d68-45cb-839f-bc220435b550-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:15.471704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.471509 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af43d18-1d68-45cb-839f-bc220435b550-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:15.471704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.471527 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2pk5\" (UniqueName: \"kubernetes.io/projected/7af43d18-1d68-45cb-839f-bc220435b550-kube-api-access-z2pk5\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:15.471704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.471546 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:15.471704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.471563 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af43d18-1d68-45cb-839f-bc220435b550-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:15.514997 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.514969 5549 scope.go:122] "RemoveContainer" containerID="3797c44e3da50c979557ed0e15ca8e421b69176f863fb7610b391154b4a50559" Nov 02 18:30:15.525811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.525783 5549 scope.go:122] "RemoveContainer" containerID="3797c44e3da50c979557ed0e15ca8e421b69176f863fb7610b391154b4a50559" Nov 02 18:30:15.526213 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:15.526182 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3797c44e3da50c979557ed0e15ca8e421b69176f863fb7610b391154b4a50559\": container with ID starting with 3797c44e3da50c979557ed0e15ca8e421b69176f863fb7610b391154b4a50559 not found: ID does not exist" containerID="3797c44e3da50c979557ed0e15ca8e421b69176f863fb7610b391154b4a50559" Nov 02 18:30:15.526296 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.526224 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3797c44e3da50c979557ed0e15ca8e421b69176f863fb7610b391154b4a50559"} err="failed to get container status \"3797c44e3da50c979557ed0e15ca8e421b69176f863fb7610b391154b4a50559\": rpc error: code = NotFound desc = could not find container \"3797c44e3da50c979557ed0e15ca8e421b69176f863fb7610b391154b4a50559\": container with ID starting with 3797c44e3da50c979557ed0e15ca8e421b69176f863fb7610b391154b4a50559 not found: ID does not exist" Nov 02 18:30:15.798695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.798656 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd"] Nov 02 18:30:15.801624 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:15.801591 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-dpdsd"] Nov 02 18:30:16.263876 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:16.263721 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:16.313056 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:16.313018 5549 scope.go:122] "RemoveContainer" containerID="f83d95195827423688699a53d0a0d8d0860c582bf256c238d1666af3acb8e51a" Nov 02 18:30:16.328033 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:16.327998 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7af43d18-1d68-45cb-839f-bc220435b550" path="/var/lib/kubelet/pods/7af43d18-1d68-45cb-839f-bc220435b550/volumes" Nov 02 18:30:16.503731 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:16.503705 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz"] Nov 02 18:30:16.659367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:16.659339 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz" Nov 02 18:30:16.701125 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:16.701093 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3ac9a6-4ae1-43f9-acdc-45a36e893708-kube-api-access-8kw7q\" (UniqueName: \"kubernetes.io/projected/5d3ac9a6-4ae1-43f9-acdc-45a36e893708-kube-api-access-8kw7q\") pod \"5d3ac9a6-4ae1-43f9-acdc-45a36e893708\" (UID: \"5d3ac9a6-4ae1-43f9-acdc-45a36e893708\") " Nov 02 18:30:16.701257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:16.701227 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d3ac9a6-4ae1-43f9-acdc-45a36e893708-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3ac9a6-4ae1-43f9-acdc-45a36e893708-maas-api-service-ca\") pod \"5d3ac9a6-4ae1-43f9-acdc-45a36e893708\" (UID: \"5d3ac9a6-4ae1-43f9-acdc-45a36e893708\") " Nov 02 18:30:16.701896 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:16.701853 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3ac9a6-4ae1-43f9-acdc-45a36e893708-maas-api-service-ca" pod "5d3ac9a6-4ae1-43f9-acdc-45a36e893708" (UID: "5d3ac9a6-4ae1-43f9-acdc-45a36e893708"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:16.703768 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:16.703741 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3ac9a6-4ae1-43f9-acdc-45a36e893708-kube-api-access-8kw7q" pod "5d3ac9a6-4ae1-43f9-acdc-45a36e893708" (UID: "5d3ac9a6-4ae1-43f9-acdc-45a36e893708"). InnerVolumeSpecName "kube-api-access-8kw7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:16.803655 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:16.803620 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kw7q\" (UniqueName: \"kubernetes.io/projected/5d3ac9a6-4ae1-43f9-acdc-45a36e893708-kube-api-access-8kw7q\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:16.803655 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:16.803651 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3ac9a6-4ae1-43f9-acdc-45a36e893708-maas-api-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:17.147013 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:17.146978 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:17.312757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:17.312726 5549 scope.go:122] "RemoveContainer" containerID="b5ea41fb6814412c376cc220ccdc4c790ed654cb91f889c84542d387ec59f0e5" Nov 02 18:30:17.481928 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:17.481897 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz" Nov 02 18:30:17.482143 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:17.481941 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz" event={"ID":"5d3ac9a6-4ae1-43f9-acdc-45a36e893708","Type":"ContainerDied","Data":"3a87b15b1bbbf68478f1112f212c7f880b9dc750f722ce36ad0a7807b2796973"} Nov 02 18:30:17.482143 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:17.482051 5549 scope.go:122] "RemoveContainer" containerID="44cf32ee24caf31a65e794db12eed6452618134c3c6bb86e8ade653164740631" Nov 02 18:30:17.484390 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:17.484362 5549 generic.go:356] "Generic (PLEG): container finished" podID="ef601c1e-13de-4586-8c17-2847fa6abdbe" containerID="f5f03fcd11bd158f53cbd079732f63cf90e2035d99562aef67a3243dcd34ead2" exitCode=60 Nov 02 18:30:17.484543 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:17.484403 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs" event={"ID":"ef601c1e-13de-4586-8c17-2847fa6abdbe","Type":"ContainerDied","Data":"f5f03fcd11bd158f53cbd079732f63cf90e2035d99562aef67a3243dcd34ead2"} Nov 02 18:30:17.484869 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:17.484847 5549 scope.go:122] "RemoveContainer" containerID="f5f03fcd11bd158f53cbd079732f63cf90e2035d99562aef67a3243dcd34ead2" Nov 02 18:30:17.485277 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:17.485253 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs_odh-ai-gateway-infra(ef601c1e-13de-4586-8c17-2847fa6abdbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs" podUID="ef601c1e-13de-4586-8c17-2847fa6abdbe" Nov 02 18:30:17.500897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:17.500873 5549 scope.go:122] "RemoveContainer" containerID="f83d95195827423688699a53d0a0d8d0860c582bf256c238d1666af3acb8e51a" Nov 02 18:30:17.522071 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:17.522044 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz"] Nov 02 18:30:17.529471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:17.529445 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894070-gmsxz"] Nov 02 18:30:17.590859 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:17.590820 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:30:17.690372 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:17.690325 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:17.896804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:17.896705 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:17.964805 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:17.964759 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" podUID="5e400bab-ea24-4848-92ba-c770160096b8" containerName="payload-pre-processing" containerID="cri-o://b5e4f7d395217d9a7b55ef9ef14ec1537a49d4efbf76b0a3849de375e6489b38" gracePeriod=25 Nov 02 18:30:18.052290 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:18.052242 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:18.336204 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:18.336161 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d3ac9a6-4ae1-43f9-acdc-45a36e893708" path="/var/lib/kubelet/pods/5d3ac9a6-4ae1-43f9-acdc-45a36e893708/volumes" Nov 02 18:30:18.349286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:18.349250 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:18.495876 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:18.495845 5549 generic.go:356] "Generic (PLEG): container finished" podID="f319a273-37c3-41d3-9aa1-14df989794db" containerID="9c6987b216bcd4bb79235e697c3e1e6aff706320fb657ad4b4ac4fb2ffa897ac" exitCode=60 Nov 02 18:30:18.495876 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:18.495880 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75" event={"ID":"f319a273-37c3-41d3-9aa1-14df989794db","Type":"ContainerDied","Data":"9c6987b216bcd4bb79235e697c3e1e6aff706320fb657ad4b4ac4fb2ffa897ac"} Nov 02 18:30:18.496148 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:18.495927 5549 scope.go:122] "RemoveContainer" containerID="b5ea41fb6814412c376cc220ccdc4c790ed654cb91f889c84542d387ec59f0e5" Nov 02 18:30:18.496375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:18.496354 5549 scope.go:122] "RemoveContainer" containerID="9c6987b216bcd4bb79235e697c3e1e6aff706320fb657ad4b4ac4fb2ffa897ac" Nov 02 18:30:18.496703 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:18.496678 5549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75_odh-ai-gateway-infra(f319a273-37c3-41d3-9aa1-14df989794db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75" podUID="f319a273-37c3-41d3-9aa1-14df989794db" Nov 02 18:30:18.515150 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:18.515122 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs"] Nov 02 18:30:18.525295 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:18.525264 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:18.676381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:18.676354 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs" Nov 02 18:30:18.739148 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:18.739112 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef601c1e-13de-4586-8c17-2847fa6abdbe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef601c1e-13de-4586-8c17-2847fa6abdbe-maas-api-service-ca\") pod \"ef601c1e-13de-4586-8c17-2847fa6abdbe\" (UID: \"ef601c1e-13de-4586-8c17-2847fa6abdbe\") " Nov 02 18:30:18.739352 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:18.739261 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef601c1e-13de-4586-8c17-2847fa6abdbe-kube-api-access-2mz82\" (UniqueName: \"kubernetes.io/projected/ef601c1e-13de-4586-8c17-2847fa6abdbe-kube-api-access-2mz82\") pod \"ef601c1e-13de-4586-8c17-2847fa6abdbe\" (UID: \"ef601c1e-13de-4586-8c17-2847fa6abdbe\") " Nov 02 18:30:18.739556 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:18.739530 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef601c1e-13de-4586-8c17-2847fa6abdbe-maas-api-service-ca" pod "ef601c1e-13de-4586-8c17-2847fa6abdbe" (UID: "ef601c1e-13de-4586-8c17-2847fa6abdbe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:18.739755 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:18.739736 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef601c1e-13de-4586-8c17-2847fa6abdbe-maas-api-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:18.741338 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:18.741316 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef601c1e-13de-4586-8c17-2847fa6abdbe-kube-api-access-2mz82" pod "ef601c1e-13de-4586-8c17-2847fa6abdbe" (UID: "ef601c1e-13de-4586-8c17-2847fa6abdbe"). InnerVolumeSpecName "kube-api-access-2mz82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:18.840784 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:18.840707 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mz82\" (UniqueName: \"kubernetes.io/projected/ef601c1e-13de-4586-8c17-2847fa6abdbe-kube-api-access-2mz82\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:18.875545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:18.875508 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:19.061506 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.061463 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:19.214434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.214393 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:19.402510 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.402472 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" containerID="cri-o://b7f1122692abed2551abb878a79bd691af783825b87fab61c22ab7fd9d0df329" gracePeriod=45 Nov 02 18:30:19.422429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.421001 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" containerID="cri-o://06117bd3ec453ae0b4dc24b5ca5430f4068896f862e13bedf59f225a014a0bcf" gracePeriod=45 Nov 02 18:30:19.506469 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.506428 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs" event={"ID":"ef601c1e-13de-4586-8c17-2847fa6abdbe","Type":"ContainerDied","Data":"19786c90606fac26ffa7ca09bdb4b6c3f8a53572685c6d727d6d7f2156748879"} Nov 02 18:30:19.506610 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.506460 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs" Nov 02 18:30:19.506610 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.506542 5549 scope.go:122] "RemoveContainer" containerID="f5f03fcd11bd158f53cbd079732f63cf90e2035d99562aef67a3243dcd34ead2" Nov 02 18:30:19.534393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.534362 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75"] Nov 02 18:30:19.562058 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.561930 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": dial tcp 10.133.0.80:8000: connect: connection refused" Nov 02 18:30:19.565250 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.565215 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": dial tcp 10.133.0.82:8000: connect: connection refused" Nov 02 18:30:19.602612 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.602555 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs"] Nov 02 18:30:19.614103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.614060 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894070-4k5zs"] Nov 02 18:30:19.760219 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.760190 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75" Nov 02 18:30:19.763597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.763575 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 18:30:19.768408 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.768388 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 18:30:19.866720 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.866689 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186b2aa4-3dd7-4329-bbcf-72dd270c371d-tls-certs\" (UniqueName: \"kubernetes.io/secret/186b2aa4-3dd7-4329-bbcf-72dd270c371d-tls-certs\") pod \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " Nov 02 18:30:19.866911 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.866743 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f319a273-37c3-41d3-9aa1-14df989794db-kube-api-access-pp5wv\" (UniqueName: \"kubernetes.io/projected/f319a273-37c3-41d3-9aa1-14df989794db-kube-api-access-pp5wv\") pod \"f319a273-37c3-41d3-9aa1-14df989794db\" (UID: \"f319a273-37c3-41d3-9aa1-14df989794db\") " Nov 02 18:30:19.866911 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.866781 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f319a273-37c3-41d3-9aa1-14df989794db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f319a273-37c3-41d3-9aa1-14df989794db-maas-api-service-ca\") pod \"f319a273-37c3-41d3-9aa1-14df989794db\" (UID: \"f319a273-37c3-41d3-9aa1-14df989794db\") " Nov 02 18:30:19.866911 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.866827 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-kube-api-access-d5jnn\" (UniqueName: \"kubernetes.io/projected/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-kube-api-access-d5jnn\") pod \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " Nov 02 18:30:19.866911 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.866859 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-home\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-home\") pod \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " Nov 02 18:30:19.867246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.866980 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-tmp-dir\") pod \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " Nov 02 18:30:19.867246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.867024 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-model-cache\") pod \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " Nov 02 18:30:19.867246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.867083 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-home\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-home\") pod \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " Nov 02 18:30:19.867246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.867119 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-tmp-dir\") pod \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " Nov 02 18:30:19.867246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.867181 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-home" pod "186b2aa4-3dd7-4329-bbcf-72dd270c371d" (UID: "186b2aa4-3dd7-4329-bbcf-72dd270c371d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:19.867246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.867195 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-dshm\") pod \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " Nov 02 18:30:19.867574 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.867384 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-dshm\") pod \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " Nov 02 18:30:19.867574 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.867422 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186b2aa4-3dd7-4329-bbcf-72dd270c371d-kube-api-access-d8xcw\" (UniqueName: \"kubernetes.io/projected/186b2aa4-3dd7-4329-bbcf-72dd270c371d-kube-api-access-d8xcw\") pod \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\" (UID: \"186b2aa4-3dd7-4329-bbcf-72dd270c371d\") " Nov 02 18:30:19.867574 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.867451 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-model-cache\") pod \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " Nov 02 18:30:19.867574 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.867496 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-tls-certs\" (UniqueName: \"kubernetes.io/secret/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-tls-certs\") pod \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\" (UID: \"2fa67943-f0d5-401a-a6fa-79be9f6bdb95\") " Nov 02 18:30:19.867800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.867733 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-model-cache" pod "186b2aa4-3dd7-4329-bbcf-72dd270c371d" (UID: "186b2aa4-3dd7-4329-bbcf-72dd270c371d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:19.868482 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.868225 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.868482 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.868253 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.868919 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.868739 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-home" pod "2fa67943-f0d5-401a-a6fa-79be9f6bdb95" (UID: "2fa67943-f0d5-401a-a6fa-79be9f6bdb95"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:19.869181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.869150 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-tmp-dir" pod "2fa67943-f0d5-401a-a6fa-79be9f6bdb95" (UID: "2fa67943-f0d5-401a-a6fa-79be9f6bdb95"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:19.869504 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.869480 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f319a273-37c3-41d3-9aa1-14df989794db-maas-api-service-ca" pod "f319a273-37c3-41d3-9aa1-14df989794db" (UID: "f319a273-37c3-41d3-9aa1-14df989794db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:19.869812 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.869794 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-tmp-dir" pod "186b2aa4-3dd7-4329-bbcf-72dd270c371d" (UID: "186b2aa4-3dd7-4329-bbcf-72dd270c371d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:19.869927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.869891 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f319a273-37c3-41d3-9aa1-14df989794db-kube-api-access-pp5wv" pod "f319a273-37c3-41d3-9aa1-14df989794db" (UID: "f319a273-37c3-41d3-9aa1-14df989794db"). InnerVolumeSpecName "kube-api-access-pp5wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:19.870054 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.870023 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-model-cache" pod "2fa67943-f0d5-401a-a6fa-79be9f6bdb95" (UID: "2fa67943-f0d5-401a-a6fa-79be9f6bdb95"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:19.870327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.870304 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-dshm" pod "2fa67943-f0d5-401a-a6fa-79be9f6bdb95" (UID: "2fa67943-f0d5-401a-a6fa-79be9f6bdb95"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:19.870398 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.870379 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-dshm" pod "186b2aa4-3dd7-4329-bbcf-72dd270c371d" (UID: "186b2aa4-3dd7-4329-bbcf-72dd270c371d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:19.870485 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.870466 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-tls-certs" pod "2fa67943-f0d5-401a-a6fa-79be9f6bdb95" (UID: "2fa67943-f0d5-401a-a6fa-79be9f6bdb95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:30:19.870546 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.870505 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-kube-api-access-d5jnn" pod "2fa67943-f0d5-401a-a6fa-79be9f6bdb95" (UID: "2fa67943-f0d5-401a-a6fa-79be9f6bdb95"). InnerVolumeSpecName "kube-api-access-d5jnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:19.871614 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.871596 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186b2aa4-3dd7-4329-bbcf-72dd270c371d-kube-api-access-d8xcw" pod "186b2aa4-3dd7-4329-bbcf-72dd270c371d" (UID: "186b2aa4-3dd7-4329-bbcf-72dd270c371d"). InnerVolumeSpecName "kube-api-access-d8xcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:19.871932 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.871914 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186b2aa4-3dd7-4329-bbcf-72dd270c371d-tls-certs" pod "186b2aa4-3dd7-4329-bbcf-72dd270c371d" (UID: "186b2aa4-3dd7-4329-bbcf-72dd270c371d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:30:19.931716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.931674 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:19.969471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.969428 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp5wv\" (UniqueName: \"kubernetes.io/projected/f319a273-37c3-41d3-9aa1-14df989794db-kube-api-access-pp5wv\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.969471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.969470 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f319a273-37c3-41d3-9aa1-14df989794db-maas-api-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.969672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.969490 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5jnn\" (UniqueName: \"kubernetes.io/projected/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-kube-api-access-d5jnn\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.969672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.969505 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.969672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.969520 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.969672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.969534 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.969672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.969546 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.969672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.969559 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/186b2aa4-3dd7-4329-bbcf-72dd270c371d-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.969672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.969571 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8xcw\" (UniqueName: \"kubernetes.io/projected/186b2aa4-3dd7-4329-bbcf-72dd270c371d-kube-api-access-d8xcw\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.969672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.969584 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.969672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.969598 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fa67943-f0d5-401a-a6fa-79be9f6bdb95-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.969672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:19.969613 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/186b2aa4-3dd7-4329-bbcf-72dd270c371d-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:20.208282 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.206220 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:20.321109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.321073 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef601c1e-13de-4586-8c17-2847fa6abdbe" path="/var/lib/kubelet/pods/ef601c1e-13de-4586-8c17-2847fa6abdbe/volumes" Nov 02 18:30:20.515203 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.515111 5549 generic.go:356] "Generic (PLEG): container finished" podID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerID="b7f1122692abed2551abb878a79bd691af783825b87fab61c22ab7fd9d0df329" exitCode=0 Nov 02 18:30:20.515203 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.515165 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" Nov 02 18:30:20.515421 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.515197 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" event={"ID":"2fa67943-f0d5-401a-a6fa-79be9f6bdb95","Type":"ContainerDied","Data":"b7f1122692abed2551abb878a79bd691af783825b87fab61c22ab7fd9d0df329"} Nov 02 18:30:20.515421 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.515243 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq" event={"ID":"2fa67943-f0d5-401a-a6fa-79be9f6bdb95","Type":"ContainerDied","Data":"a5a8345c45d5a8ce13123e28068f8434126521a0f152b686cd8030314f5bac78"} Nov 02 18:30:20.515421 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.515261 5549 scope.go:122] "RemoveContainer" containerID="b7f1122692abed2551abb878a79bd691af783825b87fab61c22ab7fd9d0df329" Nov 02 18:30:20.518129 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.518101 5549 generic.go:356] "Generic (PLEG): container finished" podID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerID="06117bd3ec453ae0b4dc24b5ca5430f4068896f862e13bedf59f225a014a0bcf" exitCode=0 Nov 02 18:30:20.518265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.518138 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" Nov 02 18:30:20.518265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.518164 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" event={"ID":"186b2aa4-3dd7-4329-bbcf-72dd270c371d","Type":"ContainerDied","Data":"06117bd3ec453ae0b4dc24b5ca5430f4068896f862e13bedf59f225a014a0bcf"} Nov 02 18:30:20.518265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.518199 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg" event={"ID":"186b2aa4-3dd7-4329-bbcf-72dd270c371d","Type":"ContainerDied","Data":"b07bf6348e94ee6f485b2db4f09444a2cfbcbfe604b3df46eaa5ceea28bee064"} Nov 02 18:30:20.519834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.519751 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75" event={"ID":"f319a273-37c3-41d3-9aa1-14df989794db","Type":"ContainerDied","Data":"8e89d3f6263becae181af8a0a1d9544740882a7171a3527eea7f3c02e875d2eb"} Nov 02 18:30:20.519834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.519819 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75" Nov 02 18:30:20.526382 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.526361 5549 scope.go:122] "RemoveContainer" containerID="b7f1122692abed2551abb878a79bd691af783825b87fab61c22ab7fd9d0df329" Nov 02 18:30:20.526660 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:20.526639 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f1122692abed2551abb878a79bd691af783825b87fab61c22ab7fd9d0df329\": container with ID starting with b7f1122692abed2551abb878a79bd691af783825b87fab61c22ab7fd9d0df329 not found: ID does not exist" containerID="b7f1122692abed2551abb878a79bd691af783825b87fab61c22ab7fd9d0df329" Nov 02 18:30:20.526740 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.526669 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f1122692abed2551abb878a79bd691af783825b87fab61c22ab7fd9d0df329"} err="failed to get container status \"b7f1122692abed2551abb878a79bd691af783825b87fab61c22ab7fd9d0df329\": rpc error: code = NotFound desc = could not find container \"b7f1122692abed2551abb878a79bd691af783825b87fab61c22ab7fd9d0df329\": container with ID starting with b7f1122692abed2551abb878a79bd691af783825b87fab61c22ab7fd9d0df329 not found: ID does not exist" Nov 02 18:30:20.526740 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.526694 5549 scope.go:122] "RemoveContainer" containerID="06117bd3ec453ae0b4dc24b5ca5430f4068896f862e13bedf59f225a014a0bcf" Nov 02 18:30:20.535781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.535756 5549 scope.go:122] "RemoveContainer" containerID="06117bd3ec453ae0b4dc24b5ca5430f4068896f862e13bedf59f225a014a0bcf" Nov 02 18:30:20.536189 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:20.536067 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06117bd3ec453ae0b4dc24b5ca5430f4068896f862e13bedf59f225a014a0bcf\": container with ID starting with 06117bd3ec453ae0b4dc24b5ca5430f4068896f862e13bedf59f225a014a0bcf not found: ID does not exist" containerID="06117bd3ec453ae0b4dc24b5ca5430f4068896f862e13bedf59f225a014a0bcf" Nov 02 18:30:20.536189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.536092 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06117bd3ec453ae0b4dc24b5ca5430f4068896f862e13bedf59f225a014a0bcf"} err="failed to get container status \"06117bd3ec453ae0b4dc24b5ca5430f4068896f862e13bedf59f225a014a0bcf\": rpc error: code = NotFound desc = could not find container \"06117bd3ec453ae0b4dc24b5ca5430f4068896f862e13bedf59f225a014a0bcf\": container with ID starting with 06117bd3ec453ae0b4dc24b5ca5430f4068896f862e13bedf59f225a014a0bcf not found: ID does not exist" Nov 02 18:30:20.536189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.536111 5549 scope.go:122] "RemoveContainer" containerID="9c6987b216bcd4bb79235e697c3e1e6aff706320fb657ad4b4ac4fb2ffa897ac" Nov 02 18:30:20.537723 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.537701 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq"] Nov 02 18:30:20.547600 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.547577 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/distinct-2-w3-a37cf0-kserve-79c748b8b5-fljdq"] Nov 02 18:30:20.569062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.569033 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg"] Nov 02 18:30:20.573932 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.573905 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/distinct-w3-a37cf0-kserve-76b4dc8c94-dl5kg"] Nov 02 18:30:20.587038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.587012 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75"] Nov 02 18:30:20.592597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.592573 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894070-cdp75"] Nov 02 18:30:20.897371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:20.896916 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:21.613145 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:21.613066 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:22.103063 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:22.103027 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:22.251663 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:22.251627 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:22.320206 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:22.320169 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" path="/var/lib/kubelet/pods/186b2aa4-3dd7-4329-bbcf-72dd270c371d/volumes" Nov 02 18:30:22.320655 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:22.320640 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" path="/var/lib/kubelet/pods/2fa67943-f0d5-401a-a6fa-79be9f6bdb95/volumes" Nov 02 18:30:22.321032 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:22.321020 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f319a273-37c3-41d3-9aa1-14df989794db" path="/var/lib/kubelet/pods/f319a273-37c3-41d3-9aa1-14df989794db/volumes" Nov 02 18:30:23.015555 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:23.015513 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:23.660295 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:23.660251 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:24.348435 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:24.348405 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:25.100180 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:25.100133 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:25.597334 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:25.597290 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v"] Nov 02 18:30:25.613174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:25.613139 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27"] Nov 02 18:30:25.720531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:25.720495 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:25.849190 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:25.849111 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:25.977550 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:25.977500 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:26.496158 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:26.496123 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:26.704650 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:26.704612 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:27.304188 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:27.304153 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:27.710695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:27.710649 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" podUID="5e400bab-ea24-4848-92ba-c770160096b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:30:27.903886 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:27.903842 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:28.358318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:28.358275 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:28.606976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:28.603726 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:29.184909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:29.184868 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:29.431432 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:29.431388 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:30.264277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:30.264236 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:30.598562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:30.598461 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" podUID="924d0e07-2010-4007-8148-5b7623447adb" containerName="payload-processing" containerID="cri-o://bc7bb080824974272a3098bff74707eeb6ae69fdcbe0b7386e7acbbb66edce87" gracePeriod=25 Nov 02 18:30:30.614178 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:30.614140 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" podUID="5a3d4c2f-d60c-41dd-af49-4923ee32729e" containerName="payload-pre-processing" containerID="cri-o://b09c2bce08988ad036a89217b2bed2580aee6394c46b8ea48f9d40e3714be92f" gracePeriod=25 Nov 02 18:30:30.739603 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:30.739570 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:31.155485 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:31.155449 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:31.827976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:31.821563 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:32.507604 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:32.507503 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:32.726775 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:32.726738 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:32.858017 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:32.857922 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:33.193616 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:33.193542 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:33.331799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:33.331763 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:33.606407 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:33.606378 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:34.044267 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:34.044229 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:34.202495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:34.202464 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:34.432090 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:34.432010 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" podUID="5a3d4c2f-d60c-41dd-af49-4923ee32729e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:30:34.432090 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:34.432010 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" podUID="924d0e07-2010-4007-8148-5b7623447adb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:30:34.712893 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:34.712853 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:34.846136 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:34.846100 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:34.977900 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:34.977820 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:35.598399 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:35.594339 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:36.311843 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:36.311803 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:36.918624 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:36.918588 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:37.588541 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:37.588505 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:37.711115 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:37.711078 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" podUID="5e400bab-ea24-4848-92ba-c770160096b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:30:38.209639 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.209609 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:38.271905 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.271879 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" Nov 02 18:30:38.356266 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.356236 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e400bab-ea24-4848-92ba-c770160096b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e400bab-ea24-4848-92ba-c770160096b8-plugins-config-volume\") pod \"5e400bab-ea24-4848-92ba-c770160096b8\" (UID: \"5e400bab-ea24-4848-92ba-c770160096b8\") " Nov 02 18:30:38.356504 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.356287 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e400bab-ea24-4848-92ba-c770160096b8-kube-api-access-cj4j8\" (UniqueName: \"kubernetes.io/projected/5e400bab-ea24-4848-92ba-c770160096b8-kube-api-access-cj4j8\") pod \"5e400bab-ea24-4848-92ba-c770160096b8\" (UID: \"5e400bab-ea24-4848-92ba-c770160096b8\") " Nov 02 18:30:38.356607 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.356570 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e400bab-ea24-4848-92ba-c770160096b8-plugins-config-volume" pod "5e400bab-ea24-4848-92ba-c770160096b8" (UID: "5e400bab-ea24-4848-92ba-c770160096b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:38.356728 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.356713 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e400bab-ea24-4848-92ba-c770160096b8-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:38.358367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.358343 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e400bab-ea24-4848-92ba-c770160096b8-kube-api-access-cj4j8" pod "5e400bab-ea24-4848-92ba-c770160096b8" (UID: "5e400bab-ea24-4848-92ba-c770160096b8"). InnerVolumeSpecName "kube-api-access-cj4j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:38.458272 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.458236 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj4j8\" (UniqueName: \"kubernetes.io/projected/5e400bab-ea24-4848-92ba-c770160096b8-kube-api-access-cj4j8\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:38.717905 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.717869 5549 generic.go:356] "Generic (PLEG): container finished" podID="5e400bab-ea24-4848-92ba-c770160096b8" containerID="b5e4f7d395217d9a7b55ef9ef14ec1537a49d4efbf76b0a3849de375e6489b38" exitCode=0 Nov 02 18:30:38.718131 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.717921 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" Nov 02 18:30:38.718131 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.717965 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" event={"ID":"5e400bab-ea24-4848-92ba-c770160096b8","Type":"ContainerDied","Data":"b5e4f7d395217d9a7b55ef9ef14ec1537a49d4efbf76b0a3849de375e6489b38"} Nov 02 18:30:38.718131 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.718023 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk" event={"ID":"5e400bab-ea24-4848-92ba-c770160096b8","Type":"ContainerDied","Data":"07e061412f36946c7bbf0ae48d4ab2dc1d396781687a10fae065bf99393564db"} Nov 02 18:30:38.718131 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.718045 5549 scope.go:122] "RemoveContainer" containerID="b5e4f7d395217d9a7b55ef9ef14ec1537a49d4efbf76b0a3849de375e6489b38" Nov 02 18:30:38.728466 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.728447 5549 scope.go:122] "RemoveContainer" containerID="b5e4f7d395217d9a7b55ef9ef14ec1537a49d4efbf76b0a3849de375e6489b38" Nov 02 18:30:38.728726 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:38.728705 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e4f7d395217d9a7b55ef9ef14ec1537a49d4efbf76b0a3849de375e6489b38\": container with ID starting with b5e4f7d395217d9a7b55ef9ef14ec1537a49d4efbf76b0a3849de375e6489b38 not found: ID does not exist" containerID="b5e4f7d395217d9a7b55ef9ef14ec1537a49d4efbf76b0a3849de375e6489b38" Nov 02 18:30:38.728772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.728733 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e4f7d395217d9a7b55ef9ef14ec1537a49d4efbf76b0a3849de375e6489b38"} err="failed to get container status \"b5e4f7d395217d9a7b55ef9ef14ec1537a49d4efbf76b0a3849de375e6489b38\": rpc error: code = NotFound desc = could not find container \"b5e4f7d395217d9a7b55ef9ef14ec1537a49d4efbf76b0a3849de375e6489b38\": container with ID starting with b5e4f7d395217d9a7b55ef9ef14ec1537a49d4efbf76b0a3849de375e6489b38 not found: ID does not exist" Nov 02 18:30:38.742329 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.742309 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk"] Nov 02 18:30:38.744786 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.744765 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a37cf0-fd9d757c9-6dfrk"] Nov 02 18:30:38.969474 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.969367 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:38.980960 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.980931 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns"] Nov 02 18:30:38.992935 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:38.992912 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7"] Nov 02 18:30:39.121376 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:39.121339 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:39.582381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:39.582342 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:39.781651 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:39.781619 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:40.319646 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:40.319611 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e400bab-ea24-4848-92ba-c770160096b8" path="/var/lib/kubelet/pods/5e400bab-ea24-4848-92ba-c770160096b8/volumes" Nov 02 18:30:40.364551 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:40.364512 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:41.007981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:41.007922 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:41.753635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:41.753600 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:41.893365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:41.893019 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:42.600272 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:42.600236 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:42.998992 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:42.998943 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:43.196024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:43.195993 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:43.396360 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:43.396271 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:43.904701 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:43.904651 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:44.140505 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:44.140466 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn"] Nov 02 18:30:44.152741 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:44.140721 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" podUID="dd458c81-9d15-45ba-92b1-473e1378c087" containerName="istio-proxy" containerID="cri-o://f24e0b9c2c90090fd6804fe972a7ee823ee9d5dc27d870a160a642dd495488df" gracePeriod=30 Nov 02 18:30:44.223159 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:44.223125 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:44.350016 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:44.349974 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:44.432297 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:44.432252 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" podUID="5a3d4c2f-d60c-41dd-af49-4923ee32729e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:30:44.432466 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:44.432322 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" podUID="924d0e07-2010-4007-8148-5b7623447adb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:30:44.544096 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:44.544008 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:30:45.056741 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:45.056702 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:45.223506 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:45.223463 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6"] Nov 02 18:30:45.235377 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:45.235343 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97"] Nov 02 18:30:45.366778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:45.366692 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:45.507068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:45.507026 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:46.119527 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:46.119465 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:47.022470 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:47.022400 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:47.587179 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:47.587139 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:30:47.749517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:47.749486 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:47.880499 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:47.880380 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:48.518942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:48.518898 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:49.208009 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.205282 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:49.409556 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.409494 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 18:30:49.512115 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.512077 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-istio-envoy\") pod \"dd458c81-9d15-45ba-92b1-473e1378c087\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " Nov 02 18:30:49.512309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.512124 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd458c81-9d15-45ba-92b1-473e1378c087-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd458c81-9d15-45ba-92b1-473e1378c087-istio-podinfo\") pod \"dd458c81-9d15-45ba-92b1-473e1378c087\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " Nov 02 18:30:49.512309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.512175 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-istio-data\") pod \"dd458c81-9d15-45ba-92b1-473e1378c087\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " Nov 02 18:30:49.512309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.512212 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-workload-socket\") pod \"dd458c81-9d15-45ba-92b1-473e1378c087\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " Nov 02 18:30:49.512309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.512294 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-workload-certs\") pod \"dd458c81-9d15-45ba-92b1-473e1378c087\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " Nov 02 18:30:49.512540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.512325 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd458c81-9d15-45ba-92b1-473e1378c087-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd458c81-9d15-45ba-92b1-473e1378c087-istiod-ca-cert\") pod \"dd458c81-9d15-45ba-92b1-473e1378c087\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " Nov 02 18:30:49.512540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.512377 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd458c81-9d15-45ba-92b1-473e1378c087-kube-api-access-trhj7\" (UniqueName: \"kubernetes.io/projected/dd458c81-9d15-45ba-92b1-473e1378c087-kube-api-access-trhj7\") pod \"dd458c81-9d15-45ba-92b1-473e1378c087\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " Nov 02 18:30:49.512540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.512441 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd458c81-9d15-45ba-92b1-473e1378c087-istio-token\" (UniqueName: \"kubernetes.io/projected/dd458c81-9d15-45ba-92b1-473e1378c087-istio-token\") pod \"dd458c81-9d15-45ba-92b1-473e1378c087\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " Nov 02 18:30:49.512540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.512499 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-istio-data" pod "dd458c81-9d15-45ba-92b1-473e1378c087" (UID: "dd458c81-9d15-45ba-92b1-473e1378c087"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:49.512775 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.512556 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-workload-socket" pod "dd458c81-9d15-45ba-92b1-473e1378c087" (UID: "dd458c81-9d15-45ba-92b1-473e1378c087"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:49.512775 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.512580 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-workload-certs" pod "dd458c81-9d15-45ba-92b1-473e1378c087" (UID: "dd458c81-9d15-45ba-92b1-473e1378c087"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:49.512775 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.512607 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-credential-socket\") pod \"dd458c81-9d15-45ba-92b1-473e1378c087\" (UID: \"dd458c81-9d15-45ba-92b1-473e1378c087\") " Nov 02 18:30:49.512775 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.512702 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd458c81-9d15-45ba-92b1-473e1378c087-istiod-ca-cert" pod "dd458c81-9d15-45ba-92b1-473e1378c087" (UID: "dd458c81-9d15-45ba-92b1-473e1378c087"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:49.513056 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.512869 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-credential-socket" pod "dd458c81-9d15-45ba-92b1-473e1378c087" (UID: "dd458c81-9d15-45ba-92b1-473e1378c087"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:49.513300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.513277 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:49.513300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.513305 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd458c81-9d15-45ba-92b1-473e1378c087-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:49.513300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.513321 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:49.513560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.513336 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:49.513560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.513349 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:49.514573 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.514550 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-istio-envoy" pod "dd458c81-9d15-45ba-92b1-473e1378c087" (UID: "dd458c81-9d15-45ba-92b1-473e1378c087"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:49.514785 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.514770 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd458c81-9d15-45ba-92b1-473e1378c087-istio-token" pod "dd458c81-9d15-45ba-92b1-473e1378c087" (UID: "dd458c81-9d15-45ba-92b1-473e1378c087"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:49.515095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.515079 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd458c81-9d15-45ba-92b1-473e1378c087-kube-api-access-trhj7" pod "dd458c81-9d15-45ba-92b1-473e1378c087" (UID: "dd458c81-9d15-45ba-92b1-473e1378c087"). InnerVolumeSpecName "kube-api-access-trhj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:49.515152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.515110 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd458c81-9d15-45ba-92b1-473e1378c087-istio-podinfo" pod "dd458c81-9d15-45ba-92b1-473e1378c087" (UID: "dd458c81-9d15-45ba-92b1-473e1378c087"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:30:49.614529 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.614499 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trhj7\" (UniqueName: \"kubernetes.io/projected/dd458c81-9d15-45ba-92b1-473e1378c087-kube-api-access-trhj7\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:49.614529 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.614525 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd458c81-9d15-45ba-92b1-473e1378c087-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:49.614529 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.614535 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd458c81-9d15-45ba-92b1-473e1378c087-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:49.614790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.614544 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd458c81-9d15-45ba-92b1-473e1378c087-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:49.801630 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.801477 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:49.814023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.813220 5549 generic.go:356] "Generic (PLEG): container finished" podID="dd458c81-9d15-45ba-92b1-473e1378c087" containerID="f24e0b9c2c90090fd6804fe972a7ee823ee9d5dc27d870a160a642dd495488df" exitCode=0 Nov 02 18:30:49.814023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.813290 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" Nov 02 18:30:49.814023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.813269 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" event={"ID":"dd458c81-9d15-45ba-92b1-473e1378c087","Type":"ContainerDied","Data":"f24e0b9c2c90090fd6804fe972a7ee823ee9d5dc27d870a160a642dd495488df"} Nov 02 18:30:49.814023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.813387 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn" event={"ID":"dd458c81-9d15-45ba-92b1-473e1378c087","Type":"ContainerDied","Data":"3d7132681b96eda9235a96a8bb1098d4c02e1ad631b3709d5fda1ce677dba7bd"} Nov 02 18:30:49.814023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.813421 5549 scope.go:122] "RemoveContainer" containerID="f24e0b9c2c90090fd6804fe972a7ee823ee9d5dc27d870a160a642dd495488df" Nov 02 18:30:49.828169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.828065 5549 scope.go:122] "RemoveContainer" containerID="f24e0b9c2c90090fd6804fe972a7ee823ee9d5dc27d870a160a642dd495488df" Nov 02 18:30:49.828410 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:49.828391 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f24e0b9c2c90090fd6804fe972a7ee823ee9d5dc27d870a160a642dd495488df\": container with ID starting with f24e0b9c2c90090fd6804fe972a7ee823ee9d5dc27d870a160a642dd495488df not found: ID does not exist" containerID="f24e0b9c2c90090fd6804fe972a7ee823ee9d5dc27d870a160a642dd495488df" Nov 02 18:30:49.828475 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.828418 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f24e0b9c2c90090fd6804fe972a7ee823ee9d5dc27d870a160a642dd495488df"} err="failed to get container status \"f24e0b9c2c90090fd6804fe972a7ee823ee9d5dc27d870a160a642dd495488df\": rpc error: code = NotFound desc = could not find container \"f24e0b9c2c90090fd6804fe972a7ee823ee9d5dc27d870a160a642dd495488df\": container with ID starting with f24e0b9c2c90090fd6804fe972a7ee823ee9d5dc27d870a160a642dd495488df not found: ID does not exist" Nov 02 18:30:49.841087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.841057 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn"] Nov 02 18:30:49.844521 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:49.844494 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-tv9qn"] Nov 02 18:30:50.224623 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.224585 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" podUID="f5c74aae-fd45-4ba6-b30d-99fb59e22c11" containerName="payload-processing" containerID="cri-o://dda7dc55e039bb6afaebb05a9d41917182af703242d326dae7f73b14001ed63d" gracePeriod=25 Nov 02 18:30:50.236089 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.236057 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" podUID="fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb" containerName="payload-pre-processing" containerID="cri-o://eea96c58c7cc8e36a7c079b01efa5ae509f1f7248782db5214ce113f6e146a5c" gracePeriod=25 Nov 02 18:30:50.314360 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.314317 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:50.320770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.320743 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd458c81-9d15-45ba-92b1-473e1378c087" path="/var/lib/kubelet/pods/dd458c81-9d15-45ba-92b1-473e1378c087/volumes" Nov 02 18:30:50.459116 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.459082 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:50.822945 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.822902 5549 generic.go:356] "Generic (PLEG): container finished" podID="924d0e07-2010-4007-8148-5b7623447adb" containerID="bc7bb080824974272a3098bff74707eeb6ae69fdcbe0b7386e7acbbb66edce87" exitCode=0 Nov 02 18:30:50.823252 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.823224 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" event={"ID":"924d0e07-2010-4007-8148-5b7623447adb","Type":"ContainerDied","Data":"bc7bb080824974272a3098bff74707eeb6ae69fdcbe0b7386e7acbbb66edce87"} Nov 02 18:30:50.825659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.825632 5549 generic.go:356] "Generic (PLEG): container finished" podID="5a3d4c2f-d60c-41dd-af49-4923ee32729e" containerID="b09c2bce08988ad036a89217b2bed2580aee6394c46b8ea48f9d40e3714be92f" exitCode=0 Nov 02 18:30:50.825771 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.825685 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" event={"ID":"5a3d4c2f-d60c-41dd-af49-4923ee32729e","Type":"ContainerDied","Data":"b09c2bce08988ad036a89217b2bed2580aee6394c46b8ea48f9d40e3714be92f"} Nov 02 18:30:50.881408 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.881376 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" Nov 02 18:30:50.927463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.927428 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" Nov 02 18:30:50.931786 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.931759 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924d0e07-2010-4007-8148-5b7623447adb-kube-api-access-lrhvb\" (UniqueName: \"kubernetes.io/projected/924d0e07-2010-4007-8148-5b7623447adb-kube-api-access-lrhvb\") pod \"924d0e07-2010-4007-8148-5b7623447adb\" (UID: \"924d0e07-2010-4007-8148-5b7623447adb\") " Nov 02 18:30:50.931946 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.931893 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3d4c2f-d60c-41dd-af49-4923ee32729e-kube-api-access-pkvth\" (UniqueName: \"kubernetes.io/projected/5a3d4c2f-d60c-41dd-af49-4923ee32729e-kube-api-access-pkvth\") pod \"5a3d4c2f-d60c-41dd-af49-4923ee32729e\" (UID: \"5a3d4c2f-d60c-41dd-af49-4923ee32729e\") " Nov 02 18:30:50.931946 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.931933 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a3d4c2f-d60c-41dd-af49-4923ee32729e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3d4c2f-d60c-41dd-af49-4923ee32729e-plugins-config-volume\") pod \"5a3d4c2f-d60c-41dd-af49-4923ee32729e\" (UID: \"5a3d4c2f-d60c-41dd-af49-4923ee32729e\") " Nov 02 18:30:50.932103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.932011 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924d0e07-2010-4007-8148-5b7623447adb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924d0e07-2010-4007-8148-5b7623447adb-plugins-config-volume\") pod \"924d0e07-2010-4007-8148-5b7623447adb\" (UID: \"924d0e07-2010-4007-8148-5b7623447adb\") " Nov 02 18:30:50.932705 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.932656 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3d4c2f-d60c-41dd-af49-4923ee32729e-plugins-config-volume" pod "5a3d4c2f-d60c-41dd-af49-4923ee32729e" (UID: "5a3d4c2f-d60c-41dd-af49-4923ee32729e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:50.932966 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.932924 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924d0e07-2010-4007-8148-5b7623447adb-plugins-config-volume" pod "924d0e07-2010-4007-8148-5b7623447adb" (UID: "924d0e07-2010-4007-8148-5b7623447adb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:50.936120 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.936091 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924d0e07-2010-4007-8148-5b7623447adb-kube-api-access-lrhvb" pod "924d0e07-2010-4007-8148-5b7623447adb" (UID: "924d0e07-2010-4007-8148-5b7623447adb"). InnerVolumeSpecName "kube-api-access-lrhvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:50.936244 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:50.936125 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3d4c2f-d60c-41dd-af49-4923ee32729e-kube-api-access-pkvth" pod "5a3d4c2f-d60c-41dd-af49-4923ee32729e" (UID: "5a3d4c2f-d60c-41dd-af49-4923ee32729e"). InnerVolumeSpecName "kube-api-access-pkvth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:51.033705 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:51.033670 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrhvb\" (UniqueName: \"kubernetes.io/projected/924d0e07-2010-4007-8148-5b7623447adb-kube-api-access-lrhvb\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:51.033705 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:51.033703 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkvth\" (UniqueName: \"kubernetes.io/projected/5a3d4c2f-d60c-41dd-af49-4923ee32729e-kube-api-access-pkvth\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:51.033705 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:51.033713 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3d4c2f-d60c-41dd-af49-4923ee32729e-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:51.034004 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:51.033723 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924d0e07-2010-4007-8148-5b7623447adb-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:51.187135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:51.186451 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:51.758101 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:51.758060 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" podUID="f5c74aae-fd45-4ba6-b30d-99fb59e22c11" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:30:51.758490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:51.758066 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" podUID="fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:30:51.830836 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:51.830800 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:51.837405 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:51.837376 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" Nov 02 18:30:51.837594 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:51.837407 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v" event={"ID":"924d0e07-2010-4007-8148-5b7623447adb","Type":"ContainerDied","Data":"fd9a647bb2d0981a4a8bf3328ff61ea632b77076efab1de4da7fef0d8e6f00b0"} Nov 02 18:30:51.837594 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:51.837468 5549 scope.go:122] "RemoveContainer" containerID="bc7bb080824974272a3098bff74707eeb6ae69fdcbe0b7386e7acbbb66edce87" Nov 02 18:30:51.839177 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:51.839152 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" event={"ID":"5a3d4c2f-d60c-41dd-af49-4923ee32729e","Type":"ContainerDied","Data":"838ff6a1d7f4e2a4f1919a49c7dbf3492fc87b86af6c4d94fe38190a82e8bdd7"} Nov 02 18:30:51.839335 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:51.839182 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27" Nov 02 18:30:51.852108 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:51.852087 5549 scope.go:122] "RemoveContainer" containerID="b09c2bce08988ad036a89217b2bed2580aee6394c46b8ea48f9d40e3714be92f" Nov 02 18:30:51.864282 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:51.864258 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v"] Nov 02 18:30:51.868509 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:51.868446 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b1a788-7f8b8cbf-vhr5v"] Nov 02 18:30:51.878305 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:51.878282 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27"] Nov 02 18:30:51.884160 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:51.884137 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b1a788-5fb65846f9-dhh27"] Nov 02 18:30:52.321467 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:52.321434 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a3d4c2f-d60c-41dd-af49-4923ee32729e" path="/var/lib/kubelet/pods/5a3d4c2f-d60c-41dd-af49-4923ee32729e/volumes" Nov 02 18:30:52.321898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:52.321884 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924d0e07-2010-4007-8148-5b7623447adb" path="/var/lib/kubelet/pods/924d0e07-2010-4007-8148-5b7623447adb/volumes" Nov 02 18:30:52.523527 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:52.523492 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:53.283738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:53.283684 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:53.913676 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:53.913643 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:54.003037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.002998 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" containerID="cri-o://ba04c73f502d01fac0691e95a04d6d265d98842eda58f7f31d36fa1094dac21e" gracePeriod=45 Nov 02 18:30:54.016410 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.016380 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" containerID="cri-o://a409a9c8a19c7681b4119ebebf449ec21a4cec36702a80b17e61a5c6d573b6ee" gracePeriod=45 Nov 02 18:30:54.126313 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.125127 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:54.323347 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.323321 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 18:30:54.326581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.326561 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 18:30:54.402401 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.402368 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-tmp-dir\") pod \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " Nov 02 18:30:54.402621 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.402480 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-model-cache\") pod \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " Nov 02 18:30:54.402621 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.402577 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-tls-certs\" (UniqueName: \"kubernetes.io/secret/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-tls-certs\") pod \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " Nov 02 18:30:54.402621 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.402613 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a10854-4c5f-446a-ac7e-8d12767fa1bf-kube-api-access-jzzpj\" (UniqueName: \"kubernetes.io/projected/64a10854-4c5f-446a-ac7e-8d12767fa1bf-kube-api-access-jzzpj\") pod \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " Nov 02 18:30:54.402853 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.402658 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-dshm\") pod \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " Nov 02 18:30:54.402853 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.402681 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-tmp-dir" pod "6296c79a-bf7c-40ff-b2d2-37d3d9f46309" (UID: "6296c79a-bf7c-40ff-b2d2-37d3d9f46309"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:54.402853 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.402776 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-dshm\") pod \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " Nov 02 18:30:54.402853 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.402820 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-home\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-home\") pod \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " Nov 02 18:30:54.403111 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.402924 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-home\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-home\") pod \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " Nov 02 18:30:54.403111 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.402970 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-model-cache\") pod \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " Nov 02 18:30:54.403111 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.403025 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-tmp-dir\") pod \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " Nov 02 18:30:54.403111 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.403054 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64a10854-4c5f-446a-ac7e-8d12767fa1bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/64a10854-4c5f-446a-ac7e-8d12767fa1bf-tls-certs\") pod \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\" (UID: \"64a10854-4c5f-446a-ac7e-8d12767fa1bf\") " Nov 02 18:30:54.403334 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.403127 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-kube-api-access-66w5v\" (UniqueName: \"kubernetes.io/projected/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-kube-api-access-66w5v\") pod \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\" (UID: \"6296c79a-bf7c-40ff-b2d2-37d3d9f46309\") " Nov 02 18:30:54.404172 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.404144 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:54.404726 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.404602 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-home" pod "6296c79a-bf7c-40ff-b2d2-37d3d9f46309" (UID: "6296c79a-bf7c-40ff-b2d2-37d3d9f46309"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:54.404876 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.404754 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-model-cache" pod "6296c79a-bf7c-40ff-b2d2-37d3d9f46309" (UID: "6296c79a-bf7c-40ff-b2d2-37d3d9f46309"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:54.405155 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.404927 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-tmp-dir" pod "64a10854-4c5f-446a-ac7e-8d12767fa1bf" (UID: "64a10854-4c5f-446a-ac7e-8d12767fa1bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:54.407997 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.406742 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-tls-certs" pod "6296c79a-bf7c-40ff-b2d2-37d3d9f46309" (UID: "6296c79a-bf7c-40ff-b2d2-37d3d9f46309"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:30:54.407997 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.407399 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64a10854-4c5f-446a-ac7e-8d12767fa1bf-tls-certs" pod "64a10854-4c5f-446a-ac7e-8d12767fa1bf" (UID: "64a10854-4c5f-446a-ac7e-8d12767fa1bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:30:54.407997 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.407939 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-home" pod "64a10854-4c5f-446a-ac7e-8d12767fa1bf" (UID: "64a10854-4c5f-446a-ac7e-8d12767fa1bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:54.408551 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.408530 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-model-cache" pod "64a10854-4c5f-446a-ac7e-8d12767fa1bf" (UID: "64a10854-4c5f-446a-ac7e-8d12767fa1bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:54.409198 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.409172 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-dshm" pod "6296c79a-bf7c-40ff-b2d2-37d3d9f46309" (UID: "6296c79a-bf7c-40ff-b2d2-37d3d9f46309"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:54.410169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.410134 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a10854-4c5f-446a-ac7e-8d12767fa1bf-kube-api-access-jzzpj" pod "64a10854-4c5f-446a-ac7e-8d12767fa1bf" (UID: "64a10854-4c5f-446a-ac7e-8d12767fa1bf"). InnerVolumeSpecName "kube-api-access-jzzpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:54.410388 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.410357 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-kube-api-access-66w5v" pod "6296c79a-bf7c-40ff-b2d2-37d3d9f46309" (UID: "6296c79a-bf7c-40ff-b2d2-37d3d9f46309"). InnerVolumeSpecName "kube-api-access-66w5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:54.412073 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.412045 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-dshm" pod "64a10854-4c5f-446a-ac7e-8d12767fa1bf" (UID: "64a10854-4c5f-446a-ac7e-8d12767fa1bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:54.505179 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.505139 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:54.505179 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.505171 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:54.505179 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.505184 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:54.505460 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.505196 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:54.505460 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.505210 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:54.505460 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.505225 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64a10854-4c5f-446a-ac7e-8d12767fa1bf-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:54.505460 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.505236 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66w5v\" (UniqueName: \"kubernetes.io/projected/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-kube-api-access-66w5v\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:54.505460 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.505251 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:54.505460 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.505264 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6296c79a-bf7c-40ff-b2d2-37d3d9f46309-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:54.505460 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.505276 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzzpj\" (UniqueName: \"kubernetes.io/projected/64a10854-4c5f-446a-ac7e-8d12767fa1bf-kube-api-access-jzzpj\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:54.505460 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.505287 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64a10854-4c5f-446a-ac7e-8d12767fa1bf-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:54.861667 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.861572 5549 generic.go:356] "Generic (PLEG): container finished" podID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerID="a409a9c8a19c7681b4119ebebf449ec21a4cec36702a80b17e61a5c6d573b6ee" exitCode=0 Nov 02 18:30:54.861667 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.861611 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" Nov 02 18:30:54.861667 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.861650 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" event={"ID":"6296c79a-bf7c-40ff-b2d2-37d3d9f46309","Type":"ContainerDied","Data":"a409a9c8a19c7681b4119ebebf449ec21a4cec36702a80b17e61a5c6d573b6ee"} Nov 02 18:30:54.861973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.861695 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7" event={"ID":"6296c79a-bf7c-40ff-b2d2-37d3d9f46309","Type":"ContainerDied","Data":"e85bb884cc28a310c48b83089b5f6d974071f4821fbb8692d225e60d4d96c2b9"} Nov 02 18:30:54.861973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.861714 5549 scope.go:122] "RemoveContainer" containerID="a409a9c8a19c7681b4119ebebf449ec21a4cec36702a80b17e61a5c6d573b6ee" Nov 02 18:30:54.863300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.863275 5549 generic.go:356] "Generic (PLEG): container finished" podID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerID="ba04c73f502d01fac0691e95a04d6d265d98842eda58f7f31d36fa1094dac21e" exitCode=0 Nov 02 18:30:54.863391 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.863320 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" Nov 02 18:30:54.863391 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.863355 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" event={"ID":"64a10854-4c5f-446a-ac7e-8d12767fa1bf","Type":"ContainerDied","Data":"ba04c73f502d01fac0691e95a04d6d265d98842eda58f7f31d36fa1094dac21e"} Nov 02 18:30:54.863472 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.863396 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns" event={"ID":"64a10854-4c5f-446a-ac7e-8d12767fa1bf","Type":"ContainerDied","Data":"40a13bc9153566dd11bff08086171660162688b6d05b5c5cce4ee300e0dacd5a"} Nov 02 18:30:54.875204 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.875125 5549 scope.go:122] "RemoveContainer" containerID="a409a9c8a19c7681b4119ebebf449ec21a4cec36702a80b17e61a5c6d573b6ee" Nov 02 18:30:54.877004 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:54.875507 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a409a9c8a19c7681b4119ebebf449ec21a4cec36702a80b17e61a5c6d573b6ee\": container with ID starting with a409a9c8a19c7681b4119ebebf449ec21a4cec36702a80b17e61a5c6d573b6ee not found: ID does not exist" containerID="a409a9c8a19c7681b4119ebebf449ec21a4cec36702a80b17e61a5c6d573b6ee" Nov 02 18:30:54.877004 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.875545 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a409a9c8a19c7681b4119ebebf449ec21a4cec36702a80b17e61a5c6d573b6ee"} err="failed to get container status \"a409a9c8a19c7681b4119ebebf449ec21a4cec36702a80b17e61a5c6d573b6ee\": rpc error: code = NotFound desc = could not find container \"a409a9c8a19c7681b4119ebebf449ec21a4cec36702a80b17e61a5c6d573b6ee\": container with ID starting with a409a9c8a19c7681b4119ebebf449ec21a4cec36702a80b17e61a5c6d573b6ee not found: ID does not exist" Nov 02 18:30:54.877004 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.875567 5549 scope.go:122] "RemoveContainer" containerID="ba04c73f502d01fac0691e95a04d6d265d98842eda58f7f31d36fa1094dac21e" Nov 02 18:30:54.885082 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.885061 5549 scope.go:122] "RemoveContainer" containerID="ba04c73f502d01fac0691e95a04d6d265d98842eda58f7f31d36fa1094dac21e" Nov 02 18:30:54.885342 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:30:54.885320 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba04c73f502d01fac0691e95a04d6d265d98842eda58f7f31d36fa1094dac21e\": container with ID starting with ba04c73f502d01fac0691e95a04d6d265d98842eda58f7f31d36fa1094dac21e not found: ID does not exist" containerID="ba04c73f502d01fac0691e95a04d6d265d98842eda58f7f31d36fa1094dac21e" Nov 02 18:30:54.885409 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.885348 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba04c73f502d01fac0691e95a04d6d265d98842eda58f7f31d36fa1094dac21e"} err="failed to get container status \"ba04c73f502d01fac0691e95a04d6d265d98842eda58f7f31d36fa1094dac21e\": rpc error: code = NotFound desc = could not find container \"ba04c73f502d01fac0691e95a04d6d265d98842eda58f7f31d36fa1094dac21e\": container with ID starting with ba04c73f502d01fac0691e95a04d6d265d98842eda58f7f31d36fa1094dac21e not found: ID does not exist" Nov 02 18:30:54.890758 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.890736 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns"] Nov 02 18:30:54.896205 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.896184 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/distinct-w2-66eed1-kserve-bcf9f794-7pvns"] Nov 02 18:30:54.910919 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.910782 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:54.922095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.918765 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7"] Nov 02 18:30:54.922095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:54.920494 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/premium-w2-66eed1-kserve-7864b56cf-4jmh7"] Nov 02 18:30:55.735657 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:55.729440 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:56.320467 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:56.320421 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" path="/var/lib/kubelet/pods/6296c79a-bf7c-40ff-b2d2-37d3d9f46309/volumes" Nov 02 18:30:56.321048 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:56.321025 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" path="/var/lib/kubelet/pods/64a10854-4c5f-446a-ac7e-8d12767fa1bf/volumes" Nov 02 18:30:56.419292 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:56.417731 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:57.121565 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:57.121521 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:57.740545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:57.740507 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:58.419975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:58.418740 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:59.066339 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:59.066251 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:30:59.726721 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:30:59.726678 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:00.180221 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.180127 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6"] Nov 02 18:31:00.180452 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.180417 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" podUID="71734cf2-241f-47d9-ba84-c4fededcda62" containerName="maas-api" containerID="cri-o://ccc7e107a710dea5ccbbe3f376b0300c721ac3ac850eab232508c922f9f95f7c" gracePeriod=30 Nov 02 18:31:00.359985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.358606 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:00.493464 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.493436 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 18:31:00.550993 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.550936 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71734cf2-241f-47d9-ba84-c4fededcda62-kube-api-access-w45zt\" (UniqueName: \"kubernetes.io/projected/71734cf2-241f-47d9-ba84-c4fededcda62-kube-api-access-w45zt\") pod \"71734cf2-241f-47d9-ba84-c4fededcda62\" (UID: \"71734cf2-241f-47d9-ba84-c4fededcda62\") " Nov 02 18:31:00.551183 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.551088 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71734cf2-241f-47d9-ba84-c4fededcda62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71734cf2-241f-47d9-ba84-c4fededcda62-maas-api-tls\") pod \"71734cf2-241f-47d9-ba84-c4fededcda62\" (UID: \"71734cf2-241f-47d9-ba84-c4fededcda62\") " Nov 02 18:31:00.551260 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.551210 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71734cf2-241f-47d9-ba84-c4fededcda62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71734cf2-241f-47d9-ba84-c4fededcda62-metrics-certs\") pod \"71734cf2-241f-47d9-ba84-c4fededcda62\" (UID: \"71734cf2-241f-47d9-ba84-c4fededcda62\") " Nov 02 18:31:00.551320 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.551259 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71734cf2-241f-47d9-ba84-c4fededcda62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71734cf2-241f-47d9-ba84-c4fededcda62-openshift-service-ca\") pod \"71734cf2-241f-47d9-ba84-c4fededcda62\" (UID: \"71734cf2-241f-47d9-ba84-c4fededcda62\") " Nov 02 18:31:00.551674 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.551646 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71734cf2-241f-47d9-ba84-c4fededcda62-openshift-service-ca" pod "71734cf2-241f-47d9-ba84-c4fededcda62" (UID: "71734cf2-241f-47d9-ba84-c4fededcda62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:31:00.551841 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.551812 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71734cf2-241f-47d9-ba84-c4fededcda62-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:31:00.553272 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.553245 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71734cf2-241f-47d9-ba84-c4fededcda62-kube-api-access-w45zt" pod "71734cf2-241f-47d9-ba84-c4fededcda62" (UID: "71734cf2-241f-47d9-ba84-c4fededcda62"). InnerVolumeSpecName "kube-api-access-w45zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:31:00.553409 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.553389 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71734cf2-241f-47d9-ba84-c4fededcda62-metrics-certs" pod "71734cf2-241f-47d9-ba84-c4fededcda62" (UID: "71734cf2-241f-47d9-ba84-c4fededcda62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:31:00.553627 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.553611 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71734cf2-241f-47d9-ba84-c4fededcda62-maas-api-tls" pod "71734cf2-241f-47d9-ba84-c4fededcda62" (UID: "71734cf2-241f-47d9-ba84-c4fededcda62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:31:00.564217 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.564194 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:00.652535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.652499 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w45zt\" (UniqueName: \"kubernetes.io/projected/71734cf2-241f-47d9-ba84-c4fededcda62-kube-api-access-w45zt\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:31:00.652535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.652531 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71734cf2-241f-47d9-ba84-c4fededcda62-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:31:00.652535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.652544 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71734cf2-241f-47d9-ba84-c4fededcda62-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:31:00.694361 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.694327 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:00.904234 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.904144 5549 generic.go:356] "Generic (PLEG): container finished" podID="71734cf2-241f-47d9-ba84-c4fededcda62" containerID="ccc7e107a710dea5ccbbe3f376b0300c721ac3ac850eab232508c922f9f95f7c" exitCode=0 Nov 02 18:31:00.904234 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.904188 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" Nov 02 18:31:00.904737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.904229 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" event={"ID":"71734cf2-241f-47d9-ba84-c4fededcda62","Type":"ContainerDied","Data":"ccc7e107a710dea5ccbbe3f376b0300c721ac3ac850eab232508c922f9f95f7c"} Nov 02 18:31:00.904737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.904274 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6" event={"ID":"71734cf2-241f-47d9-ba84-c4fededcda62","Type":"ContainerDied","Data":"ae5eca0cbeb63bf9415f7e46d88e55ede6a041038a9bba855e1d04173f4a92bf"} Nov 02 18:31:00.904737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.904296 5549 scope.go:122] "RemoveContainer" containerID="ccc7e107a710dea5ccbbe3f376b0300c721ac3ac850eab232508c922f9f95f7c" Nov 02 18:31:00.915252 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.915232 5549 scope.go:122] "RemoveContainer" containerID="ccc7e107a710dea5ccbbe3f376b0300c721ac3ac850eab232508c922f9f95f7c" Nov 02 18:31:00.915545 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:31:00.915524 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccc7e107a710dea5ccbbe3f376b0300c721ac3ac850eab232508c922f9f95f7c\": container with ID starting with ccc7e107a710dea5ccbbe3f376b0300c721ac3ac850eab232508c922f9f95f7c not found: ID does not exist" containerID="ccc7e107a710dea5ccbbe3f376b0300c721ac3ac850eab232508c922f9f95f7c" Nov 02 18:31:00.915617 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.915558 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccc7e107a710dea5ccbbe3f376b0300c721ac3ac850eab232508c922f9f95f7c"} err="failed to get container status \"ccc7e107a710dea5ccbbe3f376b0300c721ac3ac850eab232508c922f9f95f7c\": rpc error: code = NotFound desc = could not find container \"ccc7e107a710dea5ccbbe3f376b0300c721ac3ac850eab232508c922f9f95f7c\": container with ID starting with ccc7e107a710dea5ccbbe3f376b0300c721ac3ac850eab232508c922f9f95f7c not found: ID does not exist" Nov 02 18:31:00.926933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.926900 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6"] Nov 02 18:31:00.933121 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:00.933091 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-66eed1-56c7b5644-hkrr6"] Nov 02 18:31:01.317837 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:01.317621 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:01.758171 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:01.758131 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" podUID="fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:31:01.758348 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:01.758130 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" podUID="f5c74aae-fd45-4ba6-b30d-99fb59e22c11" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:31:02.021795 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:02.018275 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:02.320200 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:02.320123 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71734cf2-241f-47d9-ba84-c4fededcda62" path="/var/lib/kubelet/pods/71734cf2-241f-47d9-ba84-c4fededcda62/volumes" Nov 02 18:31:02.708741 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:02.708654 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:02.917588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:02.917550 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:03.392239 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:03.392197 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:04.000979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:03.997974 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:04.706165 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:04.706126 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:05.302604 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:05.301965 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:05.904304 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:05.904268 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:06.375103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:06.375067 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:31:06.631747 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:06.631660 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:06.770855 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:06.770826 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:07.503561 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:07.503525 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:08.337905 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:08.337864 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:09.105564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:09.105472 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:09.818343 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:09.818299 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:10.017198 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.017164 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:10.509409 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.509384 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" Nov 02 18:31:10.549723 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.549692 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" Nov 02 18:31:10.591632 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.591601 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb-plugins-config-volume\") pod \"fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb\" (UID: \"fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb\") " Nov 02 18:31:10.591834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.591740 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb-kube-api-access-7j9dq\" (UniqueName: \"kubernetes.io/projected/fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb-kube-api-access-7j9dq\") pod \"fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb\" (UID: \"fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb\") " Nov 02 18:31:10.591834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.591803 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5c74aae-fd45-4ba6-b30d-99fb59e22c11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c74aae-fd45-4ba6-b30d-99fb59e22c11-plugins-config-volume\") pod \"f5c74aae-fd45-4ba6-b30d-99fb59e22c11\" (UID: \"f5c74aae-fd45-4ba6-b30d-99fb59e22c11\") " Nov 02 18:31:10.591987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.591841 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5c74aae-fd45-4ba6-b30d-99fb59e22c11-kube-api-access-qj5m4\" (UniqueName: \"kubernetes.io/projected/f5c74aae-fd45-4ba6-b30d-99fb59e22c11-kube-api-access-qj5m4\") pod \"f5c74aae-fd45-4ba6-b30d-99fb59e22c11\" (UID: \"f5c74aae-fd45-4ba6-b30d-99fb59e22c11\") " Nov 02 18:31:10.592096 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.592067 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb-plugins-config-volume" pod "fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb" (UID: "fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:31:10.592334 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.592309 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c74aae-fd45-4ba6-b30d-99fb59e22c11-plugins-config-volume" pod "f5c74aae-fd45-4ba6-b30d-99fb59e22c11" (UID: "f5c74aae-fd45-4ba6-b30d-99fb59e22c11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:31:10.594082 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.593638 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:31:10.594082 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.593667 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c74aae-fd45-4ba6-b30d-99fb59e22c11-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:31:10.594513 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.594460 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c74aae-fd45-4ba6-b30d-99fb59e22c11-kube-api-access-qj5m4" pod "f5c74aae-fd45-4ba6-b30d-99fb59e22c11" (UID: "f5c74aae-fd45-4ba6-b30d-99fb59e22c11"). InnerVolumeSpecName "kube-api-access-qj5m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:31:10.606711 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.606664 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb-kube-api-access-7j9dq" pod "fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb" (UID: "fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb"). InnerVolumeSpecName "kube-api-access-7j9dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:31:10.695409 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.695013 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j9dq\" (UniqueName: \"kubernetes.io/projected/fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb-kube-api-access-7j9dq\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:31:10.695717 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.695673 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj5m4\" (UniqueName: \"kubernetes.io/projected/f5c74aae-fd45-4ba6-b30d-99fb59e22c11-kube-api-access-qj5m4\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:31:10.698051 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.698020 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:10.968261 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.968227 5549 generic.go:356] "Generic (PLEG): container finished" podID="f5c74aae-fd45-4ba6-b30d-99fb59e22c11" containerID="dda7dc55e039bb6afaebb05a9d41917182af703242d326dae7f73b14001ed63d" exitCode=0 Nov 02 18:31:10.968458 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.968288 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" Nov 02 18:31:10.968458 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.968315 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" event={"ID":"f5c74aae-fd45-4ba6-b30d-99fb59e22c11","Type":"ContainerDied","Data":"dda7dc55e039bb6afaebb05a9d41917182af703242d326dae7f73b14001ed63d"} Nov 02 18:31:10.968458 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.968402 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6" event={"ID":"f5c74aae-fd45-4ba6-b30d-99fb59e22c11","Type":"ContainerDied","Data":"e62748934f70e2f78b7caf29748441c1d3abf0ae35c0089cf1269c1b01c894dd"} Nov 02 18:31:10.968458 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.968430 5549 scope.go:122] "RemoveContainer" containerID="dda7dc55e039bb6afaebb05a9d41917182af703242d326dae7f73b14001ed63d" Nov 02 18:31:10.969753 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.969732 5549 generic.go:356] "Generic (PLEG): container finished" podID="fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb" containerID="eea96c58c7cc8e36a7c079b01efa5ae509f1f7248782db5214ce113f6e146a5c" exitCode=0 Nov 02 18:31:10.969975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.969761 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" event={"ID":"fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb","Type":"ContainerDied","Data":"eea96c58c7cc8e36a7c079b01efa5ae509f1f7248782db5214ce113f6e146a5c"} Nov 02 18:31:10.970082 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.970010 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" event={"ID":"fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb","Type":"ContainerDied","Data":"971446573f40c5894508a9bbf9bbd9604736158480777bcee6411f8dda26ab97"} Nov 02 18:31:10.970082 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.969793 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97" Nov 02 18:31:10.980720 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.980697 5549 scope.go:122] "RemoveContainer" containerID="dda7dc55e039bb6afaebb05a9d41917182af703242d326dae7f73b14001ed63d" Nov 02 18:31:10.981031 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:31:10.980976 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda7dc55e039bb6afaebb05a9d41917182af703242d326dae7f73b14001ed63d\": container with ID starting with dda7dc55e039bb6afaebb05a9d41917182af703242d326dae7f73b14001ed63d not found: ID does not exist" containerID="dda7dc55e039bb6afaebb05a9d41917182af703242d326dae7f73b14001ed63d" Nov 02 18:31:10.981031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.981007 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda7dc55e039bb6afaebb05a9d41917182af703242d326dae7f73b14001ed63d"} err="failed to get container status \"dda7dc55e039bb6afaebb05a9d41917182af703242d326dae7f73b14001ed63d\": rpc error: code = NotFound desc = could not find container \"dda7dc55e039bb6afaebb05a9d41917182af703242d326dae7f73b14001ed63d\": container with ID starting with dda7dc55e039bb6afaebb05a9d41917182af703242d326dae7f73b14001ed63d not found: ID does not exist" Nov 02 18:31:10.981174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.981032 5549 scope.go:122] "RemoveContainer" containerID="eea96c58c7cc8e36a7c079b01efa5ae509f1f7248782db5214ce113f6e146a5c" Nov 02 18:31:10.993653 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.993630 5549 scope.go:122] "RemoveContainer" containerID="eea96c58c7cc8e36a7c079b01efa5ae509f1f7248782db5214ce113f6e146a5c" Nov 02 18:31:10.994301 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:31:10.994277 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea96c58c7cc8e36a7c079b01efa5ae509f1f7248782db5214ce113f6e146a5c\": container with ID starting with eea96c58c7cc8e36a7c079b01efa5ae509f1f7248782db5214ce113f6e146a5c not found: ID does not exist" containerID="eea96c58c7cc8e36a7c079b01efa5ae509f1f7248782db5214ce113f6e146a5c" Nov 02 18:31:10.994393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.994307 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea96c58c7cc8e36a7c079b01efa5ae509f1f7248782db5214ce113f6e146a5c"} err="failed to get container status \"eea96c58c7cc8e36a7c079b01efa5ae509f1f7248782db5214ce113f6e146a5c\": rpc error: code = NotFound desc = could not find container \"eea96c58c7cc8e36a7c079b01efa5ae509f1f7248782db5214ce113f6e146a5c\": container with ID starting with eea96c58c7cc8e36a7c079b01efa5ae509f1f7248782db5214ce113f6e146a5c not found: ID does not exist" Nov 02 18:31:10.997013 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.996988 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6"] Nov 02 18:31:10.999442 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:10.999420 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-66eed1-68445545c7-hndg6"] Nov 02 18:31:11.009764 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:11.009729 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97"] Nov 02 18:31:11.011803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:11.011781 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-66eed1-54f947d84c-r4b97"] Nov 02 18:31:11.559811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:11.559727 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:11.723257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:11.723222 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:12.229753 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:12.229714 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:12.319586 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:12.319548 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5c74aae-fd45-4ba6-b30d-99fb59e22c11" path="/var/lib/kubelet/pods/f5c74aae-fd45-4ba6-b30d-99fb59e22c11/volumes" Nov 02 18:31:12.319998 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:12.319983 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb" path="/var/lib/kubelet/pods/fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb/volumes" Nov 02 18:31:12.928285 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:12.928247 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:13.664635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:13.664597 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:14.338557 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:14.338484 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:15.148366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:15.148330 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:15.750628 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:15.750599 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:15.878773 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:15.878722 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:16.008705 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:16.008610 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:16.726199 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:16.726154 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:17.586254 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:31:17.586216 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:31:17.617025 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.616932 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:17.755393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.755357 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:17.874289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874199 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf"] Nov 02 18:31:17.874796 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874779 5549 cpu_manager.go:414] "containerMap: removing container" podUID="5d3ac9a6-4ae1-43f9-acdc-45a36e893708" containerName="cleanup" Nov 02 18:31:17.874796 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874793 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3ac9a6-4ae1-43f9-acdc-45a36e893708" containerName="cleanup" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874809 5549 cpu_manager.go:414] "containerMap: removing container" podUID="060e829c-0a72-48a9-b9db-13db919d875f" containerName="payload-pre-processing" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874815 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060e829c-0a72-48a9-b9db-13db919d875f" containerName="payload-pre-processing" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874825 5549 cpu_manager.go:414] "containerMap: removing container" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874829 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874838 5549 cpu_manager.go:414] "containerMap: removing container" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874844 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874853 5549 cpu_manager.go:414] "containerMap: removing container" podUID="6fb1f539-186f-4f47-9ca1-abf514ef710e" containerName="maas-api" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874858 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fb1f539-186f-4f47-9ca1-abf514ef710e" containerName="maas-api" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874868 5549 cpu_manager.go:414] "containerMap: removing container" podUID="dd458c81-9d15-45ba-92b1-473e1378c087" containerName="istio-proxy" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874872 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd458c81-9d15-45ba-92b1-473e1378c087" containerName="istio-proxy" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874879 5549 cpu_manager.go:414] "containerMap: removing container" podUID="f319a273-37c3-41d3-9aa1-14df989794db" containerName="cleanup" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874883 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f319a273-37c3-41d3-9aa1-14df989794db" containerName="cleanup" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874892 5549 cpu_manager.go:414] "containerMap: removing container" podUID="ef601c1e-13de-4586-8c17-2847fa6abdbe" containerName="cleanup" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874897 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef601c1e-13de-4586-8c17-2847fa6abdbe" containerName="cleanup" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874906 5549 cpu_manager.go:414] "containerMap: removing container" podUID="f319a273-37c3-41d3-9aa1-14df989794db" containerName="cleanup" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874920 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f319a273-37c3-41d3-9aa1-14df989794db" containerName="cleanup" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874932 5549 cpu_manager.go:414] "containerMap: removing container" podUID="5897373f-0b89-4603-9b04-a99f0f23582b" containerName="payload-processing" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874937 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5897373f-0b89-4603-9b04-a99f0f23582b" containerName="payload-processing" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874947 5549 cpu_manager.go:414] "containerMap: removing container" podUID="f319a273-37c3-41d3-9aa1-14df989794db" containerName="cleanup" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874969 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f319a273-37c3-41d3-9aa1-14df989794db" containerName="cleanup" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874976 5549 cpu_manager.go:414] "containerMap: removing container" podUID="924d0e07-2010-4007-8148-5b7623447adb" containerName="payload-processing" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874981 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924d0e07-2010-4007-8148-5b7623447adb" containerName="payload-processing" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874988 5549 cpu_manager.go:414] "containerMap: removing container" podUID="fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb" containerName="payload-pre-processing" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874992 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb" containerName="payload-pre-processing" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.874999 5549 cpu_manager.go:414] "containerMap: removing container" podUID="6d0ffd57-6326-4c50-aea7-36e52d0e40d0" containerName="payload-pre-processing" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875004 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0ffd57-6326-4c50-aea7-36e52d0e40d0" containerName="payload-pre-processing" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875014 5549 cpu_manager.go:414] "containerMap: removing container" podUID="3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0" containerName="payload-processing" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875019 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0" containerName="payload-processing" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875028 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7af43d18-1d68-45cb-839f-bc220435b550" containerName="istio-proxy" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875032 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af43d18-1d68-45cb-839f-bc220435b550" containerName="istio-proxy" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875041 5549 cpu_manager.go:414] "containerMap: removing container" podUID="5e400bab-ea24-4848-92ba-c770160096b8" containerName="payload-pre-processing" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875045 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e400bab-ea24-4848-92ba-c770160096b8" containerName="payload-pre-processing" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875051 5549 cpu_manager.go:414] "containerMap: removing container" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875055 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875063 5549 cpu_manager.go:414] "containerMap: removing container" podUID="ef601c1e-13de-4586-8c17-2847fa6abdbe" containerName="cleanup" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875067 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef601c1e-13de-4586-8c17-2847fa6abdbe" containerName="cleanup" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875075 5549 cpu_manager.go:414] "containerMap: removing container" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875079 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875087 5549 cpu_manager.go:414] "containerMap: removing container" podUID="f5c74aae-fd45-4ba6-b30d-99fb59e22c11" containerName="payload-processing" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875091 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c74aae-fd45-4ba6-b30d-99fb59e22c11" containerName="payload-processing" Nov 02 18:31:17.875068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875098 5549 cpu_manager.go:414] "containerMap: removing container" podUID="71734cf2-241f-47d9-ba84-c4fededcda62" containerName="maas-api" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875103 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71734cf2-241f-47d9-ba84-c4fededcda62" containerName="maas-api" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875110 5549 cpu_manager.go:414] "containerMap: removing container" podUID="5a3d4c2f-d60c-41dd-af49-4923ee32729e" containerName="payload-pre-processing" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875114 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a3d4c2f-d60c-41dd-af49-4923ee32729e" containerName="payload-pre-processing" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875122 5549 cpu_manager.go:414] "containerMap: removing container" podUID="5d3ac9a6-4ae1-43f9-acdc-45a36e893708" containerName="cleanup" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875126 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3ac9a6-4ae1-43f9-acdc-45a36e893708" containerName="cleanup" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875220 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e400bab-ea24-4848-92ba-c770160096b8" containerName="payload-pre-processing" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875227 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a10854-4c5f-446a-ac7e-8d12767fa1bf" containerName="main" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875234 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef601c1e-13de-4586-8c17-2847fa6abdbe" containerName="cleanup" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875240 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa67943-f0d5-401a-a6fa-79be9f6bdb95" containerName="main" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875247 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c74aae-fd45-4ba6-b30d-99fb59e22c11" containerName="payload-processing" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875252 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="71734cf2-241f-47d9-ba84-c4fededcda62" containerName="maas-api" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875257 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a3d4c2f-d60c-41dd-af49-4923ee32729e" containerName="payload-pre-processing" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875262 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3ac9a6-4ae1-43f9-acdc-45a36e893708" containerName="cleanup" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875269 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3ac9a6-4ae1-43f9-acdc-45a36e893708" containerName="cleanup" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875274 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="060e829c-0a72-48a9-b9db-13db919d875f" containerName="payload-pre-processing" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875279 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="186b2aa4-3dd7-4329-bbcf-72dd270c371d" containerName="main" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875285 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6296c79a-bf7c-40ff-b2d2-37d3d9f46309" containerName="main" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875291 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fb1f539-186f-4f47-9ca1-abf514ef710e" containerName="maas-api" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875296 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd458c81-9d15-45ba-92b1-473e1378c087" containerName="istio-proxy" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875300 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="f319a273-37c3-41d3-9aa1-14df989794db" containerName="cleanup" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875306 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef601c1e-13de-4586-8c17-2847fa6abdbe" containerName="cleanup" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875311 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="f319a273-37c3-41d3-9aa1-14df989794db" containerName="cleanup" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875316 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5897373f-0b89-4603-9b04-a99f0f23582b" containerName="payload-processing" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875322 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="924d0e07-2010-4007-8148-5b7623447adb" containerName="payload-processing" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875327 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd37ac93-1d69-49a9-82b0-e5b77d3ebcfb" containerName="payload-pre-processing" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875333 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0ffd57-6326-4c50-aea7-36e52d0e40d0" containerName="payload-pre-processing" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875339 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef601c1e-13de-4586-8c17-2847fa6abdbe" containerName="cleanup" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875345 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f8e6a76-9bdc-47d3-9e0f-cef5d546c3d0" containerName="payload-processing" Nov 02 18:31:17.877298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.875351 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af43d18-1d68-45cb-839f-bc220435b550" containerName="istio-proxy" Nov 02 18:31:17.880576 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.880553 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:17.885939 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.885917 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-47e1c2\"/\"kube-root-ca.crt\"" Nov 02 18:31:17.886241 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.886193 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-47e1c2\"/\"openshift-service-ca.crt\"" Nov 02 18:31:17.886509 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.886490 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-47e1c2\"/\"sub-test-model-47e1c2-kserve-self-signed-certs\"" Nov 02 18:31:17.888113 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.888089 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-47e1c2\"/\"default-dockercfg-84qnw\"" Nov 02 18:31:17.889001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.888979 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf"] Nov 02 18:31:17.904964 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.904917 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:17.982599 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.982565 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-tmp-dir\") pod \"sub-test-model-47e1c2-kserve-794d8b89bf-n56qf\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:17.982778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.982611 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-dshm\") pod \"sub-test-model-47e1c2-kserve-794d8b89bf-n56qf\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:17.982778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.982686 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a575e3e4-796c-4183-9755-8d0f73d233ca-tls-certs\") pod \"sub-test-model-47e1c2-kserve-794d8b89bf-n56qf\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:17.982865 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.982803 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5hn5\" (UniqueName: \"kubernetes.io/projected/a575e3e4-796c-4183-9755-8d0f73d233ca-kube-api-access-b5hn5\") pod \"sub-test-model-47e1c2-kserve-794d8b89bf-n56qf\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:17.982902 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.982869 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-model-cache\") pod \"sub-test-model-47e1c2-kserve-794d8b89bf-n56qf\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:17.983022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:17.983007 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-home\") pod \"sub-test-model-47e1c2-kserve-794d8b89bf-n56qf\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:18.045561 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:18.045510 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:18.083929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:18.083889 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-home\") pod \"sub-test-model-47e1c2-kserve-794d8b89bf-n56qf\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:18.084154 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:18.084017 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-tmp-dir\") pod \"sub-test-model-47e1c2-kserve-794d8b89bf-n56qf\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:18.084154 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:18.084047 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-dshm\") pod \"sub-test-model-47e1c2-kserve-794d8b89bf-n56qf\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:18.084154 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:18.084067 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a575e3e4-796c-4183-9755-8d0f73d233ca-tls-certs\") pod \"sub-test-model-47e1c2-kserve-794d8b89bf-n56qf\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:18.084338 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:18.084227 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5hn5\" (UniqueName: \"kubernetes.io/projected/a575e3e4-796c-4183-9755-8d0f73d233ca-kube-api-access-b5hn5\") pod \"sub-test-model-47e1c2-kserve-794d8b89bf-n56qf\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:18.084338 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:18.084322 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-model-cache\") pod \"sub-test-model-47e1c2-kserve-794d8b89bf-n56qf\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:18.084453 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:18.084418 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-home\") pod \"sub-test-model-47e1c2-kserve-794d8b89bf-n56qf\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:18.084499 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:18.084447 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-tmp-dir\") pod \"sub-test-model-47e1c2-kserve-794d8b89bf-n56qf\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:18.084630 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:18.084610 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-model-cache\") pod \"sub-test-model-47e1c2-kserve-794d8b89bf-n56qf\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:18.086733 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:18.086704 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-dshm\") pod \"sub-test-model-47e1c2-kserve-794d8b89bf-n56qf\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:18.087040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:18.086845 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a575e3e4-796c-4183-9755-8d0f73d233ca-tls-certs\") pod \"sub-test-model-47e1c2-kserve-794d8b89bf-n56qf\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:18.096698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:18.096676 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5hn5\" (UniqueName: \"kubernetes.io/projected/a575e3e4-796c-4183-9755-8d0f73d233ca-kube-api-access-b5hn5\") pod \"sub-test-model-47e1c2-kserve-794d8b89bf-n56qf\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:18.199479 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:18.199443 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:18.218106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:18.218073 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:18.564292 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:18.564260 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf"] Nov 02 18:31:18.566132 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:31:18.566106 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda575e3e4_796c_4183_9755_8d0f73d233ca.slice/crio-de68552a94eabf7a2b957f5bafdbb08578e45ad8293251397127675e3449a282 WatchSource:0}: Error finding container de68552a94eabf7a2b957f5bafdbb08578e45ad8293251397127675e3449a282: Status 404 returned error can't find the container with id de68552a94eabf7a2b957f5bafdbb08578e45ad8293251397127675e3449a282 Nov 02 18:31:18.640364 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:18.640330 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:19.036350 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:19.036310 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" event={"ID":"a575e3e4-796c-4183-9755-8d0f73d233ca","Type":"ContainerStarted","Data":"1289e96d58f38b895f84497dbc15c42fbebccd7780b292cd368bd783b3c83650"} Nov 02 18:31:19.036559 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:19.036359 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" event={"ID":"a575e3e4-796c-4183-9755-8d0f73d233ca","Type":"ContainerStarted","Data":"de68552a94eabf7a2b957f5bafdbb08578e45ad8293251397127675e3449a282"} Nov 02 18:31:19.053792 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:19.053747 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" podStartSLOduration=2.053733736 podStartE2EDuration="2.053733736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:31:19.052607805 +0000 UTC m=+3450.841113327" watchObservedRunningTime="2026-11-02 18:31:19.053733736 +0000 UTC m=+3450.842239263" Nov 02 18:31:19.199843 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:19.199799 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:19.300087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:19.299890 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:19.927651 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:19.927607 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:20.589832 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:20.589790 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:21.392333 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:21.392298 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:22.133887 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:22.133825 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:22.825748 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:22.825710 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:23.561505 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:23.561451 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:23.834393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:23.834303 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:23.968227 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:23.968196 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:24.235436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:24.233686 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:25.115592 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:25.115557 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:25.807081 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:25.807043 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:26.575276 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:26.575242 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:27.292696 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:27.292661 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:27.899233 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:27.899164 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:28.199727 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:28.199694 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:28.212675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:28.212649 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:28.520485 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:28.520349 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:29.110319 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:29.110283 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:31:29.199263 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:29.199223 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:29.922256 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:29.921061 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:30.604639 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:30.604563 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:31.219821 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:31.219779 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:31.858134 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:31.858099 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:32.469941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:32.469900 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:33.108268 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:33.108229 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:33.820117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:33.820060 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:34.415769 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:34.415721 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:35.075482 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.075442 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:35.476815 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.476753 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9"] Nov 02 18:31:35.478323 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.478290 5549 cpu_manager.go:414] "containerMap: removing container" podUID="ef601c1e-13de-4586-8c17-2847fa6abdbe" containerName="cleanup" Nov 02 18:31:35.478323 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.478320 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef601c1e-13de-4586-8c17-2847fa6abdbe" containerName="cleanup" Nov 02 18:31:35.478529 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.478380 5549 cpu_manager.go:414] "containerMap: removing container" podUID="5d3ac9a6-4ae1-43f9-acdc-45a36e893708" containerName="cleanup" Nov 02 18:31:35.478529 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.478391 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3ac9a6-4ae1-43f9-acdc-45a36e893708" containerName="cleanup" Nov 02 18:31:35.478529 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.478495 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3ac9a6-4ae1-43f9-acdc-45a36e893708" containerName="cleanup" Nov 02 18:31:35.478529 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.478513 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="f319a273-37c3-41d3-9aa1-14df989794db" containerName="cleanup" Nov 02 18:31:35.483004 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.482945 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.489406 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.489378 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9"] Nov 02 18:31:35.491873 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.491848 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-9165c3\"/\"kube-root-ca.crt\"" Nov 02 18:31:35.492077 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.491884 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-9165c3\"/\"openshift-service-ca.crt\"" Nov 02 18:31:35.492400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.492374 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-9165c3\"/\"sub-test-model-9165c3-kserve-self-signed-certs\"" Nov 02 18:31:35.492506 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.492480 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-9165c3\"/\"default-dockercfg-njhzl\"" Nov 02 18:31:35.537406 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.537366 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-model-cache\") pod \"sub-test-model-9165c3-kserve-87464848c-xf8n9\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.537560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.537508 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-home\") pod \"sub-test-model-9165c3-kserve-87464848c-xf8n9\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.537560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.537542 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/521107be-a789-485f-8827-235aae17ce7e-tls-certs\") pod \"sub-test-model-9165c3-kserve-87464848c-xf8n9\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.537639 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.537566 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-dshm\") pod \"sub-test-model-9165c3-kserve-87464848c-xf8n9\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.537784 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.537759 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-tmp-dir\") pod \"sub-test-model-9165c3-kserve-87464848c-xf8n9\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.537893 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.537803 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fks4k\" (UniqueName: \"kubernetes.io/projected/521107be-a789-485f-8827-235aae17ce7e-kube-api-access-fks4k\") pod \"sub-test-model-9165c3-kserve-87464848c-xf8n9\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.569982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.569933 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:35.656975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.653679 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-home\") pod \"sub-test-model-9165c3-kserve-87464848c-xf8n9\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.656975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.653743 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/521107be-a789-485f-8827-235aae17ce7e-tls-certs\") pod \"sub-test-model-9165c3-kserve-87464848c-xf8n9\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.656975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.653779 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-dshm\") pod \"sub-test-model-9165c3-kserve-87464848c-xf8n9\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.656975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.653888 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-tmp-dir\") pod \"sub-test-model-9165c3-kserve-87464848c-xf8n9\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.656975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.653923 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fks4k\" (UniqueName: \"kubernetes.io/projected/521107be-a789-485f-8827-235aae17ce7e-kube-api-access-fks4k\") pod \"sub-test-model-9165c3-kserve-87464848c-xf8n9\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.656975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.654073 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-model-cache\") pod \"sub-test-model-9165c3-kserve-87464848c-xf8n9\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.656975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.654575 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-model-cache\") pod \"sub-test-model-9165c3-kserve-87464848c-xf8n9\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.656975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.654924 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-home\") pod \"sub-test-model-9165c3-kserve-87464848c-xf8n9\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.657894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.657866 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-tmp-dir\") pod \"sub-test-model-9165c3-kserve-87464848c-xf8n9\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.678117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.678080 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/521107be-a789-485f-8827-235aae17ce7e-tls-certs\") pod \"sub-test-model-9165c3-kserve-87464848c-xf8n9\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.679321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.678851 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-dshm\") pod \"sub-test-model-9165c3-kserve-87464848c-xf8n9\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.687502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.687470 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fks4k\" (UniqueName: \"kubernetes.io/projected/521107be-a789-485f-8827-235aae17ce7e-kube-api-access-fks4k\") pod \"sub-test-model-9165c3-kserve-87464848c-xf8n9\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.822376 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.822274 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:35.833756 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:35.833687 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:36.112032 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:36.112004 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9"] Nov 02 18:31:36.113772 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:31:36.113742 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod521107be_a789_485f_8827_235aae17ce7e.slice/crio-e5509d282e5b1ddcccdd0cf0fe7f4fa74250c55d9f670a51455af8ecbdb6ba1b WatchSource:0}: Error finding container e5509d282e5b1ddcccdd0cf0fe7f4fa74250c55d9f670a51455af8ecbdb6ba1b: Status 404 returned error can't find the container with id e5509d282e5b1ddcccdd0cf0fe7f4fa74250c55d9f670a51455af8ecbdb6ba1b Nov 02 18:31:36.145068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:36.145041 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" event={"ID":"521107be-a789-485f-8827-235aae17ce7e","Type":"ContainerStarted","Data":"e5509d282e5b1ddcccdd0cf0fe7f4fa74250c55d9f670a51455af8ecbdb6ba1b"} Nov 02 18:31:36.250971 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:36.250905 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:31:36.517434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:36.510259 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:37.152586 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:37.152537 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" event={"ID":"521107be-a789-485f-8827-235aae17ce7e","Type":"ContainerStarted","Data":"ebb58ccca634d87014cc627b0735b4b6ae70aede91c23bfe9c025cefd8c91761"} Nov 02 18:31:37.171373 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:37.171306 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" podStartSLOduration=2.171293093 podStartE2EDuration="2.171293093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:31:37.170438284 +0000 UTC m=+3468.958943813" watchObservedRunningTime="2026-11-02 18:31:37.171293093 +0000 UTC m=+3468.959798621" Nov 02 18:31:37.195738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:37.195710 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:37.810135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:37.810096 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:37.823585 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:37.823554 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:38.526471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:38.526437 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:39.214803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:39.214774 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:40.018742 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:40.018705 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:40.808317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:40.808280 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:41.521986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:41.519942 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:42.264104 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:42.264055 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:42.899414 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:42.899376 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:43.516671 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:43.516634 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:44.165549 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:44.165517 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:44.969286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:44.969228 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:45.619599 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:45.619567 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:45.824003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:45.823944 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:45.836930 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:45.836904 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:46.225347 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:46.225309 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:31:46.286872 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:46.286842 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:47.116849 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:47.116812 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:47.586582 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:31:47.586546 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:31:47.619440 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:47.619403 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:48.236660 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:48.236625 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:48.850967 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:48.850908 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:48.988512 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:48.988481 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:49.514886 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:49.514846 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:50.204610 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:50.204565 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:50.672983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:50.672877 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:31:50.826712 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:50.826676 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:51.604978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:51.604492 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:52.313719 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:52.313683 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:53.010739 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:53.010699 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:53.615291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:53.615256 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:54.400449 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:54.400414 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:55.159916 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:55.159877 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:55.800287 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:55.800234 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:56.427452 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:56.427414 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:57.002231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:57.002191 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:57.602582 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:57.602518 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:58.218499 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:58.218458 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:58.849297 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:58.849240 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:31:59.527388 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:31:59.527353 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:00.285653 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:00.285618 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:00.930976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:00.929263 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:01.587416 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:01.587384 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:02.223675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:02.220980 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:02.929442 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:02.929400 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:03.058241 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:03.058204 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:03.681042 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:03.680475 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:04.363413 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:04.362276 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:04.976479 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:04.976444 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:05.600899 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:05.600859 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:06.295229 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:06.295195 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:06.832242 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:06.832204 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:07.502407 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:07.502316 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:08.127986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:08.125789 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:08.803921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:08.803819 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:09.491039 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:09.489569 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:10.107547 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:10.107501 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:10.847860 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:10.847821 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:32:10.927682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:10.927643 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:11.205504 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:11.205465 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv"] Nov 02 18:32:11.205930 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:11.205677 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" podUID="7066250e-b36d-4065-a5fa-265c358a1257" containerName="istio-proxy" containerID="cri-o://afd5eb9961584b3f0c55e98ffc21c95b2f11bc73130d890aa633a3d413cb3628" gracePeriod=30 Nov 02 18:32:11.709177 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:11.709133 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:12.411525 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:12.411485 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:13.019976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:13.010639 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:13.614733 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:13.614694 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:14.204195 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:14.204157 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:14.909720 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:14.909650 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:15.519066 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:15.519023 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:16.137577 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.137539 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:16.412828 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.412746 5549 generic.go:356] "Generic (PLEG): container finished" podID="7066250e-b36d-4065-a5fa-265c358a1257" containerID="afd5eb9961584b3f0c55e98ffc21c95b2f11bc73130d890aa633a3d413cb3628" exitCode=0 Nov 02 18:32:16.412991 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.412824 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" event={"ID":"7066250e-b36d-4065-a5fa-265c358a1257","Type":"ContainerDied","Data":"afd5eb9961584b3f0c55e98ffc21c95b2f11bc73130d890aa633a3d413cb3628"} Nov 02 18:32:16.465445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.465418 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:32:16.544096 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.544054 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-istio-envoy\") pod \"7066250e-b36d-4065-a5fa-265c358a1257\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " Nov 02 18:32:16.544299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.544123 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7066250e-b36d-4065-a5fa-265c358a1257-kube-api-access-nl87r\" (UniqueName: \"kubernetes.io/projected/7066250e-b36d-4065-a5fa-265c358a1257-kube-api-access-nl87r\") pod \"7066250e-b36d-4065-a5fa-265c358a1257\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " Nov 02 18:32:16.544299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.544171 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-workload-certs\") pod \"7066250e-b36d-4065-a5fa-265c358a1257\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " Nov 02 18:32:16.544299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.544204 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-istio-data\") pod \"7066250e-b36d-4065-a5fa-265c358a1257\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " Nov 02 18:32:16.544299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.544260 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7066250e-b36d-4065-a5fa-265c358a1257-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7066250e-b36d-4065-a5fa-265c358a1257-istiod-ca-cert\") pod \"7066250e-b36d-4065-a5fa-265c358a1257\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " Nov 02 18:32:16.544562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.544317 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7066250e-b36d-4065-a5fa-265c358a1257-istio-token\" (UniqueName: \"kubernetes.io/projected/7066250e-b36d-4065-a5fa-265c358a1257-istio-token\") pod \"7066250e-b36d-4065-a5fa-265c358a1257\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " Nov 02 18:32:16.544562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.544364 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-credential-socket\") pod \"7066250e-b36d-4065-a5fa-265c358a1257\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " Nov 02 18:32:16.544562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.544467 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7066250e-b36d-4065-a5fa-265c358a1257-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7066250e-b36d-4065-a5fa-265c358a1257-istio-podinfo\") pod \"7066250e-b36d-4065-a5fa-265c358a1257\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " Nov 02 18:32:16.544562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.544537 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-workload-socket\") pod \"7066250e-b36d-4065-a5fa-265c358a1257\" (UID: \"7066250e-b36d-4065-a5fa-265c358a1257\") " Nov 02 18:32:16.544562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.544546 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-istio-data" pod "7066250e-b36d-4065-a5fa-265c358a1257" (UID: "7066250e-b36d-4065-a5fa-265c358a1257"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:32:16.544847 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.544757 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-credential-socket" pod "7066250e-b36d-4065-a5fa-265c358a1257" (UID: "7066250e-b36d-4065-a5fa-265c358a1257"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:32:16.544847 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.544784 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7066250e-b36d-4065-a5fa-265c358a1257-istiod-ca-cert" pod "7066250e-b36d-4065-a5fa-265c358a1257" (UID: "7066250e-b36d-4065-a5fa-265c358a1257"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:32:16.544847 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.544792 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-workload-certs" pod "7066250e-b36d-4065-a5fa-265c358a1257" (UID: "7066250e-b36d-4065-a5fa-265c358a1257"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:32:16.545043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.544989 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-workload-socket" pod "7066250e-b36d-4065-a5fa-265c358a1257" (UID: "7066250e-b36d-4065-a5fa-265c358a1257"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:32:16.545133 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.545111 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:16.545202 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.545139 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:16.545202 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.545155 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7066250e-b36d-4065-a5fa-265c358a1257-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:16.545202 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.545169 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:16.546585 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.546557 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-istio-envoy" pod "7066250e-b36d-4065-a5fa-265c358a1257" (UID: "7066250e-b36d-4065-a5fa-265c358a1257"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:32:16.546697 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.546592 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7066250e-b36d-4065-a5fa-265c358a1257-kube-api-access-nl87r" pod "7066250e-b36d-4065-a5fa-265c358a1257" (UID: "7066250e-b36d-4065-a5fa-265c358a1257"). InnerVolumeSpecName "kube-api-access-nl87r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:32:16.547102 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.547080 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7066250e-b36d-4065-a5fa-265c358a1257-istio-token" pod "7066250e-b36d-4065-a5fa-265c358a1257" (UID: "7066250e-b36d-4065-a5fa-265c358a1257"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:32:16.547156 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.547123 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7066250e-b36d-4065-a5fa-265c358a1257-istio-podinfo" pod "7066250e-b36d-4065-a5fa-265c358a1257" (UID: "7066250e-b36d-4065-a5fa-265c358a1257"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:32:16.646029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.645992 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl87r\" (UniqueName: \"kubernetes.io/projected/7066250e-b36d-4065-a5fa-265c358a1257-kube-api-access-nl87r\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:16.646029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.646026 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7066250e-b36d-4065-a5fa-265c358a1257-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:16.646029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.646038 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7066250e-b36d-4065-a5fa-265c358a1257-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:16.646284 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.646047 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:16.646284 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.646056 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7066250e-b36d-4065-a5fa-265c358a1257-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:16.738269 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:16.738149 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:17.429067 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:17.429032 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:17.432204 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:17.432169 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" Nov 02 18:32:17.432349 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:17.432246 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv" event={"ID":"7066250e-b36d-4065-a5fa-265c358a1257","Type":"ContainerDied","Data":"85f9d862895585a0f601e8a21949aa876a34b341712ded022c2d6bf469e3d0b6"} Nov 02 18:32:17.432563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:17.432409 5549 scope.go:122] "RemoveContainer" containerID="afd5eb9961584b3f0c55e98ffc21c95b2f11bc73130d890aa633a3d413cb3628" Nov 02 18:32:17.457259 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:17.457227 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv"] Nov 02 18:32:17.464460 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:17.464433 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wgjhv"] Nov 02 18:32:17.587073 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:32:17.587033 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:32:18.114023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:18.113986 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:18.321077 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:18.321042 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7066250e-b36d-4065-a5fa-265c358a1257" path="/var/lib/kubelet/pods/7066250e-b36d-4065-a5fa-265c358a1257/volumes" Nov 02 18:32:18.795581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:18.795512 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:19.394619 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:19.394582 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:20.087191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:20.087152 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:20.787579 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:20.787543 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:21.409493 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:21.409453 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:22.001807 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:22.001767 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:22.602541 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:22.602491 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:23.196659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:23.196613 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:23.792875 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:23.792647 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:24.202648 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:24.202613 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:24.404145 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:24.404109 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:24.530135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:24.530055 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:25.095480 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:25.095429 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:25.726771 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:25.726728 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:26.388653 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:26.388589 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:27.021548 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:27.021509 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:27.617839 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:27.617801 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:28.220596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:28.219791 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:28.886095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:28.886060 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:29.019519 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:29.019491 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:29.532733 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:29.532686 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:30.309443 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:30.302991 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:30.543905 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:30.543786 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:32:31.115475 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:31.115437 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:31.786986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:31.786398 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:32.403513 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:32.403473 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:33.033732 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:33.033670 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:33.736323 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:33.734127 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:34.402643 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:34.401858 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:35.097648 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:35.097609 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:35.805085 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:35.805023 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:35.844012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:35.843968 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g"] Nov 02 18:32:35.954344 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:35.954307 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:36.508921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:36.508492 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:37.132831 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:37.132794 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:37.276395 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:37.276354 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:37.666010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:37.665967 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:37.815351 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:37.815310 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:38.558568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:38.558530 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:39.284844 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:39.284806 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:39.897587 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:39.895486 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:40.543946 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:40.543876 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:41.264420 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:41.262842 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:41.885671 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:41.885628 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:42.515722 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:42.515682 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:43.208022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:43.204768 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:44.057350 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:44.057309 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:44.232672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:44.232637 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:32:44.353099 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:44.353004 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j"] Nov 02 18:32:44.367727 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:44.367686 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r"] Nov 02 18:32:44.471593 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:44.471559 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:44.669003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:44.668911 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:45.114645 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:45.114610 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:45.249567 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:45.249505 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:45.832928 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:45.832889 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:46.499246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:46.498858 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:47.205253 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:47.205205 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:47.585834 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:32:47.585741 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:32:47.892832 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:47.892667 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:48.403002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:48.402934 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:48.754068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:48.754033 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:49.354146 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:49.354096 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" podUID="b612d0da-2500-4091-a03e-89faeb9c435d" containerName="payload-processing" containerID="cri-o://993e4c03ff25143e6426ddfa51cc5c0be68e31dc9b4ef457b9090f473d463847" gracePeriod=25 Nov 02 18:32:49.368286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:49.368236 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" podUID="31758614-1d84-4256-ba23-aed7dd3dd328" containerName="payload-pre-processing" containerID="cri-o://e6acfb570aa43edbf6e4305690fc7665a2fbf91ece0e23730519ef1c007d2293" gracePeriod=25 Nov 02 18:32:49.399397 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:49.399335 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:50.005234 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:49.999684 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:50.612977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:50.612509 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:50.869511 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:50.869380 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" podUID="e6ddfb19-4075-43de-9fb7-395181b36a25" containerName="main" containerID="cri-o://373e3969e391cff92c6c52d7523ffecf0aa3ea643c404b5cb5cbe0e5a363744d" gracePeriod=45 Nov 02 18:32:51.147125 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.147098 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:32:51.210502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.210457 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-tmp-dir\") pod \"e6ddfb19-4075-43de-9fb7-395181b36a25\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " Nov 02 18:32:51.210761 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.210533 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-dshm\") pod \"e6ddfb19-4075-43de-9fb7-395181b36a25\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " Nov 02 18:32:51.210761 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.210722 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-home\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-home\") pod \"e6ddfb19-4075-43de-9fb7-395181b36a25\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " Nov 02 18:32:51.210908 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.210786 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6ddfb19-4075-43de-9fb7-395181b36a25-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6ddfb19-4075-43de-9fb7-395181b36a25-tls-certs\") pod \"e6ddfb19-4075-43de-9fb7-395181b36a25\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " Nov 02 18:32:51.210908 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.210827 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-model-cache\") pod \"e6ddfb19-4075-43de-9fb7-395181b36a25\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " Nov 02 18:32:51.210908 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.210859 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ddfb19-4075-43de-9fb7-395181b36a25-kube-api-access-tpx6q\" (UniqueName: \"kubernetes.io/projected/e6ddfb19-4075-43de-9fb7-395181b36a25-kube-api-access-tpx6q\") pod \"e6ddfb19-4075-43de-9fb7-395181b36a25\" (UID: \"e6ddfb19-4075-43de-9fb7-395181b36a25\") " Nov 02 18:32:51.211244 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.211219 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-home" pod "e6ddfb19-4075-43de-9fb7-395181b36a25" (UID: "e6ddfb19-4075-43de-9fb7-395181b36a25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:32:51.211458 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.211437 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:51.212170 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.211715 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-tmp-dir" pod "e6ddfb19-4075-43de-9fb7-395181b36a25" (UID: "e6ddfb19-4075-43de-9fb7-395181b36a25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:32:51.212583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.212543 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-model-cache" pod "e6ddfb19-4075-43de-9fb7-395181b36a25" (UID: "e6ddfb19-4075-43de-9fb7-395181b36a25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:32:51.213079 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.213054 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:51.214086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.214057 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ddfb19-4075-43de-9fb7-395181b36a25-kube-api-access-tpx6q" pod "e6ddfb19-4075-43de-9fb7-395181b36a25" (UID: "e6ddfb19-4075-43de-9fb7-395181b36a25"). InnerVolumeSpecName "kube-api-access-tpx6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:32:51.214463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.214434 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6ddfb19-4075-43de-9fb7-395181b36a25-tls-certs" pod "e6ddfb19-4075-43de-9fb7-395181b36a25" (UID: "e6ddfb19-4075-43de-9fb7-395181b36a25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:32:51.215297 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.215275 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-dshm" pod "e6ddfb19-4075-43de-9fb7-395181b36a25" (UID: "e6ddfb19-4075-43de-9fb7-395181b36a25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:32:51.313151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.313114 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6ddfb19-4075-43de-9fb7-395181b36a25-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:51.313151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.313144 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:51.313151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.313155 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpx6q\" (UniqueName: \"kubernetes.io/projected/e6ddfb19-4075-43de-9fb7-395181b36a25-kube-api-access-tpx6q\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:51.313413 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.313165 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:51.313413 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.313173 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6ddfb19-4075-43de-9fb7-395181b36a25-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:51.474699 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.474662 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:51.613908 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.613870 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:51.670205 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.670166 5549 generic.go:356] "Generic (PLEG): container finished" podID="e6ddfb19-4075-43de-9fb7-395181b36a25" containerID="373e3969e391cff92c6c52d7523ffecf0aa3ea643c404b5cb5cbe0e5a363744d" exitCode=0 Nov 02 18:32:51.670417 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.670373 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" event={"ID":"e6ddfb19-4075-43de-9fb7-395181b36a25","Type":"ContainerDied","Data":"373e3969e391cff92c6c52d7523ffecf0aa3ea643c404b5cb5cbe0e5a363744d"} Nov 02 18:32:51.670487 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.670413 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" event={"ID":"e6ddfb19-4075-43de-9fb7-395181b36a25","Type":"ContainerDied","Data":"b0f2655c570c3097401695e7d7d8a28780fe804eec857094c9340da2a441e985"} Nov 02 18:32:51.670487 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.670438 5549 scope.go:122] "RemoveContainer" containerID="373e3969e391cff92c6c52d7523ffecf0aa3ea643c404b5cb5cbe0e5a363744d" Nov 02 18:32:51.670623 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.670606 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g" Nov 02 18:32:51.690045 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.690022 5549 scope.go:122] "RemoveContainer" containerID="373e3969e391cff92c6c52d7523ffecf0aa3ea643c404b5cb5cbe0e5a363744d" Nov 02 18:32:51.690354 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:32:51.690334 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373e3969e391cff92c6c52d7523ffecf0aa3ea643c404b5cb5cbe0e5a363744d\": container with ID starting with 373e3969e391cff92c6c52d7523ffecf0aa3ea643c404b5cb5cbe0e5a363744d not found: ID does not exist" containerID="373e3969e391cff92c6c52d7523ffecf0aa3ea643c404b5cb5cbe0e5a363744d" Nov 02 18:32:51.690425 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.690361 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373e3969e391cff92c6c52d7523ffecf0aa3ea643c404b5cb5cbe0e5a363744d"} err="failed to get container status \"373e3969e391cff92c6c52d7523ffecf0aa3ea643c404b5cb5cbe0e5a363744d\": rpc error: code = NotFound desc = could not find container \"373e3969e391cff92c6c52d7523ffecf0aa3ea643c404b5cb5cbe0e5a363744d\": container with ID starting with 373e3969e391cff92c6c52d7523ffecf0aa3ea643c404b5cb5cbe0e5a363744d not found: ID does not exist" Nov 02 18:32:51.705751 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.705712 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" podUID="b612d0da-2500-4091-a03e-89faeb9c435d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:32:51.706291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.706260 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g"] Nov 02 18:32:51.706431 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.706266 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" podUID="31758614-1d84-4256-ba23-aed7dd3dd328" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:32:51.709824 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:51.709800 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1e2281/sub-test-model-1e2281-kserve-546ccc6679-svb5g"] Nov 02 18:32:52.247912 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:52.247878 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:52.320918 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:52.320879 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ddfb19-4075-43de-9fb7-395181b36a25" path="/var/lib/kubelet/pods/e6ddfb19-4075-43de-9fb7-395181b36a25/volumes" Nov 02 18:32:52.383024 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:52.382986 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:52.607985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:52.607887 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:53.015585 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:53.015541 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:53.264963 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:53.264916 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7350-25a722"] Nov 02 18:32:53.265532 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:53.265516 5549 cpu_manager.go:414] "containerMap: removing container" podUID="e6ddfb19-4075-43de-9fb7-395181b36a25" containerName="main" Nov 02 18:32:53.265583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:53.265535 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ddfb19-4075-43de-9fb7-395181b36a25" containerName="main" Nov 02 18:32:53.265583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:53.265573 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7066250e-b36d-4065-a5fa-265c358a1257" containerName="istio-proxy" Nov 02 18:32:53.265583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:53.265578 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7066250e-b36d-4065-a5fa-265c358a1257" containerName="istio-proxy" Nov 02 18:32:53.265707 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:53.265657 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ddfb19-4075-43de-9fb7-395181b36a25" containerName="main" Nov 02 18:32:53.265707 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:53.265672 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7066250e-b36d-4065-a5fa-265c358a1257" containerName="istio-proxy" Nov 02 18:32:53.268928 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:53.268911 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-25a722" Nov 02 18:32:53.274502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:53.274476 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7350-25a722"] Nov 02 18:32:53.307051 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:53.307023 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zf9kl\"" Nov 02 18:32:53.332470 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:53.332440 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:53.348857 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:53.348831 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkkjf\" (UniqueName: \"kubernetes.io/projected/3f949a62-e873-42d7-af58-7f3c67150703-kube-api-access-dkkjf\") pod \"test-curl-7350-25a722\" (UID: \"3f949a62-e873-42d7-af58-7f3c67150703\") " pod="opendatahub/test-curl-7350-25a722" Nov 02 18:32:53.450380 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:53.450339 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkkjf\" (UniqueName: \"kubernetes.io/projected/3f949a62-e873-42d7-af58-7f3c67150703-kube-api-access-dkkjf\") pod \"test-curl-7350-25a722\" (UID: \"3f949a62-e873-42d7-af58-7f3c67150703\") " pod="opendatahub/test-curl-7350-25a722" Nov 02 18:32:53.460994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:53.460936 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkkjf\" (UniqueName: \"kubernetes.io/projected/3f949a62-e873-42d7-af58-7f3c67150703-kube-api-access-dkkjf\") pod \"test-curl-7350-25a722\" (UID: \"3f949a62-e873-42d7-af58-7f3c67150703\") " pod="opendatahub/test-curl-7350-25a722" Nov 02 18:32:53.584490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:53.584402 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-25a722" Nov 02 18:32:53.747224 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:53.747188 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:53.860777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:53.860745 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7350-25a722"] Nov 02 18:32:53.862501 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:32:53.862470 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f949a62_e873_42d7_af58_7f3c67150703.slice/crio-66736db75662187102addf11ac6fadc39d2a1e9fbe30d573174aec77fd6e46ac WatchSource:0}: Error finding container 66736db75662187102addf11ac6fadc39d2a1e9fbe30d573174aec77fd6e46ac: Status 404 returned error can't find the container with id 66736db75662187102addf11ac6fadc39d2a1e9fbe30d573174aec77fd6e46ac Nov 02 18:32:54.337438 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:54.337393 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:54.717478 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:54.717439 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7350-25a722" event={"ID":"3f949a62-e873-42d7-af58-7f3c67150703","Type":"ContainerStarted","Data":"911f41e14846a02500431810b327a4bc770b18e41c9b2b90fa1eabb18e0ae9e1"} Nov 02 18:32:54.717685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:54.717491 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7350-25a722" event={"ID":"3f949a62-e873-42d7-af58-7f3c67150703","Type":"ContainerStarted","Data":"66736db75662187102addf11ac6fadc39d2a1e9fbe30d573174aec77fd6e46ac"} Nov 02 18:32:54.731211 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:54.731139 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7350-25a722" podStartSLOduration=1.7311182029999999 podStartE2EDuration="1.731118203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:32:54.730819115 +0000 UTC m=+3546.519324644" watchObservedRunningTime="2026-11-02 18:32:54.731118203 +0000 UTC m=+3546.519623732" Nov 02 18:32:54.917704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:54.917660 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:55.368348 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:55.368315 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7350-25a722"] Nov 02 18:32:55.372049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:55.372020 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7350-25a722"] Nov 02 18:32:55.519311 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:55.519266 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7350-2ca161"] Nov 02 18:32:55.520008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:55.519987 5549 cpu_manager.go:414] "containerMap: removing container" podUID="3f949a62-e873-42d7-af58-7f3c67150703" containerName="test-curl-7350-25a722" Nov 02 18:32:55.520134 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:55.520008 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f949a62-e873-42d7-af58-7f3c67150703" containerName="test-curl-7350-25a722" Nov 02 18:32:55.520195 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:55.520135 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f949a62-e873-42d7-af58-7f3c67150703" containerName="test-curl-7350-25a722" Nov 02 18:32:55.523599 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:55.523579 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-2ca161" Nov 02 18:32:55.527591 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:55.527560 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7350-2ca161"] Nov 02 18:32:55.614787 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:55.614665 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:55.678652 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:55.678556 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq6jm\" (UniqueName: \"kubernetes.io/projected/12621cb6-273e-4730-85c7-101d872c1b71-kube-api-access-tq6jm\") pod \"test-curl-7350-2ca161\" (UID: \"12621cb6-273e-4730-85c7-101d872c1b71\") " pod="opendatahub/test-curl-7350-2ca161" Nov 02 18:32:55.779642 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:55.779585 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq6jm\" (UniqueName: \"kubernetes.io/projected/12621cb6-273e-4730-85c7-101d872c1b71-kube-api-access-tq6jm\") pod \"test-curl-7350-2ca161\" (UID: \"12621cb6-273e-4730-85c7-101d872c1b71\") " pod="opendatahub/test-curl-7350-2ca161" Nov 02 18:32:55.787053 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:55.787021 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq6jm\" (UniqueName: \"kubernetes.io/projected/12621cb6-273e-4730-85c7-101d872c1b71-kube-api-access-tq6jm\") pod \"test-curl-7350-2ca161\" (UID: \"12621cb6-273e-4730-85c7-101d872c1b71\") " pod="opendatahub/test-curl-7350-2ca161" Nov 02 18:32:55.843575 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:55.843537 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-2ca161" Nov 02 18:32:56.119809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:56.119776 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7350-2ca161"] Nov 02 18:32:56.121345 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:32:56.121316 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12621cb6_273e_4730_85c7_101d872c1b71.slice/crio-a57d29ad531b2b1866fe6f01d1460be0b34008ba0f8e364c192173aeee19ac31 WatchSource:0}: Error finding container a57d29ad531b2b1866fe6f01d1460be0b34008ba0f8e364c192173aeee19ac31: Status 404 returned error can't find the container with id a57d29ad531b2b1866fe6f01d1460be0b34008ba0f8e364c192173aeee19ac31 Nov 02 18:32:56.307332 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:56.307287 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:56.731019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:56.730982 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7350-2ca161" event={"ID":"12621cb6-273e-4730-85c7-101d872c1b71","Type":"ContainerStarted","Data":"73289d633355b128b65021b95164cca3887011fc77674e21e4d7ac8b535133c3"} Nov 02 18:32:56.731019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:56.731030 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7350-2ca161" event={"ID":"12621cb6-273e-4730-85c7-101d872c1b71","Type":"ContainerStarted","Data":"a57d29ad531b2b1866fe6f01d1460be0b34008ba0f8e364c192173aeee19ac31"} Nov 02 18:32:56.731513 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:56.731093 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7350-25a722" podUID="3f949a62-e873-42d7-af58-7f3c67150703" containerName="test-curl-7350-25a722" containerID="cri-o://911f41e14846a02500431810b327a4bc770b18e41c9b2b90fa1eabb18e0ae9e1" gracePeriod=2 Nov 02 18:32:56.745664 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:56.745614 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7350-2ca161" podStartSLOduration=1.7456011070000002 podStartE2EDuration="1.745601107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:32:56.744030581 +0000 UTC m=+3548.532536108" watchObservedRunningTime="2026-11-02 18:32:56.745601107 +0000 UTC m=+3548.534106635" Nov 02 18:32:56.901676 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:56.901638 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:57.362749 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:57.362710 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7350-2ca161"] Nov 02 18:32:57.366892 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:57.366855 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7350-2ca161"] Nov 02 18:32:57.497987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:57.497568 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:58.183519 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:58.183478 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:32:58.219069 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:58.219032 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:58.725453 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:58.725412 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv"] Nov 02 18:32:58.740440 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:58.740159 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw"] Nov 02 18:32:58.746353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:58.746271 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7350-2ca161" podUID="12621cb6-273e-4730-85c7-101d872c1b71" containerName="test-curl-7350-2ca161" containerID="cri-o://73289d633355b128b65021b95164cca3887011fc77674e21e4d7ac8b535133c3" gracePeriod=2 Nov 02 18:32:58.804114 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:58.804071 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:58.945586 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:58.945556 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:32:58.947200 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:58.947177 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-25a722" Nov 02 18:32:59.142384 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:59.142289 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f949a62-e873-42d7-af58-7f3c67150703-kube-api-access-dkkjf\" (UniqueName: \"kubernetes.io/projected/3f949a62-e873-42d7-af58-7f3c67150703-kube-api-access-dkkjf\") pod \"3f949a62-e873-42d7-af58-7f3c67150703\" (UID: \"3f949a62-e873-42d7-af58-7f3c67150703\") " Nov 02 18:32:59.144525 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:59.144495 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f949a62-e873-42d7-af58-7f3c67150703-kube-api-access-dkkjf" pod "3f949a62-e873-42d7-af58-7f3c67150703" (UID: "3f949a62-e873-42d7-af58-7f3c67150703"). InnerVolumeSpecName "kube-api-access-dkkjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:32:59.244159 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:59.244119 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkkjf\" (UniqueName: \"kubernetes.io/projected/3f949a62-e873-42d7-af58-7f3c67150703-kube-api-access-dkkjf\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:59.753350 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:59.753315 5549 generic.go:356] "Generic (PLEG): container finished" podID="3f949a62-e873-42d7-af58-7f3c67150703" containerID="911f41e14846a02500431810b327a4bc770b18e41c9b2b90fa1eabb18e0ae9e1" exitCode=137 Nov 02 18:32:59.753540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:59.753362 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-25a722" Nov 02 18:32:59.753540 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:59.753425 5549 scope.go:122] "RemoveContainer" containerID="911f41e14846a02500431810b327a4bc770b18e41c9b2b90fa1eabb18e0ae9e1" Nov 02 18:32:59.763202 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:59.763181 5549 scope.go:122] "RemoveContainer" containerID="911f41e14846a02500431810b327a4bc770b18e41c9b2b90fa1eabb18e0ae9e1" Nov 02 18:32:59.763489 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:32:59.763468 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"911f41e14846a02500431810b327a4bc770b18e41c9b2b90fa1eabb18e0ae9e1\": container with ID starting with 911f41e14846a02500431810b327a4bc770b18e41c9b2b90fa1eabb18e0ae9e1 not found: ID does not exist" containerID="911f41e14846a02500431810b327a4bc770b18e41c9b2b90fa1eabb18e0ae9e1" Nov 02 18:32:59.763566 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:32:59.763497 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"911f41e14846a02500431810b327a4bc770b18e41c9b2b90fa1eabb18e0ae9e1"} err="failed to get container status \"911f41e14846a02500431810b327a4bc770b18e41c9b2b90fa1eabb18e0ae9e1\": rpc error: code = NotFound desc = could not find container \"911f41e14846a02500431810b327a4bc770b18e41c9b2b90fa1eabb18e0ae9e1\": container with ID starting with 911f41e14846a02500431810b327a4bc770b18e41c9b2b90fa1eabb18e0ae9e1 not found: ID does not exist" Nov 02 18:32:59.764939 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:32:59.764913 5549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7350-25a722\" is forbidden: User \"system:node:ip-10-0-142-133.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-133.us-east-2.compute.internal' and this object" podUID="3f949a62-e873-42d7-af58-7f3c67150703" pod="opendatahub/test-curl-7350-25a722" Nov 02 18:33:00.321635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:00.321601 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f949a62-e873-42d7-af58-7f3c67150703" path="/var/lib/kubelet/pods/3f949a62-e873-42d7-af58-7f3c67150703/volumes" Nov 02 18:33:00.577101 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:00.577011 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:00.711356 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:00.711322 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:00.837531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:00.837415 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:00.965373 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:00.965346 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-2ca161" Nov 02 18:33:00.969647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:00.969627 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12621cb6-273e-4730-85c7-101d872c1b71-kube-api-access-tq6jm\" (UniqueName: \"kubernetes.io/projected/12621cb6-273e-4730-85c7-101d872c1b71-kube-api-access-tq6jm\") pod \"12621cb6-273e-4730-85c7-101d872c1b71\" (UID: \"12621cb6-273e-4730-85c7-101d872c1b71\") " Nov 02 18:33:00.971773 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:00.971747 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12621cb6-273e-4730-85c7-101d872c1b71-kube-api-access-tq6jm" pod "12621cb6-273e-4730-85c7-101d872c1b71" (UID: "12621cb6-273e-4730-85c7-101d872c1b71"). InnerVolumeSpecName "kube-api-access-tq6jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:01.071299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:01.071260 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq6jm\" (UniqueName: \"kubernetes.io/projected/12621cb6-273e-4730-85c7-101d872c1b71-kube-api-access-tq6jm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:01.323210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:01.323172 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:01.670940 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:01.670850 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:01.705498 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:01.705457 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" podUID="31758614-1d84-4256-ba23-aed7dd3dd328" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:33:01.705631 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:01.705457 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" podUID="b612d0da-2500-4091-a03e-89faeb9c435d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:33:01.768593 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:01.768558 5549 generic.go:356] "Generic (PLEG): container finished" podID="12621cb6-273e-4730-85c7-101d872c1b71" containerID="73289d633355b128b65021b95164cca3887011fc77674e21e4d7ac8b535133c3" exitCode=137 Nov 02 18:33:01.768782 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:01.768602 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-2ca161" Nov 02 18:33:01.768782 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:01.768621 5549 scope.go:122] "RemoveContainer" containerID="73289d633355b128b65021b95164cca3887011fc77674e21e4d7ac8b535133c3" Nov 02 18:33:01.778462 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:01.778442 5549 scope.go:122] "RemoveContainer" containerID="73289d633355b128b65021b95164cca3887011fc77674e21e4d7ac8b535133c3" Nov 02 18:33:01.778736 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:33:01.778713 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73289d633355b128b65021b95164cca3887011fc77674e21e4d7ac8b535133c3\": container with ID starting with 73289d633355b128b65021b95164cca3887011fc77674e21e4d7ac8b535133c3 not found: ID does not exist" containerID="73289d633355b128b65021b95164cca3887011fc77674e21e4d7ac8b535133c3" Nov 02 18:33:01.778791 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:01.778748 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73289d633355b128b65021b95164cca3887011fc77674e21e4d7ac8b535133c3"} err="failed to get container status \"73289d633355b128b65021b95164cca3887011fc77674e21e4d7ac8b535133c3\": rpc error: code = NotFound desc = could not find container \"73289d633355b128b65021b95164cca3887011fc77674e21e4d7ac8b535133c3\": container with ID starting with 73289d633355b128b65021b95164cca3887011fc77674e21e4d7ac8b535133c3 not found: ID does not exist" Nov 02 18:33:01.780589 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:33:01.780563 5549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7350-2ca161\" is forbidden: User \"system:node:ip-10-0-142-133.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-133.us-east-2.compute.internal' and this object" podUID="12621cb6-273e-4730-85c7-101d872c1b71" pod="opendatahub/test-curl-7350-2ca161" Nov 02 18:33:02.310867 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:02.309114 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:02.324659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:02.324616 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12621cb6-273e-4730-85c7-101d872c1b71" path="/var/lib/kubelet/pods/12621cb6-273e-4730-85c7-101d872c1b71/volumes" Nov 02 18:33:02.930432 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:02.930393 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:03.165127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:03.165093 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:03.624452 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:03.624410 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:03.725907 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:03.725859 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" podUID="d132c2c1-5fab-4ab4-82f1-892f5aad2771" containerName="payload-processing" containerID="cri-o://39299b653b1bd4eeea63c84ba65a457d0659af67b23358e96a795126a2ea8084" gracePeriod=25 Nov 02 18:33:03.741435 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:03.741391 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" podUID="95748775-ad15-4a15-8031-a0486a2cf317" containerName="payload-pre-processing" containerID="cri-o://b145bf25487f02697495c258d53fef72f69773b77ab077306d89c9e737318600" gracePeriod=25 Nov 02 18:33:04.369673 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:04.369633 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:05.044041 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:05.044000 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:05.225254 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:05.225210 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" podUID="d132c2c1-5fab-4ab4-82f1-892f5aad2771" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:33:05.225433 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:05.225274 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" podUID="95748775-ad15-4a15-8031-a0486a2cf317" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:33:05.702732 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:05.702698 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:06.337414 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:06.337374 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:06.481839 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:06.481797 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:06.647215 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:06.647131 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:07.103109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:07.103073 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:07.299785 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:07.299744 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:07.431507 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:07.431421 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:07.896081 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:07.896042 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:08.044520 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:08.044487 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:08.220160 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:08.220118 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:08.714799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:08.714340 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:08.897316 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:08.897277 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:09.331533 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.324815 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:09.648053 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.647974 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" Nov 02 18:33:09.674708 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.674673 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b612d0da-2500-4091-a03e-89faeb9c435d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b612d0da-2500-4091-a03e-89faeb9c435d-plugins-config-volume\") pod \"b612d0da-2500-4091-a03e-89faeb9c435d\" (UID: \"b612d0da-2500-4091-a03e-89faeb9c435d\") " Nov 02 18:33:09.674877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.674800 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b612d0da-2500-4091-a03e-89faeb9c435d-kube-api-access-bpcgd\" (UniqueName: \"kubernetes.io/projected/b612d0da-2500-4091-a03e-89faeb9c435d-kube-api-access-bpcgd\") pod \"b612d0da-2500-4091-a03e-89faeb9c435d\" (UID: \"b612d0da-2500-4091-a03e-89faeb9c435d\") " Nov 02 18:33:09.675175 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.675141 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b612d0da-2500-4091-a03e-89faeb9c435d-plugins-config-volume" pod "b612d0da-2500-4091-a03e-89faeb9c435d" (UID: "b612d0da-2500-4091-a03e-89faeb9c435d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:33:09.677506 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.677479 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b612d0da-2500-4091-a03e-89faeb9c435d-kube-api-access-bpcgd" pod "b612d0da-2500-4091-a03e-89faeb9c435d" (UID: "b612d0da-2500-4091-a03e-89faeb9c435d"). InnerVolumeSpecName "kube-api-access-bpcgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:09.678633 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.678610 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" Nov 02 18:33:09.776064 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.776025 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31758614-1d84-4256-ba23-aed7dd3dd328-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31758614-1d84-4256-ba23-aed7dd3dd328-plugins-config-volume\") pod \"31758614-1d84-4256-ba23-aed7dd3dd328\" (UID: \"31758614-1d84-4256-ba23-aed7dd3dd328\") " Nov 02 18:33:09.776557 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.776193 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31758614-1d84-4256-ba23-aed7dd3dd328-kube-api-access-qp5tr\" (UniqueName: \"kubernetes.io/projected/31758614-1d84-4256-ba23-aed7dd3dd328-kube-api-access-qp5tr\") pod \"31758614-1d84-4256-ba23-aed7dd3dd328\" (UID: \"31758614-1d84-4256-ba23-aed7dd3dd328\") " Nov 02 18:33:09.776557 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.776418 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31758614-1d84-4256-ba23-aed7dd3dd328-plugins-config-volume" pod "31758614-1d84-4256-ba23-aed7dd3dd328" (UID: "31758614-1d84-4256-ba23-aed7dd3dd328"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:33:09.776750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.776729 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b612d0da-2500-4091-a03e-89faeb9c435d-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:09.776813 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.776755 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpcgd\" (UniqueName: \"kubernetes.io/projected/b612d0da-2500-4091-a03e-89faeb9c435d-kube-api-access-bpcgd\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:09.776813 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.776771 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31758614-1d84-4256-ba23-aed7dd3dd328-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:09.778378 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.778357 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31758614-1d84-4256-ba23-aed7dd3dd328-kube-api-access-qp5tr" pod "31758614-1d84-4256-ba23-aed7dd3dd328" (UID: "31758614-1d84-4256-ba23-aed7dd3dd328"). InnerVolumeSpecName "kube-api-access-qp5tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:09.826439 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.826396 5549 generic.go:356] "Generic (PLEG): container finished" podID="31758614-1d84-4256-ba23-aed7dd3dd328" containerID="e6acfb570aa43edbf6e4305690fc7665a2fbf91ece0e23730519ef1c007d2293" exitCode=0 Nov 02 18:33:09.826625 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.826447 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" Nov 02 18:33:09.826625 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.826478 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" event={"ID":"31758614-1d84-4256-ba23-aed7dd3dd328","Type":"ContainerDied","Data":"e6acfb570aa43edbf6e4305690fc7665a2fbf91ece0e23730519ef1c007d2293"} Nov 02 18:33:09.826625 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.826548 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r" event={"ID":"31758614-1d84-4256-ba23-aed7dd3dd328","Type":"ContainerDied","Data":"35fa50fe64c24a7d87412c4e62a8b03b156c20d3a07f2d239dbc636e485c29a8"} Nov 02 18:33:09.826625 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.826580 5549 scope.go:122] "RemoveContainer" containerID="e6acfb570aa43edbf6e4305690fc7665a2fbf91ece0e23730519ef1c007d2293" Nov 02 18:33:09.828035 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.828010 5549 generic.go:356] "Generic (PLEG): container finished" podID="b612d0da-2500-4091-a03e-89faeb9c435d" containerID="993e4c03ff25143e6426ddfa51cc5c0be68e31dc9b4ef457b9090f473d463847" exitCode=0 Nov 02 18:33:09.828132 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.828041 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" event={"ID":"b612d0da-2500-4091-a03e-89faeb9c435d","Type":"ContainerDied","Data":"993e4c03ff25143e6426ddfa51cc5c0be68e31dc9b4ef457b9090f473d463847"} Nov 02 18:33:09.828132 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.828094 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" event={"ID":"b612d0da-2500-4091-a03e-89faeb9c435d","Type":"ContainerDied","Data":"bff7ee067bc4eb580fea8e6c4b73c5ec80c840b12ed3974864a3415759d44ff3"} Nov 02 18:33:09.828132 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.828058 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j" Nov 02 18:33:09.838249 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.838226 5549 scope.go:122] "RemoveContainer" containerID="e6acfb570aa43edbf6e4305690fc7665a2fbf91ece0e23730519ef1c007d2293" Nov 02 18:33:09.838585 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:33:09.838557 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6acfb570aa43edbf6e4305690fc7665a2fbf91ece0e23730519ef1c007d2293\": container with ID starting with e6acfb570aa43edbf6e4305690fc7665a2fbf91ece0e23730519ef1c007d2293 not found: ID does not exist" containerID="e6acfb570aa43edbf6e4305690fc7665a2fbf91ece0e23730519ef1c007d2293" Nov 02 18:33:09.838718 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.838594 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6acfb570aa43edbf6e4305690fc7665a2fbf91ece0e23730519ef1c007d2293"} err="failed to get container status \"e6acfb570aa43edbf6e4305690fc7665a2fbf91ece0e23730519ef1c007d2293\": rpc error: code = NotFound desc = could not find container \"e6acfb570aa43edbf6e4305690fc7665a2fbf91ece0e23730519ef1c007d2293\": container with ID starting with e6acfb570aa43edbf6e4305690fc7665a2fbf91ece0e23730519ef1c007d2293 not found: ID does not exist" Nov 02 18:33:09.838718 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.838617 5549 scope.go:122] "RemoveContainer" containerID="993e4c03ff25143e6426ddfa51cc5c0be68e31dc9b4ef457b9090f473d463847" Nov 02 18:33:09.848710 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.848397 5549 scope.go:122] "RemoveContainer" containerID="993e4c03ff25143e6426ddfa51cc5c0be68e31dc9b4ef457b9090f473d463847" Nov 02 18:33:09.848975 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:33:09.848932 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"993e4c03ff25143e6426ddfa51cc5c0be68e31dc9b4ef457b9090f473d463847\": container with ID starting with 993e4c03ff25143e6426ddfa51cc5c0be68e31dc9b4ef457b9090f473d463847 not found: ID does not exist" containerID="993e4c03ff25143e6426ddfa51cc5c0be68e31dc9b4ef457b9090f473d463847" Nov 02 18:33:09.849067 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.848977 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"993e4c03ff25143e6426ddfa51cc5c0be68e31dc9b4ef457b9090f473d463847"} err="failed to get container status \"993e4c03ff25143e6426ddfa51cc5c0be68e31dc9b4ef457b9090f473d463847\": rpc error: code = NotFound desc = could not find container \"993e4c03ff25143e6426ddfa51cc5c0be68e31dc9b4ef457b9090f473d463847\": container with ID starting with 993e4c03ff25143e6426ddfa51cc5c0be68e31dc9b4ef457b9090f473d463847 not found: ID does not exist" Nov 02 18:33:09.851400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.851378 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r"] Nov 02 18:33:09.854931 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.854903 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-36a991-7d5bf9cbc-kmq5r"] Nov 02 18:33:09.865688 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.865660 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j"] Nov 02 18:33:09.868913 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.868889 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-36a991-9dfd66d49-xtk6j"] Nov 02 18:33:09.878185 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:09.878165 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp5tr\" (UniqueName: \"kubernetes.io/projected/31758614-1d84-4256-ba23-aed7dd3dd328-kube-api-access-qp5tr\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:10.002297 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:10.000428 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:10.319830 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:10.319796 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31758614-1d84-4256-ba23-aed7dd3dd328" path="/var/lib/kubelet/pods/31758614-1d84-4256-ba23-aed7dd3dd328/volumes" Nov 02 18:33:10.320271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:10.320256 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b612d0da-2500-4091-a03e-89faeb9c435d" path="/var/lib/kubelet/pods/b612d0da-2500-4091-a03e-89faeb9c435d/volumes" Nov 02 18:33:12.729669 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:12.729636 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:13.256646 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:13.256608 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:13.808234 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:13.808192 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:14.565416 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:14.565380 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg"] Nov 02 18:33:14.581425 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:14.581381 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc"] Nov 02 18:33:14.595969 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:14.595927 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:14.736182 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:14.736146 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:15.224934 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:15.224892 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" podUID="d132c2c1-5fab-4ab4-82f1-892f5aad2771" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:33:15.225555 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:15.224942 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" podUID="95748775-ad15-4a15-8031-a0486a2cf317" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:33:15.313785 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:15.313743 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:15.762864 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:15.762829 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:16.097940 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:16.097856 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:16.715634 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:16.715537 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:17.072464 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.072376 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:33:17.130260 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.130216 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859"] Nov 02 18:33:17.130548 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.130498 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" podUID="189c356c-d893-4df2-b800-97fd592fbe62" containerName="maas-api" containerID="cri-o://4e7787213d4e4c25e6d6f32aba8ea6f6e2c8e45a77007b8672ad9622c0dc1b82" gracePeriod=30 Nov 02 18:33:17.235331 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.235291 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:17.456736 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.456709 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 18:33:17.553425 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.553377 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:17.585346 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.585310 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/189c356c-d893-4df2-b800-97fd592fbe62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/189c356c-d893-4df2-b800-97fd592fbe62-maas-api-tls\") pod \"189c356c-d893-4df2-b800-97fd592fbe62\" (UID: \"189c356c-d893-4df2-b800-97fd592fbe62\") " Nov 02 18:33:17.585522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.585409 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/189c356c-d893-4df2-b800-97fd592fbe62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/189c356c-d893-4df2-b800-97fd592fbe62-metrics-certs\") pod \"189c356c-d893-4df2-b800-97fd592fbe62\" (UID: \"189c356c-d893-4df2-b800-97fd592fbe62\") " Nov 02 18:33:17.585522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.585482 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189c356c-d893-4df2-b800-97fd592fbe62-kube-api-access-rnq46\" (UniqueName: \"kubernetes.io/projected/189c356c-d893-4df2-b800-97fd592fbe62-kube-api-access-rnq46\") pod \"189c356c-d893-4df2-b800-97fd592fbe62\" (UID: \"189c356c-d893-4df2-b800-97fd592fbe62\") " Nov 02 18:33:17.589109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.585591 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189c356c-d893-4df2-b800-97fd592fbe62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/189c356c-d893-4df2-b800-97fd592fbe62-openshift-service-ca\") pod \"189c356c-d893-4df2-b800-97fd592fbe62\" (UID: \"189c356c-d893-4df2-b800-97fd592fbe62\") " Nov 02 18:33:17.589109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.586127 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189c356c-d893-4df2-b800-97fd592fbe62-openshift-service-ca" pod "189c356c-d893-4df2-b800-97fd592fbe62" (UID: "189c356c-d893-4df2-b800-97fd592fbe62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:33:17.589109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.588099 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/189c356c-d893-4df2-b800-97fd592fbe62-metrics-certs" pod "189c356c-d893-4df2-b800-97fd592fbe62" (UID: "189c356c-d893-4df2-b800-97fd592fbe62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:33:17.589109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.588253 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/189c356c-d893-4df2-b800-97fd592fbe62-maas-api-tls" pod "189c356c-d893-4df2-b800-97fd592fbe62" (UID: "189c356c-d893-4df2-b800-97fd592fbe62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:33:17.597748 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.592797 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189c356c-d893-4df2-b800-97fd592fbe62-kube-api-access-rnq46" pod "189c356c-d893-4df2-b800-97fd592fbe62" (UID: "189c356c-d893-4df2-b800-97fd592fbe62"). InnerVolumeSpecName "kube-api-access-rnq46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:17.597748 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:33:17.593993 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:33:17.687157 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.687069 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/189c356c-d893-4df2-b800-97fd592fbe62-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:17.687157 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.687104 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/189c356c-d893-4df2-b800-97fd592fbe62-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:17.687157 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.687115 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/189c356c-d893-4df2-b800-97fd592fbe62-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:17.687157 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.687126 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnq46\" (UniqueName: \"kubernetes.io/projected/189c356c-d893-4df2-b800-97fd592fbe62-kube-api-access-rnq46\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:17.889478 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.889439 5549 generic.go:356] "Generic (PLEG): container finished" podID="189c356c-d893-4df2-b800-97fd592fbe62" containerID="4e7787213d4e4c25e6d6f32aba8ea6f6e2c8e45a77007b8672ad9622c0dc1b82" exitCode=0 Nov 02 18:33:17.889984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.889525 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" Nov 02 18:33:17.889984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.889545 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" event={"ID":"189c356c-d893-4df2-b800-97fd592fbe62","Type":"ContainerDied","Data":"4e7787213d4e4c25e6d6f32aba8ea6f6e2c8e45a77007b8672ad9622c0dc1b82"} Nov 02 18:33:17.889984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.889647 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859" event={"ID":"189c356c-d893-4df2-b800-97fd592fbe62","Type":"ContainerDied","Data":"4fd5c32ac8d9262f5e73f96d3328d8ae2b3353dead41be038acc356061011501"} Nov 02 18:33:17.889984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.889683 5549 scope.go:122] "RemoveContainer" containerID="4e7787213d4e4c25e6d6f32aba8ea6f6e2c8e45a77007b8672ad9622c0dc1b82" Nov 02 18:33:17.903406 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.903379 5549 scope.go:122] "RemoveContainer" containerID="4e7787213d4e4c25e6d6f32aba8ea6f6e2c8e45a77007b8672ad9622c0dc1b82" Nov 02 18:33:17.904252 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:33:17.903787 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e7787213d4e4c25e6d6f32aba8ea6f6e2c8e45a77007b8672ad9622c0dc1b82\": container with ID starting with 4e7787213d4e4c25e6d6f32aba8ea6f6e2c8e45a77007b8672ad9622c0dc1b82 not found: ID does not exist" containerID="4e7787213d4e4c25e6d6f32aba8ea6f6e2c8e45a77007b8672ad9622c0dc1b82" Nov 02 18:33:17.904252 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.903826 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e7787213d4e4c25e6d6f32aba8ea6f6e2c8e45a77007b8672ad9622c0dc1b82"} err="failed to get container status \"4e7787213d4e4c25e6d6f32aba8ea6f6e2c8e45a77007b8672ad9622c0dc1b82\": rpc error: code = NotFound desc = could not find container \"4e7787213d4e4c25e6d6f32aba8ea6f6e2c8e45a77007b8672ad9622c0dc1b82\": container with ID starting with 4e7787213d4e4c25e6d6f32aba8ea6f6e2c8e45a77007b8672ad9622c0dc1b82 not found: ID does not exist" Nov 02 18:33:17.917921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.917890 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859"] Nov 02 18:33:17.923115 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:17.923086 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1e2281-6c86bd6d47-d4859"] Nov 02 18:33:18.312394 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:18.311832 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:18.331064 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:18.331029 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189c356c-d893-4df2-b800-97fd592fbe62" path="/var/lib/kubelet/pods/189c356c-d893-4df2-b800-97fd592fbe62/volumes" Nov 02 18:33:18.998855 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:18.998690 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:19.566538 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:19.566497 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" podUID="bc67d335-3268-420d-ab7b-e63af8d1165f" containerName="payload-processing" containerID="cri-o://75c4d40d27053a96893d5f4307e89aa8399e470b3303c8ff5df71052c2bce0d0" gracePeriod=25 Nov 02 18:33:19.582118 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:19.582081 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" podUID="3ea5ef1e-e3a6-47cf-a908-843442f86e83" containerName="payload-pre-processing" containerID="cri-o://6587f53e125cdb5a9a5a2b0f01c716e03455e6bbf61d979306a352a616e064ab" gracePeriod=25 Nov 02 18:33:19.598489 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:19.598372 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:20.320945 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:20.320908 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:20.912343 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:20.912304 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:21.598692 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:21.598655 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:21.740412 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:21.740378 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:22.096920 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:22.096879 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:22.393562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:22.393481 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp"] Nov 02 18:33:22.393738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:22.393701 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" podUID="da89e636-cb83-41a2-ae31-cfbbaff79e79" containerName="istio-proxy" containerID="cri-o://bd174499444ccf7def32684f615c5c1df7b7e969875cdd96b9b47a61f215fe28" gracePeriod=30 Nov 02 18:33:22.401013 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:22.400984 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:22.538273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:22.538239 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:23.199084 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:23.199045 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:23.331852 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:23.331814 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:23.918903 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:23.918803 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:23.948646 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:23.947825 5549 generic.go:356] "Generic (PLEG): container finished" podID="95748775-ad15-4a15-8031-a0486a2cf317" containerID="b145bf25487f02697495c258d53fef72f69773b77ab077306d89c9e737318600" exitCode=0 Nov 02 18:33:23.948646 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:23.948074 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" event={"ID":"95748775-ad15-4a15-8031-a0486a2cf317","Type":"ContainerDied","Data":"b145bf25487f02697495c258d53fef72f69773b77ab077306d89c9e737318600"} Nov 02 18:33:23.951103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:23.950778 5549 generic.go:356] "Generic (PLEG): container finished" podID="d132c2c1-5fab-4ab4-82f1-892f5aad2771" containerID="39299b653b1bd4eeea63c84ba65a457d0659af67b23358e96a795126a2ea8084" exitCode=0 Nov 02 18:33:23.951103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:23.950812 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" event={"ID":"d132c2c1-5fab-4ab4-82f1-892f5aad2771","Type":"ContainerDied","Data":"39299b653b1bd4eeea63c84ba65a457d0659af67b23358e96a795126a2ea8084"} Nov 02 18:33:24.030176 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.030145 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" Nov 02 18:33:24.065683 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.065646 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" Nov 02 18:33:24.159845 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.159809 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95748775-ad15-4a15-8031-a0486a2cf317-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95748775-ad15-4a15-8031-a0486a2cf317-plugins-config-volume\") pod \"95748775-ad15-4a15-8031-a0486a2cf317\" (UID: \"95748775-ad15-4a15-8031-a0486a2cf317\") " Nov 02 18:33:24.160089 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.159866 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d132c2c1-5fab-4ab4-82f1-892f5aad2771-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d132c2c1-5fab-4ab4-82f1-892f5aad2771-plugins-config-volume\") pod \"d132c2c1-5fab-4ab4-82f1-892f5aad2771\" (UID: \"d132c2c1-5fab-4ab4-82f1-892f5aad2771\") " Nov 02 18:33:24.160089 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.159896 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95748775-ad15-4a15-8031-a0486a2cf317-kube-api-access-jb9gv\" (UniqueName: \"kubernetes.io/projected/95748775-ad15-4a15-8031-a0486a2cf317-kube-api-access-jb9gv\") pod \"95748775-ad15-4a15-8031-a0486a2cf317\" (UID: \"95748775-ad15-4a15-8031-a0486a2cf317\") " Nov 02 18:33:24.160089 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.159917 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d132c2c1-5fab-4ab4-82f1-892f5aad2771-kube-api-access-nxwfp\" (UniqueName: \"kubernetes.io/projected/d132c2c1-5fab-4ab4-82f1-892f5aad2771-kube-api-access-nxwfp\") pod \"d132c2c1-5fab-4ab4-82f1-892f5aad2771\" (UID: \"d132c2c1-5fab-4ab4-82f1-892f5aad2771\") " Nov 02 18:33:24.160339 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.160316 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95748775-ad15-4a15-8031-a0486a2cf317-plugins-config-volume" pod "95748775-ad15-4a15-8031-a0486a2cf317" (UID: "95748775-ad15-4a15-8031-a0486a2cf317"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:33:24.160399 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.160317 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d132c2c1-5fab-4ab4-82f1-892f5aad2771-plugins-config-volume" pod "d132c2c1-5fab-4ab4-82f1-892f5aad2771" (UID: "d132c2c1-5fab-4ab4-82f1-892f5aad2771"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:33:24.160562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.160542 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95748775-ad15-4a15-8031-a0486a2cf317-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:24.160598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.160565 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d132c2c1-5fab-4ab4-82f1-892f5aad2771-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:24.162174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.162148 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d132c2c1-5fab-4ab4-82f1-892f5aad2771-kube-api-access-nxwfp" pod "d132c2c1-5fab-4ab4-82f1-892f5aad2771" (UID: "d132c2c1-5fab-4ab4-82f1-892f5aad2771"). InnerVolumeSpecName "kube-api-access-nxwfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:24.162294 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.162182 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95748775-ad15-4a15-8031-a0486a2cf317-kube-api-access-jb9gv" pod "95748775-ad15-4a15-8031-a0486a2cf317" (UID: "95748775-ad15-4a15-8031-a0486a2cf317"). InnerVolumeSpecName "kube-api-access-jb9gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:24.261814 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.261728 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb9gv\" (UniqueName: \"kubernetes.io/projected/95748775-ad15-4a15-8031-a0486a2cf317-kube-api-access-jb9gv\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:24.261814 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.261761 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxwfp\" (UniqueName: \"kubernetes.io/projected/d132c2c1-5fab-4ab4-82f1-892f5aad2771-kube-api-access-nxwfp\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:24.695187 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.695159 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:24.957896 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.957812 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" Nov 02 18:33:24.957896 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.957849 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv" event={"ID":"d132c2c1-5fab-4ab4-82f1-892f5aad2771","Type":"ContainerDied","Data":"bf91f898f0590d2819c93eb53ded361f7969e292cf5ecde178b377f9ba780e5d"} Nov 02 18:33:24.958152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.957911 5549 scope.go:122] "RemoveContainer" containerID="39299b653b1bd4eeea63c84ba65a457d0659af67b23358e96a795126a2ea8084" Nov 02 18:33:24.959355 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.959328 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" Nov 02 18:33:24.959355 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.959335 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw" event={"ID":"95748775-ad15-4a15-8031-a0486a2cf317","Type":"ContainerDied","Data":"739f1d6caea4ce1da77e8a051191c6ab54d7f7a8eb113f496fd19df5a39aa378"} Nov 02 18:33:24.968214 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.968177 5549 scope.go:122] "RemoveContainer" containerID="b145bf25487f02697495c258d53fef72f69773b77ab077306d89c9e737318600" Nov 02 18:33:24.981694 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.981661 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv"] Nov 02 18:33:24.988777 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.988751 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2db162-75bf5bd459-hqrzv"] Nov 02 18:33:24.997884 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:24.997860 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw"] Nov 02 18:33:25.017418 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:25.017395 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2db162-5776877f-nrsjw"] Nov 02 18:33:25.388152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:25.388065 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:26.106876 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:26.106841 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:26.144340 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:26.144300 5549 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:33:26.144523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:26.144359 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" podUID="da89e636-cb83-41a2-ae31-cfbbaff79e79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:33:26.319597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:26.319563 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95748775-ad15-4a15-8031-a0486a2cf317" path="/var/lib/kubelet/pods/95748775-ad15-4a15-8031-a0486a2cf317/volumes" Nov 02 18:33:26.320180 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:26.320156 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d132c2c1-5fab-4ab4-82f1-892f5aad2771" path="/var/lib/kubelet/pods/d132c2c1-5fab-4ab4-82f1-892f5aad2771/volumes" Nov 02 18:33:26.726728 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:26.726679 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:27.388148 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.388087 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:27.670180 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.670150 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:33:27.804177 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.804141 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da89e636-cb83-41a2-ae31-cfbbaff79e79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da89e636-cb83-41a2-ae31-cfbbaff79e79-istiod-ca-cert\") pod \"da89e636-cb83-41a2-ae31-cfbbaff79e79\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " Nov 02 18:33:27.804658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.804219 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da89e636-cb83-41a2-ae31-cfbbaff79e79-kube-api-access-6wsgg\" (UniqueName: \"kubernetes.io/projected/da89e636-cb83-41a2-ae31-cfbbaff79e79-kube-api-access-6wsgg\") pod \"da89e636-cb83-41a2-ae31-cfbbaff79e79\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " Nov 02 18:33:27.804658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.804259 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-podinfo\") pod \"da89e636-cb83-41a2-ae31-cfbbaff79e79\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " Nov 02 18:33:27.804658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.804287 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-data\") pod \"da89e636-cb83-41a2-ae31-cfbbaff79e79\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " Nov 02 18:33:27.804658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.804314 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-workload-socket\") pod \"da89e636-cb83-41a2-ae31-cfbbaff79e79\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " Nov 02 18:33:27.804658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.804409 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-credential-socket\") pod \"da89e636-cb83-41a2-ae31-cfbbaff79e79\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " Nov 02 18:33:27.804658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.804486 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-envoy\") pod \"da89e636-cb83-41a2-ae31-cfbbaff79e79\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " Nov 02 18:33:27.804658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.804515 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-workload-certs\") pod \"da89e636-cb83-41a2-ae31-cfbbaff79e79\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " Nov 02 18:33:27.804658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.804557 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-token\" (UniqueName: \"kubernetes.io/projected/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-token\") pod \"da89e636-cb83-41a2-ae31-cfbbaff79e79\" (UID: \"da89e636-cb83-41a2-ae31-cfbbaff79e79\") " Nov 02 18:33:27.805117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.804717 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-workload-socket" pod "da89e636-cb83-41a2-ae31-cfbbaff79e79" (UID: "da89e636-cb83-41a2-ae31-cfbbaff79e79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:27.805117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.804641 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da89e636-cb83-41a2-ae31-cfbbaff79e79-istiod-ca-cert" pod "da89e636-cb83-41a2-ae31-cfbbaff79e79" (UID: "da89e636-cb83-41a2-ae31-cfbbaff79e79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:33:27.805117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.804939 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-workload-certs" pod "da89e636-cb83-41a2-ae31-cfbbaff79e79" (UID: "da89e636-cb83-41a2-ae31-cfbbaff79e79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:27.805117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.804992 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-credential-socket" pod "da89e636-cb83-41a2-ae31-cfbbaff79e79" (UID: "da89e636-cb83-41a2-ae31-cfbbaff79e79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:27.805117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.805057 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-data" pod "da89e636-cb83-41a2-ae31-cfbbaff79e79" (UID: "da89e636-cb83-41a2-ae31-cfbbaff79e79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:27.805334 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.805236 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da89e636-cb83-41a2-ae31-cfbbaff79e79-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.805334 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.805259 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.805334 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.805275 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.805334 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.805290 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.805334 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.805304 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.806698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.806672 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-envoy" pod "da89e636-cb83-41a2-ae31-cfbbaff79e79" (UID: "da89e636-cb83-41a2-ae31-cfbbaff79e79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:27.806832 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.806743 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-token" pod "da89e636-cb83-41a2-ae31-cfbbaff79e79" (UID: "da89e636-cb83-41a2-ae31-cfbbaff79e79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:27.806933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.806917 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-podinfo" pod "da89e636-cb83-41a2-ae31-cfbbaff79e79" (UID: "da89e636-cb83-41a2-ae31-cfbbaff79e79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:33:27.807042 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.807018 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da89e636-cb83-41a2-ae31-cfbbaff79e79-kube-api-access-6wsgg" pod "da89e636-cb83-41a2-ae31-cfbbaff79e79" (UID: "da89e636-cb83-41a2-ae31-cfbbaff79e79"). InnerVolumeSpecName "kube-api-access-6wsgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:27.906241 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.906158 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.906241 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.906190 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wsgg\" (UniqueName: \"kubernetes.io/projected/da89e636-cb83-41a2-ae31-cfbbaff79e79-kube-api-access-6wsgg\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.906241 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.906205 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.906241 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.906216 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da89e636-cb83-41a2-ae31-cfbbaff79e79-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.980110 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.980068 5549 generic.go:356] "Generic (PLEG): container finished" podID="da89e636-cb83-41a2-ae31-cfbbaff79e79" containerID="bd174499444ccf7def32684f615c5c1df7b7e969875cdd96b9b47a61f215fe28" exitCode=0 Nov 02 18:33:27.980286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.980122 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" Nov 02 18:33:27.980286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.980157 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" event={"ID":"da89e636-cb83-41a2-ae31-cfbbaff79e79","Type":"ContainerDied","Data":"bd174499444ccf7def32684f615c5c1df7b7e969875cdd96b9b47a61f215fe28"} Nov 02 18:33:27.980286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.980210 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp" event={"ID":"da89e636-cb83-41a2-ae31-cfbbaff79e79","Type":"ContainerDied","Data":"07d316ec2d39af3b249bf4ec2a505536251e69ad9278357c1edb21a2de92c1e2"} Nov 02 18:33:27.980286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.980236 5549 scope.go:122] "RemoveContainer" containerID="bd174499444ccf7def32684f615c5c1df7b7e969875cdd96b9b47a61f215fe28" Nov 02 18:33:27.991093 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.991072 5549 scope.go:122] "RemoveContainer" containerID="bd174499444ccf7def32684f615c5c1df7b7e969875cdd96b9b47a61f215fe28" Nov 02 18:33:27.991379 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:33:27.991358 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd174499444ccf7def32684f615c5c1df7b7e969875cdd96b9b47a61f215fe28\": container with ID starting with bd174499444ccf7def32684f615c5c1df7b7e969875cdd96b9b47a61f215fe28 not found: ID does not exist" containerID="bd174499444ccf7def32684f615c5c1df7b7e969875cdd96b9b47a61f215fe28" Nov 02 18:33:27.991436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:27.991388 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd174499444ccf7def32684f615c5c1df7b7e969875cdd96b9b47a61f215fe28"} err="failed to get container status \"bd174499444ccf7def32684f615c5c1df7b7e969875cdd96b9b47a61f215fe28\": rpc error: code = NotFound desc = could not find container \"bd174499444ccf7def32684f615c5c1df7b7e969875cdd96b9b47a61f215fe28\": container with ID starting with bd174499444ccf7def32684f615c5c1df7b7e969875cdd96b9b47a61f215fe28 not found: ID does not exist" Nov 02 18:33:28.005991 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:28.005965 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp"] Nov 02 18:33:28.030995 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:28.026841 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-qjfgp"] Nov 02 18:33:28.044734 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:28.044693 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:28.319272 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:28.319236 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da89e636-cb83-41a2-ae31-cfbbaff79e79" path="/var/lib/kubelet/pods/da89e636-cb83-41a2-ae31-cfbbaff79e79/volumes" Nov 02 18:33:28.713506 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:28.713435 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:28.856277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:28.856238 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" podUID="bc67d335-3268-420d-ab7b-e63af8d1165f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:33:28.856658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:28.856248 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" podUID="3ea5ef1e-e3a6-47cf-a908-843442f86e83" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:33:29.387761 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:29.387723 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq"] Nov 02 18:33:29.403183 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:29.403144 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55"] Nov 02 18:33:29.413692 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:29.413644 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:29.541005 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:29.540961 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:30.103147 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:30.103105 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:30.440075 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:30.439983 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:30.791429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:30.791394 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:31.493596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:31.493557 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:31.936254 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:31.936168 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk"] Nov 02 18:33:31.936477 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:31.936420 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" podUID="c15281fd-8907-4d15-a06e-85a772e3c837" containerName="maas-api" containerID="cri-o://b3efb0073cc9509bcb74ed53acaff1ad21be1cf4811d6a25651173059fcebbea" gracePeriod=30 Nov 02 18:33:32.031018 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:32.030988 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:32.242735 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:32.242706 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 18:33:32.268349 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:32.268301 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15281fd-8907-4d15-a06e-85a772e3c837-kube-api-access-l4qvd\" (UniqueName: \"kubernetes.io/projected/c15281fd-8907-4d15-a06e-85a772e3c837-kube-api-access-l4qvd\") pod \"c15281fd-8907-4d15-a06e-85a772e3c837\" (UID: \"c15281fd-8907-4d15-a06e-85a772e3c837\") " Nov 02 18:33:32.268507 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:32.268427 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15281fd-8907-4d15-a06e-85a772e3c837-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15281fd-8907-4d15-a06e-85a772e3c837-openshift-service-ca\") pod \"c15281fd-8907-4d15-a06e-85a772e3c837\" (UID: \"c15281fd-8907-4d15-a06e-85a772e3c837\") " Nov 02 18:33:32.268507 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:32.268480 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c15281fd-8907-4d15-a06e-85a772e3c837-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15281fd-8907-4d15-a06e-85a772e3c837-maas-api-tls\") pod \"c15281fd-8907-4d15-a06e-85a772e3c837\" (UID: \"c15281fd-8907-4d15-a06e-85a772e3c837\") " Nov 02 18:33:32.268672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:32.268548 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c15281fd-8907-4d15-a06e-85a772e3c837-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15281fd-8907-4d15-a06e-85a772e3c837-metrics-certs\") pod \"c15281fd-8907-4d15-a06e-85a772e3c837\" (UID: \"c15281fd-8907-4d15-a06e-85a772e3c837\") " Nov 02 18:33:32.268868 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:32.268832 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15281fd-8907-4d15-a06e-85a772e3c837-openshift-service-ca" pod "c15281fd-8907-4d15-a06e-85a772e3c837" (UID: "c15281fd-8907-4d15-a06e-85a772e3c837"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:33:32.269152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:32.269128 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15281fd-8907-4d15-a06e-85a772e3c837-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:32.270791 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:32.270760 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15281fd-8907-4d15-a06e-85a772e3c837-kube-api-access-l4qvd" pod "c15281fd-8907-4d15-a06e-85a772e3c837" (UID: "c15281fd-8907-4d15-a06e-85a772e3c837"). InnerVolumeSpecName "kube-api-access-l4qvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:32.271027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:32.271005 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c15281fd-8907-4d15-a06e-85a772e3c837-maas-api-tls" pod "c15281fd-8907-4d15-a06e-85a772e3c837" (UID: "c15281fd-8907-4d15-a06e-85a772e3c837"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:33:32.271257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:32.271238 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c15281fd-8907-4d15-a06e-85a772e3c837-metrics-certs" pod "c15281fd-8907-4d15-a06e-85a772e3c837" (UID: "c15281fd-8907-4d15-a06e-85a772e3c837"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:33:32.315698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:32.315667 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:32.381413 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:32.378660 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4qvd\" (UniqueName: \"kubernetes.io/projected/c15281fd-8907-4d15-a06e-85a772e3c837-kube-api-access-l4qvd\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:32.381413 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:32.378705 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15281fd-8907-4d15-a06e-85a772e3c837-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:32.381413 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:32.378720 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15281fd-8907-4d15-a06e-85a772e3c837-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:33.015850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:33.015817 5549 generic.go:356] "Generic (PLEG): container finished" podID="c15281fd-8907-4d15-a06e-85a772e3c837" containerID="b3efb0073cc9509bcb74ed53acaff1ad21be1cf4811d6a25651173059fcebbea" exitCode=0 Nov 02 18:33:33.016514 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:33.015895 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" event={"ID":"c15281fd-8907-4d15-a06e-85a772e3c837","Type":"ContainerDied","Data":"b3efb0073cc9509bcb74ed53acaff1ad21be1cf4811d6a25651173059fcebbea"} Nov 02 18:33:33.016514 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:33.015941 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" Nov 02 18:33:33.016514 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:33.015988 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk" event={"ID":"c15281fd-8907-4d15-a06e-85a772e3c837","Type":"ContainerDied","Data":"d76e8349513f270716369bd3fd62b4b3f1b52a018321cb63bc4d1e0f8096ec95"} Nov 02 18:33:33.016514 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:33.016020 5549 scope.go:122] "RemoveContainer" containerID="b3efb0073cc9509bcb74ed53acaff1ad21be1cf4811d6a25651173059fcebbea" Nov 02 18:33:33.032931 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:33.032901 5549 scope.go:122] "RemoveContainer" containerID="b3efb0073cc9509bcb74ed53acaff1ad21be1cf4811d6a25651173059fcebbea" Nov 02 18:33:33.033325 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:33:33.033298 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3efb0073cc9509bcb74ed53acaff1ad21be1cf4811d6a25651173059fcebbea\": container with ID starting with b3efb0073cc9509bcb74ed53acaff1ad21be1cf4811d6a25651173059fcebbea not found: ID does not exist" containerID="b3efb0073cc9509bcb74ed53acaff1ad21be1cf4811d6a25651173059fcebbea" Nov 02 18:33:33.033436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:33.033335 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3efb0073cc9509bcb74ed53acaff1ad21be1cf4811d6a25651173059fcebbea"} err="failed to get container status \"b3efb0073cc9509bcb74ed53acaff1ad21be1cf4811d6a25651173059fcebbea\": rpc error: code = NotFound desc = could not find container \"b3efb0073cc9509bcb74ed53acaff1ad21be1cf4811d6a25651173059fcebbea\": container with ID starting with b3efb0073cc9509bcb74ed53acaff1ad21be1cf4811d6a25651173059fcebbea not found: ID does not exist" Nov 02 18:33:33.035848 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:33.035827 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk"] Nov 02 18:33:33.037427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:33.037401 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:33.038421 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:33.038403 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-7771bf-6b5b4b55d5-qq2sk"] Nov 02 18:33:33.550430 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:33.550393 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:33:33.694330 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:33.694200 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:34.320125 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:34.320093 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15281fd-8907-4d15-a06e-85a772e3c837" path="/var/lib/kubelet/pods/c15281fd-8907-4d15-a06e-85a772e3c837/volumes" Nov 02 18:33:34.382350 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:34.382316 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:34.389132 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:34.389080 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" podUID="a011b9f5-8629-4514-b5db-52b58316d975" containerName="payload-processing" containerID="cri-o://f6efcbcf63da2784a53bf62ee323c90942de6d75cdf08738cc06b45efef86d97" gracePeriod=25 Nov 02 18:33:34.403530 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:34.403482 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" podUID="f2ab9120-6ff3-489c-8dd2-bfb0f1201172" containerName="payload-pre-processing" containerID="cri-o://b0356e704211f1e024e9c0b35f922726e65dd2668566094309212fafdf7d74b3" gracePeriod=25 Nov 02 18:33:34.753752 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:34.753702 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" podUID="a011b9f5-8629-4514-b5db-52b58316d975" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:33:34.753752 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:34.753737 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" podUID="f2ab9120-6ff3-489c-8dd2-bfb0f1201172" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:33:35.092101 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:35.091884 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:35.717123 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:35.716981 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:36.311622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:36.311577 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:37.014311 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:37.014271 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:37.159980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:37.159927 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:37.293026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:37.292920 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:37.612370 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:37.612283 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:37.806336 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:37.806292 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:37.894431 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:37.894336 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7"] Nov 02 18:33:37.894628 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:37.894550 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" podUID="4603b3a2-b9a8-4a06-97c0-a97d08f96470" containerName="istio-proxy" containerID="cri-o://3bf1c88325b40080cb32b45a91a28d059648effddbc58871b506879cedb7f30d" gracePeriod=30 Nov 02 18:33:37.994070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:37.994035 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:38.404327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:38.404294 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:38.629169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:38.629121 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:38.856636 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:38.856597 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" podUID="bc67d335-3268-420d-ab7b-e63af8d1165f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:33:38.856636 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:38.856623 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" podUID="3ea5ef1e-e3a6-47cf-a908-843442f86e83" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:33:39.395293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:39.393522 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:39.621104 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:39.621066 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf"] Nov 02 18:33:39.699114 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:39.699007 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:39.909000 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:39.908888 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" Nov 02 18:33:39.966450 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:39.966419 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" Nov 02 18:33:39.968969 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:39.968930 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea5ef1e-e3a6-47cf-a908-843442f86e83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea5ef1e-e3a6-47cf-a908-843442f86e83-plugins-config-volume\") pod \"3ea5ef1e-e3a6-47cf-a908-843442f86e83\" (UID: \"3ea5ef1e-e3a6-47cf-a908-843442f86e83\") " Nov 02 18:33:39.969099 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:39.969057 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea5ef1e-e3a6-47cf-a908-843442f86e83-kube-api-access-2dsfl\" (UniqueName: \"kubernetes.io/projected/3ea5ef1e-e3a6-47cf-a908-843442f86e83-kube-api-access-2dsfl\") pod \"3ea5ef1e-e3a6-47cf-a908-843442f86e83\" (UID: \"3ea5ef1e-e3a6-47cf-a908-843442f86e83\") " Nov 02 18:33:39.969160 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:39.969102 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc67d335-3268-420d-ab7b-e63af8d1165f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc67d335-3268-420d-ab7b-e63af8d1165f-plugins-config-volume\") pod \"bc67d335-3268-420d-ab7b-e63af8d1165f\" (UID: \"bc67d335-3268-420d-ab7b-e63af8d1165f\") " Nov 02 18:33:39.969232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:39.969218 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc67d335-3268-420d-ab7b-e63af8d1165f-kube-api-access-29kw7\" (UniqueName: \"kubernetes.io/projected/bc67d335-3268-420d-ab7b-e63af8d1165f-kube-api-access-29kw7\") pod \"bc67d335-3268-420d-ab7b-e63af8d1165f\" (UID: \"bc67d335-3268-420d-ab7b-e63af8d1165f\") " Nov 02 18:33:39.969558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:39.969531 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea5ef1e-e3a6-47cf-a908-843442f86e83-plugins-config-volume" pod "3ea5ef1e-e3a6-47cf-a908-843442f86e83" (UID: "3ea5ef1e-e3a6-47cf-a908-843442f86e83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:33:39.969658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:39.969574 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc67d335-3268-420d-ab7b-e63af8d1165f-plugins-config-volume" pod "bc67d335-3268-420d-ab7b-e63af8d1165f" (UID: "bc67d335-3268-420d-ab7b-e63af8d1165f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:33:39.969809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:39.969791 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea5ef1e-e3a6-47cf-a908-843442f86e83-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:39.969850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:39.969816 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc67d335-3268-420d-ab7b-e63af8d1165f-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:39.971515 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:39.971484 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc67d335-3268-420d-ab7b-e63af8d1165f-kube-api-access-29kw7" pod "bc67d335-3268-420d-ab7b-e63af8d1165f" (UID: "bc67d335-3268-420d-ab7b-e63af8d1165f"). InnerVolumeSpecName "kube-api-access-29kw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:39.971634 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:39.971592 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea5ef1e-e3a6-47cf-a908-843442f86e83-kube-api-access-2dsfl" pod "3ea5ef1e-e3a6-47cf-a908-843442f86e83" (UID: "3ea5ef1e-e3a6-47cf-a908-843442f86e83"). InnerVolumeSpecName "kube-api-access-2dsfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:40.062268 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.062223 5549 generic.go:356] "Generic (PLEG): container finished" podID="bc67d335-3268-420d-ab7b-e63af8d1165f" containerID="75c4d40d27053a96893d5f4307e89aa8399e470b3303c8ff5df71052c2bce0d0" exitCode=0 Nov 02 18:33:40.062268 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.062272 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" Nov 02 18:33:40.062495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.062317 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" event={"ID":"bc67d335-3268-420d-ab7b-e63af8d1165f","Type":"ContainerDied","Data":"75c4d40d27053a96893d5f4307e89aa8399e470b3303c8ff5df71052c2bce0d0"} Nov 02 18:33:40.062495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.062377 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg" event={"ID":"bc67d335-3268-420d-ab7b-e63af8d1165f","Type":"ContainerDied","Data":"8fcc164951fdb38da76729981610418b0bbc81960419616410ff454146ae571e"} Nov 02 18:33:40.062495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.062406 5549 scope.go:122] "RemoveContainer" containerID="75c4d40d27053a96893d5f4307e89aa8399e470b3303c8ff5df71052c2bce0d0" Nov 02 18:33:40.064131 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.064111 5549 generic.go:356] "Generic (PLEG): container finished" podID="3ea5ef1e-e3a6-47cf-a908-843442f86e83" containerID="6587f53e125cdb5a9a5a2b0f01c716e03455e6bbf61d979306a352a616e064ab" exitCode=0 Nov 02 18:33:40.064247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.064149 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" event={"ID":"3ea5ef1e-e3a6-47cf-a908-843442f86e83","Type":"ContainerDied","Data":"6587f53e125cdb5a9a5a2b0f01c716e03455e6bbf61d979306a352a616e064ab"} Nov 02 18:33:40.064247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.064162 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" Nov 02 18:33:40.064247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.064186 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc" event={"ID":"3ea5ef1e-e3a6-47cf-a908-843442f86e83","Type":"ContainerDied","Data":"639682cfb220ff06518e4db3c4f6813d81d9351d267f6ea7d9acc569c62fdf18"} Nov 02 18:33:40.075342 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.075317 5549 scope.go:122] "RemoveContainer" containerID="75c4d40d27053a96893d5f4307e89aa8399e470b3303c8ff5df71052c2bce0d0" Nov 02 18:33:40.075851 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:33:40.075822 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c4d40d27053a96893d5f4307e89aa8399e470b3303c8ff5df71052c2bce0d0\": container with ID starting with 75c4d40d27053a96893d5f4307e89aa8399e470b3303c8ff5df71052c2bce0d0 not found: ID does not exist" containerID="75c4d40d27053a96893d5f4307e89aa8399e470b3303c8ff5df71052c2bce0d0" Nov 02 18:33:40.076037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.075871 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c4d40d27053a96893d5f4307e89aa8399e470b3303c8ff5df71052c2bce0d0"} err="failed to get container status \"75c4d40d27053a96893d5f4307e89aa8399e470b3303c8ff5df71052c2bce0d0\": rpc error: code = NotFound desc = could not find container \"75c4d40d27053a96893d5f4307e89aa8399e470b3303c8ff5df71052c2bce0d0\": container with ID starting with 75c4d40d27053a96893d5f4307e89aa8399e470b3303c8ff5df71052c2bce0d0 not found: ID does not exist" Nov 02 18:33:40.076037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.075898 5549 scope.go:122] "RemoveContainer" containerID="6587f53e125cdb5a9a5a2b0f01c716e03455e6bbf61d979306a352a616e064ab" Nov 02 18:33:40.079134 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.077530 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dsfl\" (UniqueName: \"kubernetes.io/projected/3ea5ef1e-e3a6-47cf-a908-843442f86e83-kube-api-access-2dsfl\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:40.079134 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.077556 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29kw7\" (UniqueName: \"kubernetes.io/projected/bc67d335-3268-420d-ab7b-e63af8d1165f-kube-api-access-29kw7\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:40.089841 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.089818 5549 scope.go:122] "RemoveContainer" containerID="6587f53e125cdb5a9a5a2b0f01c716e03455e6bbf61d979306a352a616e064ab" Nov 02 18:33:40.090136 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:33:40.090110 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6587f53e125cdb5a9a5a2b0f01c716e03455e6bbf61d979306a352a616e064ab\": container with ID starting with 6587f53e125cdb5a9a5a2b0f01c716e03455e6bbf61d979306a352a616e064ab not found: ID does not exist" containerID="6587f53e125cdb5a9a5a2b0f01c716e03455e6bbf61d979306a352a616e064ab" Nov 02 18:33:40.090200 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.090144 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6587f53e125cdb5a9a5a2b0f01c716e03455e6bbf61d979306a352a616e064ab"} err="failed to get container status \"6587f53e125cdb5a9a5a2b0f01c716e03455e6bbf61d979306a352a616e064ab\": rpc error: code = NotFound desc = could not find container \"6587f53e125cdb5a9a5a2b0f01c716e03455e6bbf61d979306a352a616e064ab\": container with ID starting with 6587f53e125cdb5a9a5a2b0f01c716e03455e6bbf61d979306a352a616e064ab not found: ID does not exist" Nov 02 18:33:40.096502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.096480 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg"] Nov 02 18:33:40.098661 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.098634 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-1e2281-5cb6cf9947-gb8bg"] Nov 02 18:33:40.109092 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.109060 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc"] Nov 02 18:33:40.111502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.111477 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1e2281-5dd5754475-vxpkc"] Nov 02 18:33:40.120450 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.120306 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:40.318942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.318901 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea5ef1e-e3a6-47cf-a908-843442f86e83" path="/var/lib/kubelet/pods/3ea5ef1e-e3a6-47cf-a908-843442f86e83/volumes" Nov 02 18:33:40.319353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.319338 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc67d335-3268-420d-ab7b-e63af8d1165f" path="/var/lib/kubelet/pods/bc67d335-3268-420d-ab7b-e63af8d1165f/volumes" Nov 02 18:33:40.812790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.812757 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9"] Nov 02 18:33:40.823208 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.823117 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:40.960923 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:40.960888 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:41.322526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:41.322431 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:41.598825 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:41.598730 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:42.413583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:42.413202 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:43.089062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.089027 5549 generic.go:356] "Generic (PLEG): container finished" podID="4603b3a2-b9a8-4a06-97c0-a97d08f96470" containerID="3bf1c88325b40080cb32b45a91a28d059648effddbc58871b506879cedb7f30d" exitCode=0 Nov 02 18:33:43.089239 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.089095 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" event={"ID":"4603b3a2-b9a8-4a06-97c0-a97d08f96470","Type":"ContainerDied","Data":"3bf1c88325b40080cb32b45a91a28d059648effddbc58871b506879cedb7f30d"} Nov 02 18:33:43.149967 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.149930 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:33:43.205890 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.205778 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:43.215478 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.215226 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istiod-ca-cert\") pod \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " Nov 02 18:33:43.215478 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.215286 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-credential-socket\") pod \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " Nov 02 18:33:43.215478 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.215376 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-data\") pod \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " Nov 02 18:33:43.215478 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.215425 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-workload-socket\") pod \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " Nov 02 18:33:43.215478 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.215469 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-envoy\") pod \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " Nov 02 18:33:43.216012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.215556 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-podinfo\") pod \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " Nov 02 18:33:43.216012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.215629 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-workload-certs\") pod \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " Nov 02 18:33:43.216012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.215627 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istiod-ca-cert" pod "4603b3a2-b9a8-4a06-97c0-a97d08f96470" (UID: "4603b3a2-b9a8-4a06-97c0-a97d08f96470"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:33:43.216012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.215925 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-workload-socket" pod "4603b3a2-b9a8-4a06-97c0-a97d08f96470" (UID: "4603b3a2-b9a8-4a06-97c0-a97d08f96470"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:43.216265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.216164 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-credential-socket" pod "4603b3a2-b9a8-4a06-97c0-a97d08f96470" (UID: "4603b3a2-b9a8-4a06-97c0-a97d08f96470"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:43.216604 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.216340 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-data" pod "4603b3a2-b9a8-4a06-97c0-a97d08f96470" (UID: "4603b3a2-b9a8-4a06-97c0-a97d08f96470"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:43.216721 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.216665 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-workload-certs" pod "4603b3a2-b9a8-4a06-97c0-a97d08f96470" (UID: "4603b3a2-b9a8-4a06-97c0-a97d08f96470"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:43.218069 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.218044 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-podinfo" pod "4603b3a2-b9a8-4a06-97c0-a97d08f96470" (UID: "4603b3a2-b9a8-4a06-97c0-a97d08f96470"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:33:43.222051 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.222024 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4603b3a2-b9a8-4a06-97c0-a97d08f96470-kube-api-access-z22t2\" (UniqueName: \"kubernetes.io/projected/4603b3a2-b9a8-4a06-97c0-a97d08f96470-kube-api-access-z22t2\") pod \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " Nov 02 18:33:43.222182 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.222135 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-token\" (UniqueName: \"kubernetes.io/projected/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-token\") pod \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\" (UID: \"4603b3a2-b9a8-4a06-97c0-a97d08f96470\") " Nov 02 18:33:43.222318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.222297 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-envoy" pod "4603b3a2-b9a8-4a06-97c0-a97d08f96470" (UID: "4603b3a2-b9a8-4a06-97c0-a97d08f96470"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:43.222679 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.222656 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:43.222779 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.222685 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:43.222779 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.222702 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:43.222779 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.222716 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:43.222779 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.222730 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:43.222779 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.222745 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:43.222779 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.222758 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4603b3a2-b9a8-4a06-97c0-a97d08f96470-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:43.224170 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.224145 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4603b3a2-b9a8-4a06-97c0-a97d08f96470-kube-api-access-z22t2" pod "4603b3a2-b9a8-4a06-97c0-a97d08f96470" (UID: "4603b3a2-b9a8-4a06-97c0-a97d08f96470"). InnerVolumeSpecName "kube-api-access-z22t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:43.224751 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.224731 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-token" pod "4603b3a2-b9a8-4a06-97c0-a97d08f96470" (UID: "4603b3a2-b9a8-4a06-97c0-a97d08f96470"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:43.324400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.324356 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z22t2\" (UniqueName: \"kubernetes.io/projected/4603b3a2-b9a8-4a06-97c0-a97d08f96470-kube-api-access-z22t2\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:43.324400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.324394 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4603b3a2-b9a8-4a06-97c0-a97d08f96470-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:43.439231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.439183 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7"] Nov 02 18:33:43.607084 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.607005 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:44.000779 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:43.998881 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:44.095964 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:44.095929 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" Nov 02 18:33:44.096152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:44.095970 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7" event={"ID":"4603b3a2-b9a8-4a06-97c0-a97d08f96470","Type":"ContainerDied","Data":"7fea6645e315b8cee268a52b49e4c84b6bb686453c0e408b7c53b12469b61576"} Nov 02 18:33:44.096152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:44.096040 5549 scope.go:122] "RemoveContainer" containerID="3bf1c88325b40080cb32b45a91a28d059648effddbc58871b506879cedb7f30d" Nov 02 18:33:44.119873 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:44.119843 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7"] Nov 02 18:33:44.125776 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:44.125747 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-sxhh7"] Nov 02 18:33:44.319393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:44.319318 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4603b3a2-b9a8-4a06-97c0-a97d08f96470" path="/var/lib/kubelet/pods/4603b3a2-b9a8-4a06-97c0-a97d08f96470/volumes" Nov 02 18:33:44.604004 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:44.603877 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:44.753496 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:44.753455 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" podUID="a011b9f5-8629-4514-b5db-52b58316d975" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:33:44.753672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:44.753455 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" podUID="f2ab9120-6ff3-489c-8dd2-bfb0f1201172" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:33:44.965372 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:44.965335 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:33:45.205772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:45.205732 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:46.008984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:46.008347 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:46.619701 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:46.619650 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:47.293616 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:47.293565 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:47.587901 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:33:47.587811 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:33:47.642199 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:47.642164 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc"] Nov 02 18:33:47.655128 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:47.655095 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7"] Nov 02 18:33:47.781334 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:47.781299 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:47.923142 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:47.923018 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:48.096132 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:48.096097 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:48.230134 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:48.230099 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:48.898512 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:48.898456 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:49.545627 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:49.545593 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:50.035670 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:50.035637 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:33:50.109547 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:50.109503 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:50.252837 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:50.252811 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:33:50.819450 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:50.813507 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:50.939937 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:50.939902 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:51.324044 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:51.324000 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:51.689859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:51.689750 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:52.316286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:52.316245 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:52.643093 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:52.643004 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" podUID="c2b8094f-68c0-461b-b5c9-c6a431267bf2" containerName="payload-processing" containerID="cri-o://2b5fb0f3dafc4c75c0bdf1f8b100567d3d389a28e16689f87424ec3413670cc8" gracePeriod=25 Nov 02 18:33:52.655612 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:52.655568 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" podUID="5700c62a-902c-4366-8a4e-8d8ca71b3eef" containerName="payload-pre-processing" containerID="cri-o://ec5a433d80eaf44d6b9867c361bb0a735ea3738653bf41481a2b457bbe3cfbd8" gracePeriod=25 Nov 02 18:33:53.009110 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:53.009054 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:53.137445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:53.137388 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:53.717187 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:53.717088 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:54.386195 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.385781 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:54.523012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.522977 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:54.648863 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.648817 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" podUID="a575e3e4-796c-4183-9755-8d0f73d233ca" containerName="main" containerID="cri-o://1289e96d58f38b895f84497dbc15c42fbebccd7780b292cd368bd783b3c83650" gracePeriod=45 Nov 02 18:33:54.722531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.722499 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" Nov 02 18:33:54.764348 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.764320 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" Nov 02 18:33:54.867898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.867049 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ab9120-6ff3-489c-8dd2-bfb0f1201172-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ab9120-6ff3-489c-8dd2-bfb0f1201172-plugins-config-volume\") pod \"f2ab9120-6ff3-489c-8dd2-bfb0f1201172\" (UID: \"f2ab9120-6ff3-489c-8dd2-bfb0f1201172\") " Nov 02 18:33:54.867898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.867111 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a011b9f5-8629-4514-b5db-52b58316d975-kube-api-access-h72qd\" (UniqueName: \"kubernetes.io/projected/a011b9f5-8629-4514-b5db-52b58316d975-kube-api-access-h72qd\") pod \"a011b9f5-8629-4514-b5db-52b58316d975\" (UID: \"a011b9f5-8629-4514-b5db-52b58316d975\") " Nov 02 18:33:54.867898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.867235 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a011b9f5-8629-4514-b5db-52b58316d975-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a011b9f5-8629-4514-b5db-52b58316d975-plugins-config-volume\") pod \"a011b9f5-8629-4514-b5db-52b58316d975\" (UID: \"a011b9f5-8629-4514-b5db-52b58316d975\") " Nov 02 18:33:54.867898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.867275 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ab9120-6ff3-489c-8dd2-bfb0f1201172-kube-api-access-vb222\" (UniqueName: \"kubernetes.io/projected/f2ab9120-6ff3-489c-8dd2-bfb0f1201172-kube-api-access-vb222\") pod \"f2ab9120-6ff3-489c-8dd2-bfb0f1201172\" (UID: \"f2ab9120-6ff3-489c-8dd2-bfb0f1201172\") " Nov 02 18:33:54.867898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.867580 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ab9120-6ff3-489c-8dd2-bfb0f1201172-plugins-config-volume" pod "f2ab9120-6ff3-489c-8dd2-bfb0f1201172" (UID: "f2ab9120-6ff3-489c-8dd2-bfb0f1201172"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:33:54.867898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.867804 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ab9120-6ff3-489c-8dd2-bfb0f1201172-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:54.878192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.869236 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a011b9f5-8629-4514-b5db-52b58316d975-plugins-config-volume" pod "a011b9f5-8629-4514-b5db-52b58316d975" (UID: "a011b9f5-8629-4514-b5db-52b58316d975"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:33:54.878192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.876582 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:54.880064 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.880024 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ab9120-6ff3-489c-8dd2-bfb0f1201172-kube-api-access-vb222" pod "f2ab9120-6ff3-489c-8dd2-bfb0f1201172" (UID: "f2ab9120-6ff3-489c-8dd2-bfb0f1201172"). InnerVolumeSpecName "kube-api-access-vb222". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:54.883118 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.883083 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a011b9f5-8629-4514-b5db-52b58316d975-kube-api-access-h72qd" pod "a011b9f5-8629-4514-b5db-52b58316d975" (UID: "a011b9f5-8629-4514-b5db-52b58316d975"). InnerVolumeSpecName "kube-api-access-h72qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:54.912230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.912203 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:33:54.968567 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.968534 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h72qd\" (UniqueName: \"kubernetes.io/projected/a011b9f5-8629-4514-b5db-52b58316d975-kube-api-access-h72qd\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:54.968567 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.968564 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a011b9f5-8629-4514-b5db-52b58316d975-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:54.968772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:54.968575 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb222\" (UniqueName: \"kubernetes.io/projected/f2ab9120-6ff3-489c-8dd2-bfb0f1201172-kube-api-access-vb222\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:55.069871 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.069836 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-model-cache\") pod \"a575e3e4-796c-4183-9755-8d0f73d233ca\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " Nov 02 18:33:55.070085 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.069893 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-dshm\") pod \"a575e3e4-796c-4183-9755-8d0f73d233ca\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " Nov 02 18:33:55.070085 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.069939 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-tmp-dir\") pod \"a575e3e4-796c-4183-9755-8d0f73d233ca\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " Nov 02 18:33:55.070085 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.069994 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a575e3e4-796c-4183-9755-8d0f73d233ca-kube-api-access-b5hn5\" (UniqueName: \"kubernetes.io/projected/a575e3e4-796c-4183-9755-8d0f73d233ca-kube-api-access-b5hn5\") pod \"a575e3e4-796c-4183-9755-8d0f73d233ca\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " Nov 02 18:33:55.070303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.070124 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-home\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-home\") pod \"a575e3e4-796c-4183-9755-8d0f73d233ca\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " Nov 02 18:33:55.070303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.070171 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a575e3e4-796c-4183-9755-8d0f73d233ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/a575e3e4-796c-4183-9755-8d0f73d233ca-tls-certs\") pod \"a575e3e4-796c-4183-9755-8d0f73d233ca\" (UID: \"a575e3e4-796c-4183-9755-8d0f73d233ca\") " Nov 02 18:33:55.070303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.070208 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-model-cache" pod "a575e3e4-796c-4183-9755-8d0f73d233ca" (UID: "a575e3e4-796c-4183-9755-8d0f73d233ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:55.070303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.070258 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-tmp-dir" pod "a575e3e4-796c-4183-9755-8d0f73d233ca" (UID: "a575e3e4-796c-4183-9755-8d0f73d233ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:55.070454 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.070364 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-home" pod "a575e3e4-796c-4183-9755-8d0f73d233ca" (UID: "a575e3e4-796c-4183-9755-8d0f73d233ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:55.070553 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.070535 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:55.070613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.070554 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:55.070613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.070565 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:55.072114 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.072092 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-dshm" pod "a575e3e4-796c-4183-9755-8d0f73d233ca" (UID: "a575e3e4-796c-4183-9755-8d0f73d233ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:55.072217 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.072184 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a575e3e4-796c-4183-9755-8d0f73d233ca-tls-certs" pod "a575e3e4-796c-4183-9755-8d0f73d233ca" (UID: "a575e3e4-796c-4183-9755-8d0f73d233ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:33:55.072264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.072236 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a575e3e4-796c-4183-9755-8d0f73d233ca-kube-api-access-b5hn5" pod "a575e3e4-796c-4183-9755-8d0f73d233ca" (UID: "a575e3e4-796c-4183-9755-8d0f73d233ca"). InnerVolumeSpecName "kube-api-access-b5hn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:55.159378 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.159290 5549 generic.go:356] "Generic (PLEG): container finished" podID="a011b9f5-8629-4514-b5db-52b58316d975" containerID="f6efcbcf63da2784a53bf62ee323c90942de6d75cdf08738cc06b45efef86d97" exitCode=0 Nov 02 18:33:55.159378 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.159351 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" Nov 02 18:33:55.159588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.159391 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" event={"ID":"a011b9f5-8629-4514-b5db-52b58316d975","Type":"ContainerDied","Data":"f6efcbcf63da2784a53bf62ee323c90942de6d75cdf08738cc06b45efef86d97"} Nov 02 18:33:55.159588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.159484 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq" event={"ID":"a011b9f5-8629-4514-b5db-52b58316d975","Type":"ContainerDied","Data":"954a720d5fd9b41da32f7444e8212b19b8c6eb3e477da2aa8b53a68b00fbc301"} Nov 02 18:33:55.159588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.159510 5549 scope.go:122] "RemoveContainer" containerID="f6efcbcf63da2784a53bf62ee323c90942de6d75cdf08738cc06b45efef86d97" Nov 02 18:33:55.160934 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.160908 5549 generic.go:356] "Generic (PLEG): container finished" podID="f2ab9120-6ff3-489c-8dd2-bfb0f1201172" containerID="b0356e704211f1e024e9c0b35f922726e65dd2668566094309212fafdf7d74b3" exitCode=0 Nov 02 18:33:55.161084 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.161000 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" Nov 02 18:33:55.161084 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.160988 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" event={"ID":"f2ab9120-6ff3-489c-8dd2-bfb0f1201172","Type":"ContainerDied","Data":"b0356e704211f1e024e9c0b35f922726e65dd2668566094309212fafdf7d74b3"} Nov 02 18:33:55.161203 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.161103 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" event={"ID":"f2ab9120-6ff3-489c-8dd2-bfb0f1201172","Type":"ContainerDied","Data":"8c6ee07738772d489247de8a60a6dc40c2ae669a3148d343612745e2db04bca9"} Nov 02 18:33:55.162642 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.162617 5549 generic.go:356] "Generic (PLEG): container finished" podID="a575e3e4-796c-4183-9755-8d0f73d233ca" containerID="1289e96d58f38b895f84497dbc15c42fbebccd7780b292cd368bd783b3c83650" exitCode=0 Nov 02 18:33:55.162740 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.162659 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" Nov 02 18:33:55.162740 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.162645 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" event={"ID":"a575e3e4-796c-4183-9755-8d0f73d233ca","Type":"ContainerDied","Data":"1289e96d58f38b895f84497dbc15c42fbebccd7780b292cd368bd783b3c83650"} Nov 02 18:33:55.162878 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.162744 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf" event={"ID":"a575e3e4-796c-4183-9755-8d0f73d233ca","Type":"ContainerDied","Data":"de68552a94eabf7a2b957f5bafdbb08578e45ad8293251397127675e3449a282"} Nov 02 18:33:55.171621 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.171601 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5hn5\" (UniqueName: \"kubernetes.io/projected/a575e3e4-796c-4183-9755-8d0f73d233ca-kube-api-access-b5hn5\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:55.171621 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.171621 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a575e3e4-796c-4183-9755-8d0f73d233ca-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:55.171800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.171632 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a575e3e4-796c-4183-9755-8d0f73d233ca-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:55.171800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.171725 5549 scope.go:122] "RemoveContainer" containerID="f6efcbcf63da2784a53bf62ee323c90942de6d75cdf08738cc06b45efef86d97" Nov 02 18:33:55.172108 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:33:55.172082 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6efcbcf63da2784a53bf62ee323c90942de6d75cdf08738cc06b45efef86d97\": container with ID starting with f6efcbcf63da2784a53bf62ee323c90942de6d75cdf08738cc06b45efef86d97 not found: ID does not exist" containerID="f6efcbcf63da2784a53bf62ee323c90942de6d75cdf08738cc06b45efef86d97" Nov 02 18:33:55.172192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.172110 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6efcbcf63da2784a53bf62ee323c90942de6d75cdf08738cc06b45efef86d97"} err="failed to get container status \"f6efcbcf63da2784a53bf62ee323c90942de6d75cdf08738cc06b45efef86d97\": rpc error: code = NotFound desc = could not find container \"f6efcbcf63da2784a53bf62ee323c90942de6d75cdf08738cc06b45efef86d97\": container with ID starting with f6efcbcf63da2784a53bf62ee323c90942de6d75cdf08738cc06b45efef86d97 not found: ID does not exist" Nov 02 18:33:55.172192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.172129 5549 scope.go:122] "RemoveContainer" containerID="b0356e704211f1e024e9c0b35f922726e65dd2668566094309212fafdf7d74b3" Nov 02 18:33:55.181157 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.181132 5549 scope.go:122] "RemoveContainer" containerID="b0356e704211f1e024e9c0b35f922726e65dd2668566094309212fafdf7d74b3" Nov 02 18:33:55.181423 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:33:55.181404 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0356e704211f1e024e9c0b35f922726e65dd2668566094309212fafdf7d74b3\": container with ID starting with b0356e704211f1e024e9c0b35f922726e65dd2668566094309212fafdf7d74b3 not found: ID does not exist" containerID="b0356e704211f1e024e9c0b35f922726e65dd2668566094309212fafdf7d74b3" Nov 02 18:33:55.181471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.181430 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0356e704211f1e024e9c0b35f922726e65dd2668566094309212fafdf7d74b3"} err="failed to get container status \"b0356e704211f1e024e9c0b35f922726e65dd2668566094309212fafdf7d74b3\": rpc error: code = NotFound desc = could not find container \"b0356e704211f1e024e9c0b35f922726e65dd2668566094309212fafdf7d74b3\": container with ID starting with b0356e704211f1e024e9c0b35f922726e65dd2668566094309212fafdf7d74b3 not found: ID does not exist" Nov 02 18:33:55.181471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.181445 5549 scope.go:122] "RemoveContainer" containerID="1289e96d58f38b895f84497dbc15c42fbebccd7780b292cd368bd783b3c83650" Nov 02 18:33:55.186654 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.186630 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq"] Nov 02 18:33:55.190216 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.190184 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:55.190623 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.190603 5549 scope.go:122] "RemoveContainer" containerID="1289e96d58f38b895f84497dbc15c42fbebccd7780b292cd368bd783b3c83650" Nov 02 18:33:55.190898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.190879 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7771bf-76f4975979-4xbjq"] Nov 02 18:33:55.190986 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:33:55.190966 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1289e96d58f38b895f84497dbc15c42fbebccd7780b292cd368bd783b3c83650\": container with ID starting with 1289e96d58f38b895f84497dbc15c42fbebccd7780b292cd368bd783b3c83650 not found: ID does not exist" containerID="1289e96d58f38b895f84497dbc15c42fbebccd7780b292cd368bd783b3c83650" Nov 02 18:33:55.191044 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.190995 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1289e96d58f38b895f84497dbc15c42fbebccd7780b292cd368bd783b3c83650"} err="failed to get container status \"1289e96d58f38b895f84497dbc15c42fbebccd7780b292cd368bd783b3c83650\": rpc error: code = NotFound desc = could not find container \"1289e96d58f38b895f84497dbc15c42fbebccd7780b292cd368bd783b3c83650\": container with ID starting with 1289e96d58f38b895f84497dbc15c42fbebccd7780b292cd368bd783b3c83650 not found: ID does not exist" Nov 02 18:33:55.203668 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.203642 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf"] Nov 02 18:33:55.224778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.224737 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-47e1c2/sub-test-model-47e1c2-kserve-794d8b89bf-n56qf"] Nov 02 18:33:55.249599 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.249567 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55"] Nov 02 18:33:55.249779 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.249615 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55"] Nov 02 18:33:55.753893 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.753852 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7771bf-5948cf8455-qtf55" podUID="f2ab9120-6ff3-489c-8dd2-bfb0f1201172" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.183:50052\" within 1s: context deadline exceeded" Nov 02 18:33:55.754299 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:33:55.753981 5549 logging.go:55] [core] [Channel #28959 SubChannel #28960]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.183:50052", ServerName: "10.133.0.183:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.183:50052: operation was canceled" Nov 02 18:33:55.766825 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.766790 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" podUID="c2b8094f-68c0-461b-b5c9-c6a431267bf2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:33:55.843177 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.843131 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" podUID="521107be-a789-485f-8827-235aae17ce7e" containerName="main" containerID="cri-o://ebb58ccca634d87014cc627b0735b4b6ae70aede91c23bfe9c025cefd8c91761" gracePeriod=45 Nov 02 18:33:55.920315 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.920248 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:55.997809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:55.997770 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:33:56.103620 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.103593 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:33:56.127793 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.127759 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:56.172615 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.172579 5549 generic.go:356] "Generic (PLEG): container finished" podID="521107be-a789-485f-8827-235aae17ce7e" containerID="ebb58ccca634d87014cc627b0735b4b6ae70aede91c23bfe9c025cefd8c91761" exitCode=0 Nov 02 18:33:56.172764 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.172621 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" Nov 02 18:33:56.172764 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.172665 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" event={"ID":"521107be-a789-485f-8827-235aae17ce7e","Type":"ContainerDied","Data":"ebb58ccca634d87014cc627b0735b4b6ae70aede91c23bfe9c025cefd8c91761"} Nov 02 18:33:56.172764 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.172705 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9" event={"ID":"521107be-a789-485f-8827-235aae17ce7e","Type":"ContainerDied","Data":"e5509d282e5b1ddcccdd0cf0fe7f4fa74250c55d9f670a51455af8ecbdb6ba1b"} Nov 02 18:33:56.172764 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.172724 5549 scope.go:122] "RemoveContainer" containerID="ebb58ccca634d87014cc627b0735b4b6ae70aede91c23bfe9c025cefd8c91761" Nov 02 18:33:56.180342 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.180315 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-dshm\") pod \"521107be-a789-485f-8827-235aae17ce7e\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " Nov 02 18:33:56.180473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.180347 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-tmp-dir\") pod \"521107be-a789-485f-8827-235aae17ce7e\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " Nov 02 18:33:56.180473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.180400 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/521107be-a789-485f-8827-235aae17ce7e-tls-certs\" (UniqueName: \"kubernetes.io/secret/521107be-a789-485f-8827-235aae17ce7e-tls-certs\") pod \"521107be-a789-485f-8827-235aae17ce7e\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " Nov 02 18:33:56.180473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.180452 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521107be-a789-485f-8827-235aae17ce7e-kube-api-access-fks4k\" (UniqueName: \"kubernetes.io/projected/521107be-a789-485f-8827-235aae17ce7e-kube-api-access-fks4k\") pod \"521107be-a789-485f-8827-235aae17ce7e\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " Nov 02 18:33:56.180647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.180474 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-home\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-home\") pod \"521107be-a789-485f-8827-235aae17ce7e\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " Nov 02 18:33:56.180647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.180549 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-model-cache\") pod \"521107be-a789-485f-8827-235aae17ce7e\" (UID: \"521107be-a789-485f-8827-235aae17ce7e\") " Nov 02 18:33:56.180877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.180668 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-tmp-dir" pod "521107be-a789-485f-8827-235aae17ce7e" (UID: "521107be-a789-485f-8827-235aae17ce7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:56.180934 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.180833 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-home" pod "521107be-a789-485f-8827-235aae17ce7e" (UID: "521107be-a789-485f-8827-235aae17ce7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:56.180934 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.180904 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-model-cache" pod "521107be-a789-485f-8827-235aae17ce7e" (UID: "521107be-a789-485f-8827-235aae17ce7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:56.181159 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.181136 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:56.181159 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.181157 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:56.181359 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.181166 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:56.182425 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.182407 5549 scope.go:122] "RemoveContainer" containerID="ebb58ccca634d87014cc627b0735b4b6ae70aede91c23bfe9c025cefd8c91761" Nov 02 18:33:56.182513 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.182481 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-dshm" pod "521107be-a789-485f-8827-235aae17ce7e" (UID: "521107be-a789-485f-8827-235aae17ce7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:56.182714 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:33:56.182689 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb58ccca634d87014cc627b0735b4b6ae70aede91c23bfe9c025cefd8c91761\": container with ID starting with ebb58ccca634d87014cc627b0735b4b6ae70aede91c23bfe9c025cefd8c91761 not found: ID does not exist" containerID="ebb58ccca634d87014cc627b0735b4b6ae70aede91c23bfe9c025cefd8c91761" Nov 02 18:33:56.182773 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.182729 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb58ccca634d87014cc627b0735b4b6ae70aede91c23bfe9c025cefd8c91761"} err="failed to get container status \"ebb58ccca634d87014cc627b0735b4b6ae70aede91c23bfe9c025cefd8c91761\": rpc error: code = NotFound desc = could not find container \"ebb58ccca634d87014cc627b0735b4b6ae70aede91c23bfe9c025cefd8c91761\": container with ID starting with ebb58ccca634d87014cc627b0735b4b6ae70aede91c23bfe9c025cefd8c91761 not found: ID does not exist" Nov 02 18:33:56.182773 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.182696 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/521107be-a789-485f-8827-235aae17ce7e-tls-certs" pod "521107be-a789-485f-8827-235aae17ce7e" (UID: "521107be-a789-485f-8827-235aae17ce7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:33:56.183058 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.183038 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521107be-a789-485f-8827-235aae17ce7e-kube-api-access-fks4k" pod "521107be-a789-485f-8827-235aae17ce7e" (UID: "521107be-a789-485f-8827-235aae17ce7e"). InnerVolumeSpecName "kube-api-access-fks4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:56.282404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.282319 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/521107be-a789-485f-8827-235aae17ce7e-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:56.282404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.282353 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fks4k\" (UniqueName: \"kubernetes.io/projected/521107be-a789-485f-8827-235aae17ce7e-kube-api-access-fks4k\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:56.282404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.282363 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/521107be-a789-485f-8827-235aae17ce7e-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:56.318922 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.318880 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a011b9f5-8629-4514-b5db-52b58316d975" path="/var/lib/kubelet/pods/a011b9f5-8629-4514-b5db-52b58316d975/volumes" Nov 02 18:33:56.319391 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.319375 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a575e3e4-796c-4183-9755-8d0f73d233ca" path="/var/lib/kubelet/pods/a575e3e4-796c-4183-9755-8d0f73d233ca/volumes" Nov 02 18:33:56.319731 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.319719 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ab9120-6ff3-489c-8dd2-bfb0f1201172" path="/var/lib/kubelet/pods/f2ab9120-6ff3-489c-8dd2-bfb0f1201172/volumes" Nov 02 18:33:56.498691 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.498614 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9"] Nov 02 18:33:56.502285 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.502244 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9165c3/sub-test-model-9165c3-kserve-87464848c-xf8n9"] Nov 02 18:33:56.708007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.707103 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:56.775746 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:56.775699 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" podUID="5700c62a-902c-4366-8a4e-8d8ca71b3eef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:33:57.497137 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:57.494754 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:58.136132 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.136100 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:58.189467 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.189430 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh"] Nov 02 18:33:58.311043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.310968 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:58.318415 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.318381 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="521107be-a789-485f-8827-235aae17ce7e" path="/var/lib/kubelet/pods/521107be-a789-485f-8827-235aae17ce7e/volumes" Nov 02 18:33:58.458310 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.458276 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:58.460337 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.460181 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" containerID="cri-o://78bc05cdbb5f1bacf6a2cc15e367c72e19d9b1300a59157a2cfc8d60fd699cc4" gracePeriod=45 Nov 02 18:33:58.538862 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.538824 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": dial tcp 10.133.0.58:8000: connect: connection refused" Nov 02 18:33:58.714708 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.714647 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 18:33:58.817842 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.817802 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-dshm\") pod \"3043cd28-bbab-4205-872c-2428bc46bbdc\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " Nov 02 18:33:58.818070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.817944 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-model-cache\") pod \"3043cd28-bbab-4205-872c-2428bc46bbdc\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " Nov 02 18:33:58.818070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.817996 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3043cd28-bbab-4205-872c-2428bc46bbdc-kube-api-access-pdsl7\" (UniqueName: \"kubernetes.io/projected/3043cd28-bbab-4205-872c-2428bc46bbdc-kube-api-access-pdsl7\") pod \"3043cd28-bbab-4205-872c-2428bc46bbdc\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " Nov 02 18:33:58.818070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.818043 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-tmp-dir\") pod \"3043cd28-bbab-4205-872c-2428bc46bbdc\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " Nov 02 18:33:58.818263 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.818111 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3043cd28-bbab-4205-872c-2428bc46bbdc-tls-certs\" (UniqueName: \"kubernetes.io/secret/3043cd28-bbab-4205-872c-2428bc46bbdc-tls-certs\") pod \"3043cd28-bbab-4205-872c-2428bc46bbdc\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " Nov 02 18:33:58.818263 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.818161 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-home\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-home\") pod \"3043cd28-bbab-4205-872c-2428bc46bbdc\" (UID: \"3043cd28-bbab-4205-872c-2428bc46bbdc\") " Nov 02 18:33:58.818437 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.818409 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-tmp-dir" pod "3043cd28-bbab-4205-872c-2428bc46bbdc" (UID: "3043cd28-bbab-4205-872c-2428bc46bbdc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:58.818510 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.818428 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-model-cache" pod "3043cd28-bbab-4205-872c-2428bc46bbdc" (UID: "3043cd28-bbab-4205-872c-2428bc46bbdc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:58.818706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.818649 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-home" pod "3043cd28-bbab-4205-872c-2428bc46bbdc" (UID: "3043cd28-bbab-4205-872c-2428bc46bbdc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:58.818706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.818681 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:58.818706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.818703 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:58.820939 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.820914 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-dshm" pod "3043cd28-bbab-4205-872c-2428bc46bbdc" (UID: "3043cd28-bbab-4205-872c-2428bc46bbdc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:58.821087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.821050 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3043cd28-bbab-4205-872c-2428bc46bbdc-kube-api-access-pdsl7" pod "3043cd28-bbab-4205-872c-2428bc46bbdc" (UID: "3043cd28-bbab-4205-872c-2428bc46bbdc"). InnerVolumeSpecName "kube-api-access-pdsl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:58.821144 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.821083 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3043cd28-bbab-4205-872c-2428bc46bbdc-tls-certs" pod "3043cd28-bbab-4205-872c-2428bc46bbdc" (UID: "3043cd28-bbab-4205-872c-2428bc46bbdc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:33:58.920164 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.920132 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdsl7\" (UniqueName: \"kubernetes.io/projected/3043cd28-bbab-4205-872c-2428bc46bbdc-kube-api-access-pdsl7\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:58.920330 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.920165 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3043cd28-bbab-4205-872c-2428bc46bbdc-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:58.920330 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.920184 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:58.920330 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:58.920192 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3043cd28-bbab-4205-872c-2428bc46bbdc-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:59.020048 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:59.018936 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:33:59.190458 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:59.190424 5549 generic.go:356] "Generic (PLEG): container finished" podID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerID="78bc05cdbb5f1bacf6a2cc15e367c72e19d9b1300a59157a2cfc8d60fd699cc4" exitCode=0 Nov 02 18:33:59.190948 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:59.190477 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" Nov 02 18:33:59.190948 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:59.190506 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" event={"ID":"3043cd28-bbab-4205-872c-2428bc46bbdc","Type":"ContainerDied","Data":"78bc05cdbb5f1bacf6a2cc15e367c72e19d9b1300a59157a2cfc8d60fd699cc4"} Nov 02 18:33:59.190948 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:59.190547 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7" event={"ID":"3043cd28-bbab-4205-872c-2428bc46bbdc","Type":"ContainerDied","Data":"5438695cad60f2b9492252d4d750437a253fe1ce17499e4883db4d23d773f497"} Nov 02 18:33:59.190948 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:59.190558 5549 scope.go:122] "RemoveContainer" containerID="78bc05cdbb5f1bacf6a2cc15e367c72e19d9b1300a59157a2cfc8d60fd699cc4" Nov 02 18:33:59.202789 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:59.202762 5549 scope.go:122] "RemoveContainer" containerID="78bc05cdbb5f1bacf6a2cc15e367c72e19d9b1300a59157a2cfc8d60fd699cc4" Nov 02 18:33:59.205225 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:33:59.203150 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78bc05cdbb5f1bacf6a2cc15e367c72e19d9b1300a59157a2cfc8d60fd699cc4\": container with ID starting with 78bc05cdbb5f1bacf6a2cc15e367c72e19d9b1300a59157a2cfc8d60fd699cc4 not found: ID does not exist" containerID="78bc05cdbb5f1bacf6a2cc15e367c72e19d9b1300a59157a2cfc8d60fd699cc4" Nov 02 18:33:59.205225 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:59.203193 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78bc05cdbb5f1bacf6a2cc15e367c72e19d9b1300a59157a2cfc8d60fd699cc4"} err="failed to get container status \"78bc05cdbb5f1bacf6a2cc15e367c72e19d9b1300a59157a2cfc8d60fd699cc4\": rpc error: code = NotFound desc = could not find container \"78bc05cdbb5f1bacf6a2cc15e367c72e19d9b1300a59157a2cfc8d60fd699cc4\": container with ID starting with 78bc05cdbb5f1bacf6a2cc15e367c72e19d9b1300a59157a2cfc8d60fd699cc4 not found: ID does not exist" Nov 02 18:33:59.219855 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:59.219827 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7"] Nov 02 18:33:59.225676 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:59.225648 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/premium-w0-4b97c6-kserve-bd5875887-c5sm7"] Nov 02 18:33:59.706973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:33:59.704750 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:00.073676 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:00.073593 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:00.324163 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:00.324043 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" path="/var/lib/kubelet/pods/3043cd28-bbab-4205-872c-2428bc46bbdc/volumes" Nov 02 18:34:00.358989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:00.358944 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:00.490916 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:00.490872 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:01.220198 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:01.220159 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:01.897717 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:01.897683 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:02.394595 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:02.394550 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d"] Nov 02 18:34:02.409213 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:02.409178 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t"] Nov 02 18:34:02.510528 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:02.510493 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:02.653355 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:02.653271 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:02.787844 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:02.787815 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:02.917108 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:02.917025 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:03.043372 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:03.043334 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:03.189817 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:03.189719 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" podUID="55ebe7be-c46e-4fb3-b7df-cf2a522b9f70" containerName="payload-processing" containerID="cri-o://28bab6f9e8a30d1cecd87e2d7dc87bd816b039eed24e027d9b148fbe57f5aad7" gracePeriod=25 Nov 02 18:34:03.336821 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:03.336791 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:03.476918 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:03.476884 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:04.121919 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:04.121884 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:04.921329 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:04.921269 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:05.442890 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:05.442853 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv"] Nov 02 18:34:05.443427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:05.443144 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" podUID="dca8565f-3d0f-4154-ba3f-9e99a38af3fe" containerName="maas-api" containerID="cri-o://55377d9e1845e0ddbad7b2b80fa68b60ad756d9d6ce052b3075681f735b3361a" gracePeriod=30 Nov 02 18:34:05.541942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:05.541905 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:05.734186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:05.734154 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" Nov 02 18:34:05.766902 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:05.766863 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" podUID="c2b8094f-68c0-461b-b5c9-c6a431267bf2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:34:05.901626 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:05.901335 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-kube-api-access-d7wm2\" (UniqueName: \"kubernetes.io/projected/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-kube-api-access-d7wm2\") pod \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\" (UID: \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\") " Nov 02 18:34:05.901626 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:05.901458 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-metrics-certs\") pod \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\" (UID: \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\") " Nov 02 18:34:05.901626 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:05.901508 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-maas-api-tls\") pod \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\" (UID: \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\") " Nov 02 18:34:05.902096 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:05.901740 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-openshift-service-ca\") pod \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\" (UID: \"dca8565f-3d0f-4154-ba3f-9e99a38af3fe\") " Nov 02 18:34:05.903910 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:05.903025 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-openshift-service-ca" pod "dca8565f-3d0f-4154-ba3f-9e99a38af3fe" (UID: "dca8565f-3d0f-4154-ba3f-9e99a38af3fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:05.911800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:05.911708 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:05.913301 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:05.913272 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-kube-api-access-d7wm2" pod "dca8565f-3d0f-4154-ba3f-9e99a38af3fe" (UID: "dca8565f-3d0f-4154-ba3f-9e99a38af3fe"). InnerVolumeSpecName "kube-api-access-d7wm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:05.915148 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:05.915088 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-metrics-certs" pod "dca8565f-3d0f-4154-ba3f-9e99a38af3fe" (UID: "dca8565f-3d0f-4154-ba3f-9e99a38af3fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:34:05.916920 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:05.916894 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-maas-api-tls" pod "dca8565f-3d0f-4154-ba3f-9e99a38af3fe" (UID: "dca8565f-3d0f-4154-ba3f-9e99a38af3fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:34:06.003786 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:06.003690 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:06.003786 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:06.003730 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7wm2\" (UniqueName: \"kubernetes.io/projected/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-kube-api-access-d7wm2\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:06.003786 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:06.003743 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:06.003786 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:06.003752 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca8565f-3d0f-4154-ba3f-9e99a38af3fe-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:06.248449 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:06.248408 5549 generic.go:356] "Generic (PLEG): container finished" podID="dca8565f-3d0f-4154-ba3f-9e99a38af3fe" containerID="55377d9e1845e0ddbad7b2b80fa68b60ad756d9d6ce052b3075681f735b3361a" exitCode=0 Nov 02 18:34:06.248644 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:06.248456 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" Nov 02 18:34:06.248644 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:06.248493 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" event={"ID":"dca8565f-3d0f-4154-ba3f-9e99a38af3fe","Type":"ContainerDied","Data":"55377d9e1845e0ddbad7b2b80fa68b60ad756d9d6ce052b3075681f735b3361a"} Nov 02 18:34:06.248644 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:06.248546 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv" event={"ID":"dca8565f-3d0f-4154-ba3f-9e99a38af3fe","Type":"ContainerDied","Data":"2886fddf46de6ebdc86ecd027a1ccbae933ab73b24e24211364be0853dd71b46"} Nov 02 18:34:06.248644 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:06.248574 5549 scope.go:122] "RemoveContainer" containerID="55377d9e1845e0ddbad7b2b80fa68b60ad756d9d6ce052b3075681f735b3361a" Nov 02 18:34:06.259128 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:06.259102 5549 scope.go:122] "RemoveContainer" containerID="55377d9e1845e0ddbad7b2b80fa68b60ad756d9d6ce052b3075681f735b3361a" Nov 02 18:34:06.259395 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:34:06.259373 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55377d9e1845e0ddbad7b2b80fa68b60ad756d9d6ce052b3075681f735b3361a\": container with ID starting with 55377d9e1845e0ddbad7b2b80fa68b60ad756d9d6ce052b3075681f735b3361a not found: ID does not exist" containerID="55377d9e1845e0ddbad7b2b80fa68b60ad756d9d6ce052b3075681f735b3361a" Nov 02 18:34:06.259444 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:06.259403 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55377d9e1845e0ddbad7b2b80fa68b60ad756d9d6ce052b3075681f735b3361a"} err="failed to get container status \"55377d9e1845e0ddbad7b2b80fa68b60ad756d9d6ce052b3075681f735b3361a\": rpc error: code = NotFound desc = could not find container \"55377d9e1845e0ddbad7b2b80fa68b60ad756d9d6ce052b3075681f735b3361a\": container with ID starting with 55377d9e1845e0ddbad7b2b80fa68b60ad756d9d6ce052b3075681f735b3361a not found: ID does not exist" Nov 02 18:34:06.269248 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:06.269217 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv"] Nov 02 18:34:06.273006 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:06.272980 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ef96f-86565ddcc8-cwzqv"] Nov 02 18:34:06.318189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:06.318147 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dca8565f-3d0f-4154-ba3f-9e99a38af3fe" path="/var/lib/kubelet/pods/dca8565f-3d0f-4154-ba3f-9e99a38af3fe/volumes" Nov 02 18:34:06.517752 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:06.517639 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:06.776240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:06.776154 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" podUID="5700c62a-902c-4366-8a4e-8d8ca71b3eef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:34:07.154195 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:07.154091 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:07.395566 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:07.395521 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" podUID="8062774b-5045-48ff-9846-c20c1ea18860" containerName="payload-processing" containerID="cri-o://7f5f6ef98cc6d2f8b8a177411874a92b32208376e68d17626ac9c2fc93bbeffc" gracePeriod=25 Nov 02 18:34:07.410068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:07.409992 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" podUID="821f0afb-87af-4690-a871-6244febd580d" containerName="payload-pre-processing" containerID="cri-o://baf60d7c581ff4c8b17acfd82f114983491f16adcfcf71f72efbf60b347ca8c9" gracePeriod=25 Nov 02 18:34:07.488639 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:07.488606 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:07.925972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:07.924293 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:08.618196 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:08.618161 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:08.882933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:08.882845 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" podUID="821f0afb-87af-4690-a871-6244febd580d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:34:09.045215 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:09.045179 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:09.183799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:09.183701 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:09.273446 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:09.273407 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:34:09.317217 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:09.317187 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:09.504710 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:09.504673 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:09.747019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:09.746982 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89"] Nov 02 18:34:09.747246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:09.747219 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" podUID="3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" containerName="istio-proxy" containerID="cri-o://f0e76d3456b290b8ef8494ed513e6398a2feb3ddc5a8d29f6db44023bd39d761" gracePeriod=30 Nov 02 18:34:09.841235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:09.841132 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:09.942050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:09.942002 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" podUID="8062774b-5045-48ff-9846-c20c1ea18860" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:34:10.324255 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:10.324214 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:10.593087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:10.593050 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:11.005751 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:11.005707 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:11.616535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:11.616487 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:12.223392 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:12.223359 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:12.522790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:12.522690 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:12.768452 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:12.768400 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" podUID="55ebe7be-c46e-4fb3-b7df-cf2a522b9f70" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:34:12.926014 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:12.925986 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" Nov 02 18:34:12.965399 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:12.965367 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" Nov 02 18:34:12.983043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:12.982991 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b8094f-68c0-461b-b5c9-c6a431267bf2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b8094f-68c0-461b-b5c9-c6a431267bf2-plugins-config-volume\") pod \"c2b8094f-68c0-461b-b5c9-c6a431267bf2\" (UID: \"c2b8094f-68c0-461b-b5c9-c6a431267bf2\") " Nov 02 18:34:12.983244 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:12.983080 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5700c62a-902c-4366-8a4e-8d8ca71b3eef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5700c62a-902c-4366-8a4e-8d8ca71b3eef-plugins-config-volume\") pod \"5700c62a-902c-4366-8a4e-8d8ca71b3eef\" (UID: \"5700c62a-902c-4366-8a4e-8d8ca71b3eef\") " Nov 02 18:34:12.983374 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:12.983350 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b8094f-68c0-461b-b5c9-c6a431267bf2-plugins-config-volume" pod "c2b8094f-68c0-461b-b5c9-c6a431267bf2" (UID: "c2b8094f-68c0-461b-b5c9-c6a431267bf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:12.983470 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:12.983415 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5700c62a-902c-4366-8a4e-8d8ca71b3eef-plugins-config-volume" pod "5700c62a-902c-4366-8a4e-8d8ca71b3eef" (UID: "5700c62a-902c-4366-8a4e-8d8ca71b3eef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:12.983545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:12.983487 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b8094f-68c0-461b-b5c9-c6a431267bf2-kube-api-access-nhpvw\" (UniqueName: \"kubernetes.io/projected/c2b8094f-68c0-461b-b5c9-c6a431267bf2-kube-api-access-nhpvw\") pod \"c2b8094f-68c0-461b-b5c9-c6a431267bf2\" (UID: \"c2b8094f-68c0-461b-b5c9-c6a431267bf2\") " Nov 02 18:34:12.983545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:12.983516 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5700c62a-902c-4366-8a4e-8d8ca71b3eef-kube-api-access-h5qwp\" (UniqueName: \"kubernetes.io/projected/5700c62a-902c-4366-8a4e-8d8ca71b3eef-kube-api-access-h5qwp\") pod \"5700c62a-902c-4366-8a4e-8d8ca71b3eef\" (UID: \"5700c62a-902c-4366-8a4e-8d8ca71b3eef\") " Nov 02 18:34:12.984361 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:12.984015 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b8094f-68c0-461b-b5c9-c6a431267bf2-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:12.984361 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:12.984043 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5700c62a-902c-4366-8a4e-8d8ca71b3eef-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:12.986203 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:12.986176 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b8094f-68c0-461b-b5c9-c6a431267bf2-kube-api-access-nhpvw" pod "c2b8094f-68c0-461b-b5c9-c6a431267bf2" (UID: "c2b8094f-68c0-461b-b5c9-c6a431267bf2"). InnerVolumeSpecName "kube-api-access-nhpvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:12.986884 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:12.986847 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5700c62a-902c-4366-8a4e-8d8ca71b3eef-kube-api-access-h5qwp" pod "5700c62a-902c-4366-8a4e-8d8ca71b3eef" (UID: "5700c62a-902c-4366-8a4e-8d8ca71b3eef"). InnerVolumeSpecName "kube-api-access-h5qwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:13.085213 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.085122 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhpvw\" (UniqueName: \"kubernetes.io/projected/c2b8094f-68c0-461b-b5c9-c6a431267bf2-kube-api-access-nhpvw\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:13.085213 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.085167 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5qwp\" (UniqueName: \"kubernetes.io/projected/5700c62a-902c-4366-8a4e-8d8ca71b3eef-kube-api-access-h5qwp\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:13.288153 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.288107 5549 generic.go:356] "Generic (PLEG): container finished" podID="c2b8094f-68c0-461b-b5c9-c6a431267bf2" containerID="2b5fb0f3dafc4c75c0bdf1f8b100567d3d389a28e16689f87424ec3413670cc8" exitCode=0 Nov 02 18:34:13.288153 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.288154 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" Nov 02 18:34:13.288522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.288188 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" event={"ID":"c2b8094f-68c0-461b-b5c9-c6a431267bf2","Type":"ContainerDied","Data":"2b5fb0f3dafc4c75c0bdf1f8b100567d3d389a28e16689f87424ec3413670cc8"} Nov 02 18:34:13.288522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.288245 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc" event={"ID":"c2b8094f-68c0-461b-b5c9-c6a431267bf2","Type":"ContainerDied","Data":"6be341bf8a03f0d607041a1c8e1fdb3ce0751f781b381927a3183a25b7ed853b"} Nov 02 18:34:13.288522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.288275 5549 scope.go:122] "RemoveContainer" containerID="2b5fb0f3dafc4c75c0bdf1f8b100567d3d389a28e16689f87424ec3413670cc8" Nov 02 18:34:13.290414 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.290384 5549 generic.go:356] "Generic (PLEG): container finished" podID="5700c62a-902c-4366-8a4e-8d8ca71b3eef" containerID="ec5a433d80eaf44d6b9867c361bb0a735ea3738653bf41481a2b457bbe3cfbd8" exitCode=0 Nov 02 18:34:13.310919 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.290431 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" event={"ID":"5700c62a-902c-4366-8a4e-8d8ca71b3eef","Type":"ContainerDied","Data":"ec5a433d80eaf44d6b9867c361bb0a735ea3738653bf41481a2b457bbe3cfbd8"} Nov 02 18:34:13.310919 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.290501 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" event={"ID":"5700c62a-902c-4366-8a4e-8d8ca71b3eef","Type":"ContainerDied","Data":"ee0cbbd9386cf14c230fa0caad9a5ab02d7b2975c9a7a73fd7f1c0437b35a43c"} Nov 02 18:34:13.310919 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.290446 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7" Nov 02 18:34:13.319160 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.319106 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:13.322100 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.322075 5549 scope.go:122] "RemoveContainer" containerID="2b5fb0f3dafc4c75c0bdf1f8b100567d3d389a28e16689f87424ec3413670cc8" Nov 02 18:34:13.322459 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:34:13.322431 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5fb0f3dafc4c75c0bdf1f8b100567d3d389a28e16689f87424ec3413670cc8\": container with ID starting with 2b5fb0f3dafc4c75c0bdf1f8b100567d3d389a28e16689f87424ec3413670cc8 not found: ID does not exist" containerID="2b5fb0f3dafc4c75c0bdf1f8b100567d3d389a28e16689f87424ec3413670cc8" Nov 02 18:34:13.322565 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.322468 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5fb0f3dafc4c75c0bdf1f8b100567d3d389a28e16689f87424ec3413670cc8"} err="failed to get container status \"2b5fb0f3dafc4c75c0bdf1f8b100567d3d389a28e16689f87424ec3413670cc8\": rpc error: code = NotFound desc = could not find container \"2b5fb0f3dafc4c75c0bdf1f8b100567d3d389a28e16689f87424ec3413670cc8\": container with ID starting with 2b5fb0f3dafc4c75c0bdf1f8b100567d3d389a28e16689f87424ec3413670cc8 not found: ID does not exist" Nov 02 18:34:13.322565 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.322491 5549 scope.go:122] "RemoveContainer" containerID="ec5a433d80eaf44d6b9867c361bb0a735ea3738653bf41481a2b457bbe3cfbd8" Nov 02 18:34:13.333939 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.333837 5549 scope.go:122] "RemoveContainer" containerID="ec5a433d80eaf44d6b9867c361bb0a735ea3738653bf41481a2b457bbe3cfbd8" Nov 02 18:34:13.334210 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:34:13.334183 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec5a433d80eaf44d6b9867c361bb0a735ea3738653bf41481a2b457bbe3cfbd8\": container with ID starting with ec5a433d80eaf44d6b9867c361bb0a735ea3738653bf41481a2b457bbe3cfbd8 not found: ID does not exist" containerID="ec5a433d80eaf44d6b9867c361bb0a735ea3738653bf41481a2b457bbe3cfbd8" Nov 02 18:34:13.334300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.334222 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5a433d80eaf44d6b9867c361bb0a735ea3738653bf41481a2b457bbe3cfbd8"} err="failed to get container status \"ec5a433d80eaf44d6b9867c361bb0a735ea3738653bf41481a2b457bbe3cfbd8\": rpc error: code = NotFound desc = could not find container \"ec5a433d80eaf44d6b9867c361bb0a735ea3738653bf41481a2b457bbe3cfbd8\": container with ID starting with ec5a433d80eaf44d6b9867c361bb0a735ea3738653bf41481a2b457bbe3cfbd8 not found: ID does not exist" Nov 02 18:34:13.340410 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.340351 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc"] Nov 02 18:34:13.343335 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.343308 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5ec436-cd8fd9675-splvc"] Nov 02 18:34:13.349535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.349513 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7"] Nov 02 18:34:13.353793 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.353771 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5ec436-9fc7c8ccf-xtql7"] Nov 02 18:34:13.624337 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.624256 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:13.769892 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:13.769858 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:14.028649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:14.028614 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:14.319038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:14.318944 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5700c62a-902c-4366-8a4e-8d8ca71b3eef" path="/var/lib/kubelet/pods/5700c62a-902c-4366-8a4e-8d8ca71b3eef/volumes" Nov 02 18:34:14.319391 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:14.319377 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b8094f-68c0-461b-b5c9-c6a431267bf2" path="/var/lib/kubelet/pods/c2b8094f-68c0-461b-b5c9-c6a431267bf2/volumes" Nov 02 18:34:14.706781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:14.706740 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:15.014288 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.014261 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:34:15.104332 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.104297 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-token\" (UniqueName: \"kubernetes.io/projected/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-token\") pod \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " Nov 02 18:34:15.104523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.104357 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-envoy\") pod \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " Nov 02 18:34:15.104523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.104404 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-podinfo\") pod \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " Nov 02 18:34:15.104523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.104422 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-workload-certs\") pod \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " Nov 02 18:34:15.104523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.104460 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-data\") pod \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " Nov 02 18:34:15.104800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.104596 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-kube-api-access-tb2vn\" (UniqueName: \"kubernetes.io/projected/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-kube-api-access-tb2vn\") pod \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " Nov 02 18:34:15.104800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.104655 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-workload-socket\") pod \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " Nov 02 18:34:15.104800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.104701 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-credential-socket\") pod \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " Nov 02 18:34:15.104800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.104743 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-workload-certs" pod "3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" (UID: "3c9fbbb3-23a8-4f3f-a954-085d739a4ca2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:15.104800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.104761 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-data" pod "3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" (UID: "3c9fbbb3-23a8-4f3f-a954-085d739a4ca2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:15.104800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.104769 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istiod-ca-cert\") pod \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\" (UID: \"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2\") " Nov 02 18:34:15.105190 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.104989 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-credential-socket" pod "3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" (UID: "3c9fbbb3-23a8-4f3f-a954-085d739a4ca2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:15.105190 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.105010 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-workload-socket" pod "3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" (UID: "3c9fbbb3-23a8-4f3f-a954-085d739a4ca2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:15.105310 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.105257 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:15.105310 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.105282 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:15.105310 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.105282 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istiod-ca-cert" pod "3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" (UID: "3c9fbbb3-23a8-4f3f-a954-085d739a4ca2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:15.105310 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.105300 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:15.105547 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.105315 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:15.106678 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.106651 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-podinfo" pod "3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" (UID: "3c9fbbb3-23a8-4f3f-a954-085d739a4ca2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:34:15.106791 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.106682 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-token" pod "3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" (UID: "3c9fbbb3-23a8-4f3f-a954-085d739a4ca2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:15.106791 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.106687 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-envoy" pod "3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" (UID: "3c9fbbb3-23a8-4f3f-a954-085d739a4ca2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:15.107072 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.107057 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-kube-api-access-tb2vn" pod "3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" (UID: "3c9fbbb3-23a8-4f3f-a954-085d739a4ca2"). InnerVolumeSpecName "kube-api-access-tb2vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:15.206346 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.206311 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:15.206346 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.206342 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb2vn\" (UniqueName: \"kubernetes.io/projected/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-kube-api-access-tb2vn\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:15.206346 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.206352 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:15.206587 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.206362 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:15.206587 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.206373 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:15.320942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.320853 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:15.323404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.323372 5549 generic.go:356] "Generic (PLEG): container finished" podID="3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" containerID="f0e76d3456b290b8ef8494ed513e6398a2feb3ddc5a8d29f6db44023bd39d761" exitCode=0 Nov 02 18:34:15.323551 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.323420 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" Nov 02 18:34:15.323551 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.323443 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" event={"ID":"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2","Type":"ContainerDied","Data":"f0e76d3456b290b8ef8494ed513e6398a2feb3ddc5a8d29f6db44023bd39d761"} Nov 02 18:34:15.323551 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.323515 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89" event={"ID":"3c9fbbb3-23a8-4f3f-a954-085d739a4ca2","Type":"ContainerDied","Data":"4e5e11e5d0c165507c57fcfcb9808d73de42a43fb23b816c08455b082a16937a"} Nov 02 18:34:15.323691 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.323553 5549 scope.go:122] "RemoveContainer" containerID="f0e76d3456b290b8ef8494ed513e6398a2feb3ddc5a8d29f6db44023bd39d761" Nov 02 18:34:15.343457 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.335363 5549 scope.go:122] "RemoveContainer" containerID="4e035c6af637a2468e6356bfbfe9ed42c99f034770c74211741644eed8f4f062" Nov 02 18:34:15.348055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.348023 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89"] Nov 02 18:34:15.352135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.350201 5549 scope.go:122] "RemoveContainer" containerID="f0e76d3456b290b8ef8494ed513e6398a2feb3ddc5a8d29f6db44023bd39d761" Nov 02 18:34:15.352135 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:34:15.351069 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e76d3456b290b8ef8494ed513e6398a2feb3ddc5a8d29f6db44023bd39d761\": container with ID starting with f0e76d3456b290b8ef8494ed513e6398a2feb3ddc5a8d29f6db44023bd39d761 not found: ID does not exist" containerID="f0e76d3456b290b8ef8494ed513e6398a2feb3ddc5a8d29f6db44023bd39d761" Nov 02 18:34:15.352135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.351113 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e76d3456b290b8ef8494ed513e6398a2feb3ddc5a8d29f6db44023bd39d761"} err="failed to get container status \"f0e76d3456b290b8ef8494ed513e6398a2feb3ddc5a8d29f6db44023bd39d761\": rpc error: code = NotFound desc = could not find container \"f0e76d3456b290b8ef8494ed513e6398a2feb3ddc5a8d29f6db44023bd39d761\": container with ID starting with f0e76d3456b290b8ef8494ed513e6398a2feb3ddc5a8d29f6db44023bd39d761 not found: ID does not exist" Nov 02 18:34:15.352135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.351140 5549 scope.go:122] "RemoveContainer" containerID="4e035c6af637a2468e6356bfbfe9ed42c99f034770c74211741644eed8f4f062" Nov 02 18:34:15.352135 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:34:15.351469 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e035c6af637a2468e6356bfbfe9ed42c99f034770c74211741644eed8f4f062\": container with ID starting with 4e035c6af637a2468e6356bfbfe9ed42c99f034770c74211741644eed8f4f062 not found: ID does not exist" containerID="4e035c6af637a2468e6356bfbfe9ed42c99f034770c74211741644eed8f4f062" Nov 02 18:34:15.352135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.351497 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e035c6af637a2468e6356bfbfe9ed42c99f034770c74211741644eed8f4f062"} err="failed to get container status \"4e035c6af637a2468e6356bfbfe9ed42c99f034770c74211741644eed8f4f062\": rpc error: code = NotFound desc = could not find container \"4e035c6af637a2468e6356bfbfe9ed42c99f034770c74211741644eed8f4f062\": container with ID starting with 4e035c6af637a2468e6356bfbfe9ed42c99f034770c74211741644eed8f4f062 not found: ID does not exist" Nov 02 18:34:15.352135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:15.351547 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-1ef96f-openshift-default-5995666b49-ccw89"] Nov 02 18:34:15.375286 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:34:15.375261 5549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9fbbb3_23a8_4f3f_a954_085d739a4ca2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9fbbb3_23a8_4f3f_a954_085d739a4ca2.slice/crio-4e5e11e5d0c165507c57fcfcb9808d73de42a43fb23b816c08455b082a16937a\": RecentStats: unable to find data in memory cache]" Nov 02 18:34:16.031563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:16.031525 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:16.318276 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:16.318200 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" path="/var/lib/kubelet/pods/3c9fbbb3-23a8-4f3f-a954-085d739a4ca2/volumes" Nov 02 18:34:16.578807 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:16.578718 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:34:16.698657 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:16.698620 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:16.934410 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:16.934327 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv"] Nov 02 18:34:16.955205 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:16.955165 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt"] Nov 02 18:34:17.182497 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:17.182457 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:17.336379 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:17.336341 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:17.565104 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:17.565063 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:17.598580 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:34:17.598463 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:34:17.699139 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:17.699108 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:18.402290 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:18.402239 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:18.883166 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:18.883128 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" podUID="821f0afb-87af-4690-a871-6244febd580d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:34:19.184622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:19.184523 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:19.707350 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:19.707316 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8"] Nov 02 18:34:19.707838 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:19.707578 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" podUID="7fe6f607-ad08-4d0c-b92f-1618ce0aa86b" containerName="maas-api" containerID="cri-o://379cb1f28659ec324cbb076644db1bf413b472d72cc6fe075e8acb5508353286" gracePeriod=30 Nov 02 18:34:19.797313 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:19.797281 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:19.941330 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:19.941289 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" podUID="8062774b-5045-48ff-9846-c20c1ea18860" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:34:19.976853 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:19.976824 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 18:34:20.057112 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.057071 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-maas-api-tls\") pod \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\" (UID: \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\") " Nov 02 18:34:20.057344 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.057162 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-openshift-service-ca\") pod \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\" (UID: \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\") " Nov 02 18:34:20.057344 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.057209 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-metrics-certs\") pod \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\" (UID: \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\") " Nov 02 18:34:20.057477 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.057375 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-kube-api-access-9qdw7\" (UniqueName: \"kubernetes.io/projected/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-kube-api-access-9qdw7\") pod \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\" (UID: \"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b\") " Nov 02 18:34:20.057559 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.057525 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-openshift-service-ca" pod "7fe6f607-ad08-4d0c-b92f-1618ce0aa86b" (UID: "7fe6f607-ad08-4d0c-b92f-1618ce0aa86b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:20.057789 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.057762 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:20.059483 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.059459 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-metrics-certs" pod "7fe6f607-ad08-4d0c-b92f-1618ce0aa86b" (UID: "7fe6f607-ad08-4d0c-b92f-1618ce0aa86b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:34:20.059483 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.059478 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-kube-api-access-9qdw7" pod "7fe6f607-ad08-4d0c-b92f-1618ce0aa86b" (UID: "7fe6f607-ad08-4d0c-b92f-1618ce0aa86b"). InnerVolumeSpecName "kube-api-access-9qdw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:20.059602 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.059542 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-maas-api-tls" pod "7fe6f607-ad08-4d0c-b92f-1618ce0aa86b" (UID: "7fe6f607-ad08-4d0c-b92f-1618ce0aa86b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:34:20.099232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.099194 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:20.158928 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.158879 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qdw7\" (UniqueName: \"kubernetes.io/projected/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-kube-api-access-9qdw7\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:20.158928 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.158927 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:20.159185 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.158941 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:20.349508 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.349461 5549 generic.go:356] "Generic (PLEG): container finished" podID="7fe6f607-ad08-4d0c-b92f-1618ce0aa86b" containerID="379cb1f28659ec324cbb076644db1bf413b472d72cc6fe075e8acb5508353286" exitCode=0 Nov 02 18:34:20.349687 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.349567 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" Nov 02 18:34:20.349687 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.349602 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" event={"ID":"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b","Type":"ContainerDied","Data":"379cb1f28659ec324cbb076644db1bf413b472d72cc6fe075e8acb5508353286"} Nov 02 18:34:20.349687 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.349661 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8" event={"ID":"7fe6f607-ad08-4d0c-b92f-1618ce0aa86b","Type":"ContainerDied","Data":"0f38aaa2542ca023ef5a204c01bfdef0ceb9f63a16bab128fe9428f983f7e281"} Nov 02 18:34:20.349687 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.349685 5549 scope.go:122] "RemoveContainer" containerID="379cb1f28659ec324cbb076644db1bf413b472d72cc6fe075e8acb5508353286" Nov 02 18:34:20.358873 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.358853 5549 scope.go:122] "RemoveContainer" containerID="379cb1f28659ec324cbb076644db1bf413b472d72cc6fe075e8acb5508353286" Nov 02 18:34:20.359198 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:34:20.359175 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379cb1f28659ec324cbb076644db1bf413b472d72cc6fe075e8acb5508353286\": container with ID starting with 379cb1f28659ec324cbb076644db1bf413b472d72cc6fe075e8acb5508353286 not found: ID does not exist" containerID="379cb1f28659ec324cbb076644db1bf413b472d72cc6fe075e8acb5508353286" Nov 02 18:34:20.359283 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.359213 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379cb1f28659ec324cbb076644db1bf413b472d72cc6fe075e8acb5508353286"} err="failed to get container status \"379cb1f28659ec324cbb076644db1bf413b472d72cc6fe075e8acb5508353286\": rpc error: code = NotFound desc = could not find container \"379cb1f28659ec324cbb076644db1bf413b472d72cc6fe075e8acb5508353286\": container with ID starting with 379cb1f28659ec324cbb076644db1bf413b472d72cc6fe075e8acb5508353286 not found: ID does not exist" Nov 02 18:34:20.365649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.365599 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8"] Nov 02 18:34:20.367657 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.367634 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-47e1c2-574867bdcb-pnhk8"] Nov 02 18:34:20.580612 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.580526 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:20.710426 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.710392 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:20.896809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.887481 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h"] Nov 02 18:34:20.896809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.887854 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" podUID="bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" containerName="istio-proxy" containerID="cri-o://128bd85d288bc303244a07e0aff11a1b7106c6fd7a8a78fc42c6c5afdf521ada" gracePeriod=30 Nov 02 18:34:20.949067 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:20.949030 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:21.209995 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:21.209948 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:21.934706 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:21.934648 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" podUID="ed3785d4-1670-4f59-8f60-20e66fc3210b" containerName="payload-processing" containerID="cri-o://435a9a363139b9ee473b5c44650a05c964a40c050f7063185e5f8cdcf884b374" gracePeriod=25 Nov 02 18:34:21.955891 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:21.955842 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" podUID="fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9" containerName="payload-pre-processing" containerID="cri-o://f4d8caeaeb2667a596582fa2681860eeb9e7eee4a01ec78f420454507d432df8" gracePeriod=25 Nov 02 18:34:22.009941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:22.009757 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:22.012146 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:22.012109 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" podUID="ed3785d4-1670-4f59-8f60-20e66fc3210b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:34:22.246532 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:22.246499 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:22.319401 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:22.319367 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe6f607-ad08-4d0c-b92f-1618ce0aa86b" path="/var/lib/kubelet/pods/7fe6f607-ad08-4d0c-b92f-1618ce0aa86b/volumes" Nov 02 18:34:22.768376 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:22.768331 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" podUID="55ebe7be-c46e-4fb3-b7df-cf2a522b9f70" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:34:22.805217 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:22.802595 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:23.373521 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:23.373482 5549 generic.go:356] "Generic (PLEG): container finished" podID="55ebe7be-c46e-4fb3-b7df-cf2a522b9f70" containerID="28bab6f9e8a30d1cecd87e2d7dc87bd816b039eed24e027d9b148fbe57f5aad7" exitCode=0 Nov 02 18:34:23.373918 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:23.373566 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" event={"ID":"55ebe7be-c46e-4fb3-b7df-cf2a522b9f70","Type":"ContainerDied","Data":"28bab6f9e8a30d1cecd87e2d7dc87bd816b039eed24e027d9b148fbe57f5aad7"} Nov 02 18:34:23.444235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:23.444211 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" Nov 02 18:34:23.492337 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:23.492306 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55ebe7be-c46e-4fb3-b7df-cf2a522b9f70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55ebe7be-c46e-4fb3-b7df-cf2a522b9f70-plugins-config-volume\") pod \"55ebe7be-c46e-4fb3-b7df-cf2a522b9f70\" (UID: \"55ebe7be-c46e-4fb3-b7df-cf2a522b9f70\") " Nov 02 18:34:23.492510 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:23.492399 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ebe7be-c46e-4fb3-b7df-cf2a522b9f70-kube-api-access-skd2l\" (UniqueName: \"kubernetes.io/projected/55ebe7be-c46e-4fb3-b7df-cf2a522b9f70-kube-api-access-skd2l\") pod \"55ebe7be-c46e-4fb3-b7df-cf2a522b9f70\" (UID: \"55ebe7be-c46e-4fb3-b7df-cf2a522b9f70\") " Nov 02 18:34:23.492684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:23.492661 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55ebe7be-c46e-4fb3-b7df-cf2a522b9f70-plugins-config-volume" pod "55ebe7be-c46e-4fb3-b7df-cf2a522b9f70" (UID: "55ebe7be-c46e-4fb3-b7df-cf2a522b9f70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:23.494597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:23.494569 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ebe7be-c46e-4fb3-b7df-cf2a522b9f70-kube-api-access-skd2l" pod "55ebe7be-c46e-4fb3-b7df-cf2a522b9f70" (UID: "55ebe7be-c46e-4fb3-b7df-cf2a522b9f70"). InnerVolumeSpecName "kube-api-access-skd2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:23.523734 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:23.523699 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:23.593309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:23.593272 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55ebe7be-c46e-4fb3-b7df-cf2a522b9f70-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:23.593309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:23.593305 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skd2l\" (UniqueName: \"kubernetes.io/projected/55ebe7be-c46e-4fb3-b7df-cf2a522b9f70-kube-api-access-skd2l\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:23.740835 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:23.740785 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:23.880902 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:23.880806 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:24.364550 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:24.364508 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:24.379750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:24.379718 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" Nov 02 18:34:24.380223 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:24.379750 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh" event={"ID":"55ebe7be-c46e-4fb3-b7df-cf2a522b9f70","Type":"ContainerDied","Data":"c3706c17746f00e66fed82c8a973ee2608d3ea8331e4dd108e66df340410a0f2"} Nov 02 18:34:24.380223 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:24.379813 5549 scope.go:122] "RemoveContainer" containerID="28bab6f9e8a30d1cecd87e2d7dc87bd816b039eed24e027d9b148fbe57f5aad7" Nov 02 18:34:24.401286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:24.401232 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh"] Nov 02 18:34:24.404615 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:24.404550 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4b97c6-74f55b5f69-f9kwh"] Nov 02 18:34:24.494703 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:24.494642 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:24.685226 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:24.685147 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:24.874034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:24.873998 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:34:25.272091 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:25.272054 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:25.402936 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:25.402896 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:25.998897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:25.998772 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:26.155445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.155418 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 18:34:26.225260 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.225228 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-token\" (UniqueName: \"kubernetes.io/projected/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-token\") pod \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " Nov 02 18:34:26.225451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.225271 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-envoy\") pod \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " Nov 02 18:34:26.225451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.225294 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-kube-api-access-cqb78\" (UniqueName: \"kubernetes.io/projected/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-kube-api-access-cqb78\") pod \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " Nov 02 18:34:26.225451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.225330 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istiod-ca-cert\") pod \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " Nov 02 18:34:26.225598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.225446 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-workload-certs\") pod \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " Nov 02 18:34:26.225598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.225510 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-credential-socket\") pod \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " Nov 02 18:34:26.225598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.225560 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-workload-socket\") pod \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " Nov 02 18:34:26.225742 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.225726 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-workload-certs" pod "bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" (UID: "bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:26.225917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.225732 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istiod-ca-cert" pod "bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" (UID: "bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:26.226127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.226095 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-workload-socket" pod "bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" (UID: "bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:26.226439 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.226404 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-credential-socket" pod "bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" (UID: "bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:26.228523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.228494 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-podinfo\") pod \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " Nov 02 18:34:26.228642 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.228600 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-data\") pod \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\" (UID: \"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb\") " Nov 02 18:34:26.228858 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.228833 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-token" pod "bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" (UID: "bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:26.229022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.228905 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-kube-api-access-cqb78" pod "bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" (UID: "bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb"). InnerVolumeSpecName "kube-api-access-cqb78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:26.229109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.229024 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-data" pod "bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" (UID: "bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:26.229371 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.229348 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:26.229436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.229376 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:26.229436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.229391 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:26.229436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.229413 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:26.229436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.229428 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:26.229634 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.229441 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqb78\" (UniqueName: \"kubernetes.io/projected/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-kube-api-access-cqb78\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:26.229634 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.229463 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:26.233167 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.233067 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-envoy" pod "bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" (UID: "bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:26.233676 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.233619 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-podinfo" pod "bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" (UID: "bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:34:26.318497 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.318459 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ebe7be-c46e-4fb3-b7df-cf2a522b9f70" path="/var/lib/kubelet/pods/55ebe7be-c46e-4fb3-b7df-cf2a522b9f70/volumes" Nov 02 18:34:26.330725 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.330694 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:26.330725 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.330725 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:26.393842 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.393806 5549 generic.go:356] "Generic (PLEG): container finished" podID="bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" containerID="128bd85d288bc303244a07e0aff11a1b7106c6fd7a8a78fc42c6c5afdf521ada" exitCode=0 Nov 02 18:34:26.394054 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.393887 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" Nov 02 18:34:26.394054 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.393895 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" event={"ID":"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb","Type":"ContainerDied","Data":"128bd85d288bc303244a07e0aff11a1b7106c6fd7a8a78fc42c6c5afdf521ada"} Nov 02 18:34:26.394054 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.393969 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h" event={"ID":"bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb","Type":"ContainerDied","Data":"87e71642c8479174be12305661727a5e36b1c3a239f8a9d528387942c91e46a9"} Nov 02 18:34:26.394054 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.394002 5549 scope.go:122] "RemoveContainer" containerID="128bd85d288bc303244a07e0aff11a1b7106c6fd7a8a78fc42c6c5afdf521ada" Nov 02 18:34:26.403628 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.403577 5549 scope.go:122] "RemoveContainer" containerID="128bd85d288bc303244a07e0aff11a1b7106c6fd7a8a78fc42c6c5afdf521ada" Nov 02 18:34:26.403930 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:34:26.403889 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"128bd85d288bc303244a07e0aff11a1b7106c6fd7a8a78fc42c6c5afdf521ada\": container with ID starting with 128bd85d288bc303244a07e0aff11a1b7106c6fd7a8a78fc42c6c5afdf521ada not found: ID does not exist" containerID="128bd85d288bc303244a07e0aff11a1b7106c6fd7a8a78fc42c6c5afdf521ada" Nov 02 18:34:26.404011 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.403923 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128bd85d288bc303244a07e0aff11a1b7106c6fd7a8a78fc42c6c5afdf521ada"} err="failed to get container status \"128bd85d288bc303244a07e0aff11a1b7106c6fd7a8a78fc42c6c5afdf521ada\": rpc error: code = NotFound desc = could not find container \"128bd85d288bc303244a07e0aff11a1b7106c6fd7a8a78fc42c6c5afdf521ada\": container with ID starting with 128bd85d288bc303244a07e0aff11a1b7106c6fd7a8a78fc42c6c5afdf521ada not found: ID does not exist" Nov 02 18:34:26.411348 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.411327 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h"] Nov 02 18:34:26.415530 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.415504 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-pt87h"] Nov 02 18:34:26.697027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:26.695784 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:27.354722 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:27.354674 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:27.650754 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:27.650728 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" Nov 02 18:34:27.683821 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:27.683796 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" Nov 02 18:34:27.744748 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:27.744714 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821f0afb-87af-4690-a871-6244febd580d-kube-api-access-xdsk6\" (UniqueName: \"kubernetes.io/projected/821f0afb-87af-4690-a871-6244febd580d-kube-api-access-xdsk6\") pod \"821f0afb-87af-4690-a871-6244febd580d\" (UID: \"821f0afb-87af-4690-a871-6244febd580d\") " Nov 02 18:34:27.744948 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:27.744852 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8062774b-5045-48ff-9846-c20c1ea18860-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8062774b-5045-48ff-9846-c20c1ea18860-plugins-config-volume\") pod \"8062774b-5045-48ff-9846-c20c1ea18860\" (UID: \"8062774b-5045-48ff-9846-c20c1ea18860\") " Nov 02 18:34:27.744948 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:27.744878 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8062774b-5045-48ff-9846-c20c1ea18860-kube-api-access-ftnpc\" (UniqueName: \"kubernetes.io/projected/8062774b-5045-48ff-9846-c20c1ea18860-kube-api-access-ftnpc\") pod \"8062774b-5045-48ff-9846-c20c1ea18860\" (UID: \"8062774b-5045-48ff-9846-c20c1ea18860\") " Nov 02 18:34:27.744948 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:27.744897 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/821f0afb-87af-4690-a871-6244febd580d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/821f0afb-87af-4690-a871-6244febd580d-plugins-config-volume\") pod \"821f0afb-87af-4690-a871-6244febd580d\" (UID: \"821f0afb-87af-4690-a871-6244febd580d\") " Nov 02 18:34:27.745367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:27.745338 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8062774b-5045-48ff-9846-c20c1ea18860-plugins-config-volume" pod "8062774b-5045-48ff-9846-c20c1ea18860" (UID: "8062774b-5045-48ff-9846-c20c1ea18860"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:27.745470 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:27.745340 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/821f0afb-87af-4690-a871-6244febd580d-plugins-config-volume" pod "821f0afb-87af-4690-a871-6244febd580d" (UID: "821f0afb-87af-4690-a871-6244febd580d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:27.746866 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:27.746842 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821f0afb-87af-4690-a871-6244febd580d-kube-api-access-xdsk6" pod "821f0afb-87af-4690-a871-6244febd580d" (UID: "821f0afb-87af-4690-a871-6244febd580d"). InnerVolumeSpecName "kube-api-access-xdsk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:27.746975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:27.746924 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8062774b-5045-48ff-9846-c20c1ea18860-kube-api-access-ftnpc" pod "8062774b-5045-48ff-9846-c20c1ea18860" (UID: "8062774b-5045-48ff-9846-c20c1ea18860"). InnerVolumeSpecName "kube-api-access-ftnpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:27.846257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:27.846222 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8062774b-5045-48ff-9846-c20c1ea18860-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:27.846257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:27.846254 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftnpc\" (UniqueName: \"kubernetes.io/projected/8062774b-5045-48ff-9846-c20c1ea18860-kube-api-access-ftnpc\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:27.846483 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:27.846268 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/821f0afb-87af-4690-a871-6244febd580d-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:27.846483 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:27.846282 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdsk6\" (UniqueName: \"kubernetes.io/projected/821f0afb-87af-4690-a871-6244febd580d-kube-api-access-xdsk6\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:27.912802 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:27.912705 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:28.041193 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.041145 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:28.319057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.319024 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" path="/var/lib/kubelet/pods/bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb/volumes" Nov 02 18:34:28.406394 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.406355 5549 generic.go:356] "Generic (PLEG): container finished" podID="8062774b-5045-48ff-9846-c20c1ea18860" containerID="7f5f6ef98cc6d2f8b8a177411874a92b32208376e68d17626ac9c2fc93bbeffc" exitCode=0 Nov 02 18:34:28.406589 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.406397 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" Nov 02 18:34:28.406589 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.406445 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" event={"ID":"8062774b-5045-48ff-9846-c20c1ea18860","Type":"ContainerDied","Data":"7f5f6ef98cc6d2f8b8a177411874a92b32208376e68d17626ac9c2fc93bbeffc"} Nov 02 18:34:28.406589 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.406501 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d" event={"ID":"8062774b-5045-48ff-9846-c20c1ea18860","Type":"ContainerDied","Data":"6fa21fdc5fe6a30d137c2229ccb935b48d8c30b2b79f6f540ec09c251ed67041"} Nov 02 18:34:28.406589 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.406527 5549 scope.go:122] "RemoveContainer" containerID="7f5f6ef98cc6d2f8b8a177411874a92b32208376e68d17626ac9c2fc93bbeffc" Nov 02 18:34:28.407860 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.407837 5549 generic.go:356] "Generic (PLEG): container finished" podID="821f0afb-87af-4690-a871-6244febd580d" containerID="baf60d7c581ff4c8b17acfd82f114983491f16adcfcf71f72efbf60b347ca8c9" exitCode=0 Nov 02 18:34:28.408002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.407894 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" Nov 02 18:34:28.408002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.407923 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" event={"ID":"821f0afb-87af-4690-a871-6244febd580d","Type":"ContainerDied","Data":"baf60d7c581ff4c8b17acfd82f114983491f16adcfcf71f72efbf60b347ca8c9"} Nov 02 18:34:28.408002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.407989 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t" event={"ID":"821f0afb-87af-4690-a871-6244febd580d","Type":"ContainerDied","Data":"9a8fd17d7999daf36358dce62b703a3c89ebeb34b3532d389ae0f228185e2466"} Nov 02 18:34:28.421605 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.421587 5549 scope.go:122] "RemoveContainer" containerID="7f5f6ef98cc6d2f8b8a177411874a92b32208376e68d17626ac9c2fc93bbeffc" Nov 02 18:34:28.421919 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:34:28.421889 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5f6ef98cc6d2f8b8a177411874a92b32208376e68d17626ac9c2fc93bbeffc\": container with ID starting with 7f5f6ef98cc6d2f8b8a177411874a92b32208376e68d17626ac9c2fc93bbeffc not found: ID does not exist" containerID="7f5f6ef98cc6d2f8b8a177411874a92b32208376e68d17626ac9c2fc93bbeffc" Nov 02 18:34:28.422038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.421922 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5f6ef98cc6d2f8b8a177411874a92b32208376e68d17626ac9c2fc93bbeffc"} err="failed to get container status \"7f5f6ef98cc6d2f8b8a177411874a92b32208376e68d17626ac9c2fc93bbeffc\": rpc error: code = NotFound desc = could not find container \"7f5f6ef98cc6d2f8b8a177411874a92b32208376e68d17626ac9c2fc93bbeffc\": container with ID starting with 7f5f6ef98cc6d2f8b8a177411874a92b32208376e68d17626ac9c2fc93bbeffc not found: ID does not exist" Nov 02 18:34:28.422038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.421942 5549 scope.go:122] "RemoveContainer" containerID="baf60d7c581ff4c8b17acfd82f114983491f16adcfcf71f72efbf60b347ca8c9" Nov 02 18:34:28.430549 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.430527 5549 scope.go:122] "RemoveContainer" containerID="baf60d7c581ff4c8b17acfd82f114983491f16adcfcf71f72efbf60b347ca8c9" Nov 02 18:34:28.430832 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:34:28.430803 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf60d7c581ff4c8b17acfd82f114983491f16adcfcf71f72efbf60b347ca8c9\": container with ID starting with baf60d7c581ff4c8b17acfd82f114983491f16adcfcf71f72efbf60b347ca8c9 not found: ID does not exist" containerID="baf60d7c581ff4c8b17acfd82f114983491f16adcfcf71f72efbf60b347ca8c9" Nov 02 18:34:28.430931 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.430839 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf60d7c581ff4c8b17acfd82f114983491f16adcfcf71f72efbf60b347ca8c9"} err="failed to get container status \"baf60d7c581ff4c8b17acfd82f114983491f16adcfcf71f72efbf60b347ca8c9\": rpc error: code = NotFound desc = could not find container \"baf60d7c581ff4c8b17acfd82f114983491f16adcfcf71f72efbf60b347ca8c9\": container with ID starting with baf60d7c581ff4c8b17acfd82f114983491f16adcfcf71f72efbf60b347ca8c9 not found: ID does not exist" Nov 02 18:34:28.432198 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.432178 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d"] Nov 02 18:34:28.435280 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.435259 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1ef96f-776b648c8c-8z56d"] Nov 02 18:34:28.445591 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.445569 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t"] Nov 02 18:34:28.449068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.449044 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1ef96f-6b4f5f9ff9-68w5t"] Nov 02 18:34:28.607017 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.606908 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:28.903457 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:28.903372 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:29.044984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:29.044929 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:29.314838 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:29.314477 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:30.022444 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:30.019610 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:30.318204 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:30.318125 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8062774b-5045-48ff-9846-c20c1ea18860" path="/var/lib/kubelet/pods/8062774b-5045-48ff-9846-c20c1ea18860/volumes" Nov 02 18:34:30.318527 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:30.318512 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="821f0afb-87af-4690-a871-6244febd580d" path="/var/lib/kubelet/pods/821f0afb-87af-4690-a871-6244febd580d/volumes" Nov 02 18:34:30.711657 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:30.711573 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:30.999200 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:30.999114 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" podUID="fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:34:31.180812 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:31.180774 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2"] Nov 02 18:34:31.281538 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:31.281441 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:31.416327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:31.416293 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:31.684392 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:31.684297 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:31.810077 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:31.810046 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:32.012147 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:32.012113 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" podUID="ed3785d4-1670-4f59-8f60-20e66fc3210b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:34:32.231066 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:32.231023 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:32.545651 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:32.545591 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:33.195240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:33.195208 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:33.903750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:33.903704 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:34.556506 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:34.556466 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:34:34.683222 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:34.683183 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:35.295880 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:35.295846 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:35.925884 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:35.925846 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:36.181737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:36.181627 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" podUID="3190ca5d-41b1-474c-aaf6-8859ef3d2dfd" containerName="payload-pre-processing" containerID="cri-o://d932f1c08ebec84f639291c68c29d0c03c6bb76d69884e7cdc7f8de93e794bac" gracePeriod=25 Nov 02 18:34:36.676592 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:36.676550 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:37.220574 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:37.220541 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:37.919702 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:37.919664 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:38.070071 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:38.070037 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:38.208883 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:38.208848 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:38.581725 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:38.581643 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:38.793781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:38.793745 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:38.981872 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:38.981840 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:39.454097 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:39.454064 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:39.622599 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:39.622554 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:40.396792 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:40.396751 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:40.999140 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:40.999097 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" podUID="fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:34:41.014321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:41.014286 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:41.699362 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:41.699328 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:42.011120 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:34:42.011084 5549 logging.go:55] [core] [Channel #29096 SubChannel #29097]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.166:50052", ServerName: "10.133.0.166:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.166:50052: connect: connection refused" Nov 02 18:34:42.211974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.211891 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" Nov 02 18:34:42.247023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.246987 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" Nov 02 18:34:42.290597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.290564 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed3785d4-1670-4f59-8f60-20e66fc3210b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3785d4-1670-4f59-8f60-20e66fc3210b-plugins-config-volume\") pod \"ed3785d4-1670-4f59-8f60-20e66fc3210b\" (UID: \"ed3785d4-1670-4f59-8f60-20e66fc3210b\") " Nov 02 18:34:42.290796 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.290662 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3785d4-1670-4f59-8f60-20e66fc3210b-kube-api-access-b6t5p\" (UniqueName: \"kubernetes.io/projected/ed3785d4-1670-4f59-8f60-20e66fc3210b-kube-api-access-b6t5p\") pod \"ed3785d4-1670-4f59-8f60-20e66fc3210b\" (UID: \"ed3785d4-1670-4f59-8f60-20e66fc3210b\") " Nov 02 18:34:42.291050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.291025 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3785d4-1670-4f59-8f60-20e66fc3210b-plugins-config-volume" pod "ed3785d4-1670-4f59-8f60-20e66fc3210b" (UID: "ed3785d4-1670-4f59-8f60-20e66fc3210b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:42.291181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.291162 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3785d4-1670-4f59-8f60-20e66fc3210b-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:42.292803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.292782 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3785d4-1670-4f59-8f60-20e66fc3210b-kube-api-access-b6t5p" pod "ed3785d4-1670-4f59-8f60-20e66fc3210b" (UID: "ed3785d4-1670-4f59-8f60-20e66fc3210b"). InnerVolumeSpecName "kube-api-access-b6t5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:42.299915 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.299843 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:42.391831 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.391794 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9-kube-api-access-fbs4t\" (UniqueName: \"kubernetes.io/projected/fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9-kube-api-access-fbs4t\") pod \"fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9\" (UID: \"fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9\") " Nov 02 18:34:42.392037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.391883 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9-plugins-config-volume\") pod \"fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9\" (UID: \"fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9\") " Nov 02 18:34:42.392466 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.392283 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6t5p\" (UniqueName: \"kubernetes.io/projected/ed3785d4-1670-4f59-8f60-20e66fc3210b-kube-api-access-b6t5p\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:42.392466 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.392290 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9-plugins-config-volume" pod "fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9" (UID: "fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:42.393945 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.393925 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9-kube-api-access-fbs4t" pod "fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9" (UID: "fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9"). InnerVolumeSpecName "kube-api-access-fbs4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:42.490490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.490401 5549 generic.go:356] "Generic (PLEG): container finished" podID="ed3785d4-1670-4f59-8f60-20e66fc3210b" containerID="435a9a363139b9ee473b5c44650a05c964a40c050f7063185e5f8cdcf884b374" exitCode=0 Nov 02 18:34:42.490490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.490439 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" Nov 02 18:34:42.490490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.490475 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" event={"ID":"ed3785d4-1670-4f59-8f60-20e66fc3210b","Type":"ContainerDied","Data":"435a9a363139b9ee473b5c44650a05c964a40c050f7063185e5f8cdcf884b374"} Nov 02 18:34:42.490736 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.490523 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" event={"ID":"ed3785d4-1670-4f59-8f60-20e66fc3210b","Type":"ContainerDied","Data":"c63c5a2643d7222a16f07f18e9fb24cf14c8ca56a0ca942ac1e078d05bade0fd"} Nov 02 18:34:42.490736 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.490549 5549 scope.go:122] "RemoveContainer" containerID="435a9a363139b9ee473b5c44650a05c964a40c050f7063185e5f8cdcf884b374" Nov 02 18:34:42.493981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.492029 5549 generic.go:356] "Generic (PLEG): container finished" podID="fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9" containerID="f4d8caeaeb2667a596582fa2681860eeb9e7eee4a01ec78f420454507d432df8" exitCode=0 Nov 02 18:34:42.493981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.492074 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" event={"ID":"fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9","Type":"ContainerDied","Data":"f4d8caeaeb2667a596582fa2681860eeb9e7eee4a01ec78f420454507d432df8"} Nov 02 18:34:42.493981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.492116 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" event={"ID":"fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9","Type":"ContainerDied","Data":"f55b41a5fca7939b593718bd26a7c00809a252f43c359538bbfa55b70c5f7784"} Nov 02 18:34:42.493981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.492077 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt" Nov 02 18:34:42.493981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.492926 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbs4t\" (UniqueName: \"kubernetes.io/projected/fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9-kube-api-access-fbs4t\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:42.493981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.492946 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:42.499859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.499839 5549 scope.go:122] "RemoveContainer" containerID="435a9a363139b9ee473b5c44650a05c964a40c050f7063185e5f8cdcf884b374" Nov 02 18:34:42.500144 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:34:42.500125 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"435a9a363139b9ee473b5c44650a05c964a40c050f7063185e5f8cdcf884b374\": container with ID starting with 435a9a363139b9ee473b5c44650a05c964a40c050f7063185e5f8cdcf884b374 not found: ID does not exist" containerID="435a9a363139b9ee473b5c44650a05c964a40c050f7063185e5f8cdcf884b374" Nov 02 18:34:42.500242 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.500155 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435a9a363139b9ee473b5c44650a05c964a40c050f7063185e5f8cdcf884b374"} err="failed to get container status \"435a9a363139b9ee473b5c44650a05c964a40c050f7063185e5f8cdcf884b374\": rpc error: code = NotFound desc = could not find container \"435a9a363139b9ee473b5c44650a05c964a40c050f7063185e5f8cdcf884b374\": container with ID starting with 435a9a363139b9ee473b5c44650a05c964a40c050f7063185e5f8cdcf884b374 not found: ID does not exist" Nov 02 18:34:42.500242 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.500178 5549 scope.go:122] "RemoveContainer" containerID="f4d8caeaeb2667a596582fa2681860eeb9e7eee4a01ec78f420454507d432df8" Nov 02 18:34:42.508411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.508390 5549 scope.go:122] "RemoveContainer" containerID="f4d8caeaeb2667a596582fa2681860eeb9e7eee4a01ec78f420454507d432df8" Nov 02 18:34:42.508778 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:34:42.508707 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d8caeaeb2667a596582fa2681860eeb9e7eee4a01ec78f420454507d432df8\": container with ID starting with f4d8caeaeb2667a596582fa2681860eeb9e7eee4a01ec78f420454507d432df8 not found: ID does not exist" containerID="f4d8caeaeb2667a596582fa2681860eeb9e7eee4a01ec78f420454507d432df8" Nov 02 18:34:42.508778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.508749 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d8caeaeb2667a596582fa2681860eeb9e7eee4a01ec78f420454507d432df8"} err="failed to get container status \"f4d8caeaeb2667a596582fa2681860eeb9e7eee4a01ec78f420454507d432df8\": rpc error: code = NotFound desc = could not find container \"f4d8caeaeb2667a596582fa2681860eeb9e7eee4a01ec78f420454507d432df8\": container with ID starting with f4d8caeaeb2667a596582fa2681860eeb9e7eee4a01ec78f420454507d432df8 not found: ID does not exist" Nov 02 18:34:42.510541 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.510517 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv"] Nov 02 18:34:42.514820 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.514797 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv"] Nov 02 18:34:42.528093 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.528072 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt"] Nov 02 18:34:42.533350 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.533328 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-47e1c2-84f6cb4947-lgbqt"] Nov 02 18:34:42.993299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:42.993196 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:43.011208 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:43.011146 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-47e1c2-84945f99d9-pslvv" podUID="ed3785d4-1670-4f59-8f60-20e66fc3210b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.166:50052\" within 1s: context deadline exceeded" Nov 02 18:34:43.752186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:43.752121 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:44.317810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:44.317778 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed3785d4-1670-4f59-8f60-20e66fc3210b" path="/var/lib/kubelet/pods/ed3785d4-1670-4f59-8f60-20e66fc3210b/volumes" Nov 02 18:34:44.318287 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:44.318257 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9" path="/var/lib/kubelet/pods/fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9/volumes" Nov 02 18:34:44.463004 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:44.462946 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:44.606363 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:44.606266 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:44.970028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:44.969993 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" podUID="3190ca5d-41b1-474c-aaf6-8859ef3d2dfd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:34:45.088010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:45.087970 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:45.230003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:45.229894 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:45.904185 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:45.900823 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:46.249757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:46.249722 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:34:46.611986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:46.611870 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:47.061157 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:47.061107 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:47.198819 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:47.198779 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:47.332482 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:47.332394 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:47.587194 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:34:47.587105 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:34:47.966643 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:47.966540 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:48.718261 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:48.714090 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:49.317640 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:49.317601 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:49.961306 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:49.961274 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:50.610931 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:50.610868 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:50.901344 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:50.901261 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8"] Nov 02 18:34:50.915794 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:50.915759 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl"] Nov 02 18:34:51.022068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:51.022028 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:51.167927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:51.167836 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:52.131267 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:52.131238 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:54.970330 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:54.970286 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" podUID="3190ca5d-41b1-474c-aaf6-8859ef3d2dfd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:34:55.901971 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:55.901906 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" podUID="609d560e-9146-4fbf-9aeb-1ca65e5d3971" containerName="payload-processing" containerID="cri-o://047efcc3b279a6bb8b7ba49bba4bea599c56776e417d1abc5ac90de17358d6cc" gracePeriod=25 Nov 02 18:34:55.916746 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:55.916710 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" podUID="23593f20-8dc6-4ff6-93f4-d9b1fc859341" containerName="payload-pre-processing" containerID="cri-o://5f289c32be11f7a00c9333de065ac38f7e617d07ce8b83ef8a784416e9ba1e09" gracePeriod=25 Nov 02 18:34:56.431546 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:56.431518 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" Nov 02 18:34:56.535689 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:56.535656 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3190ca5d-41b1-474c-aaf6-8859ef3d2dfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3190ca5d-41b1-474c-aaf6-8859ef3d2dfd-plugins-config-volume\") pod \"3190ca5d-41b1-474c-aaf6-8859ef3d2dfd\" (UID: \"3190ca5d-41b1-474c-aaf6-8859ef3d2dfd\") " Nov 02 18:34:56.535880 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:56.535837 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3190ca5d-41b1-474c-aaf6-8859ef3d2dfd-kube-api-access-dgm5g\" (UniqueName: \"kubernetes.io/projected/3190ca5d-41b1-474c-aaf6-8859ef3d2dfd-kube-api-access-dgm5g\") pod \"3190ca5d-41b1-474c-aaf6-8859ef3d2dfd\" (UID: \"3190ca5d-41b1-474c-aaf6-8859ef3d2dfd\") " Nov 02 18:34:56.536068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:56.536044 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3190ca5d-41b1-474c-aaf6-8859ef3d2dfd-plugins-config-volume" pod "3190ca5d-41b1-474c-aaf6-8859ef3d2dfd" (UID: "3190ca5d-41b1-474c-aaf6-8859ef3d2dfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:56.536144 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:56.536122 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3190ca5d-41b1-474c-aaf6-8859ef3d2dfd-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:56.537996 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:56.537966 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3190ca5d-41b1-474c-aaf6-8859ef3d2dfd-kube-api-access-dgm5g" pod "3190ca5d-41b1-474c-aaf6-8859ef3d2dfd" (UID: "3190ca5d-41b1-474c-aaf6-8859ef3d2dfd"). InnerVolumeSpecName "kube-api-access-dgm5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:56.564426 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:56.564400 5549 generic.go:356] "Generic (PLEG): container finished" podID="3190ca5d-41b1-474c-aaf6-8859ef3d2dfd" containerID="d932f1c08ebec84f639291c68c29d0c03c6bb76d69884e7cdc7f8de93e794bac" exitCode=0 Nov 02 18:34:56.564545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:56.564453 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" Nov 02 18:34:56.564545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:56.564488 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" event={"ID":"3190ca5d-41b1-474c-aaf6-8859ef3d2dfd","Type":"ContainerDied","Data":"d932f1c08ebec84f639291c68c29d0c03c6bb76d69884e7cdc7f8de93e794bac"} Nov 02 18:34:56.564622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:56.564543 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2" event={"ID":"3190ca5d-41b1-474c-aaf6-8859ef3d2dfd","Type":"ContainerDied","Data":"5ea23bd2eab0a08386d6af535739a88a287059141c9234c10bf4e35b8f1c3d0d"} Nov 02 18:34:56.564622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:56.564568 5549 scope.go:122] "RemoveContainer" containerID="d932f1c08ebec84f639291c68c29d0c03c6bb76d69884e7cdc7f8de93e794bac" Nov 02 18:34:56.573403 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:56.573386 5549 scope.go:122] "RemoveContainer" containerID="d932f1c08ebec84f639291c68c29d0c03c6bb76d69884e7cdc7f8de93e794bac" Nov 02 18:34:56.573669 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:34:56.573651 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d932f1c08ebec84f639291c68c29d0c03c6bb76d69884e7cdc7f8de93e794bac\": container with ID starting with d932f1c08ebec84f639291c68c29d0c03c6bb76d69884e7cdc7f8de93e794bac not found: ID does not exist" containerID="d932f1c08ebec84f639291c68c29d0c03c6bb76d69884e7cdc7f8de93e794bac" Nov 02 18:34:56.573739 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:56.573680 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d932f1c08ebec84f639291c68c29d0c03c6bb76d69884e7cdc7f8de93e794bac"} err="failed to get container status \"d932f1c08ebec84f639291c68c29d0c03c6bb76d69884e7cdc7f8de93e794bac\": rpc error: code = NotFound desc = could not find container \"d932f1c08ebec84f639291c68c29d0c03c6bb76d69884e7cdc7f8de93e794bac\": container with ID starting with d932f1c08ebec84f639291c68c29d0c03c6bb76d69884e7cdc7f8de93e794bac not found: ID does not exist" Nov 02 18:34:56.587567 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:56.587536 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2"] Nov 02 18:34:56.590022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:56.589996 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9165c3-74885664f5-wvkn2"] Nov 02 18:34:56.636889 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:56.636863 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgm5g\" (UniqueName: \"kubernetes.io/projected/3190ca5d-41b1-474c-aaf6-8859ef3d2dfd-kube-api-access-dgm5g\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:57.839164 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:57.839122 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:34:58.317545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:34:58.317506 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3190ca5d-41b1-474c-aaf6-8859ef3d2dfd" path="/var/lib/kubelet/pods/3190ca5d-41b1-474c-aaf6-8859ef3d2dfd/volumes" Nov 02 18:35:01.989442 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:01.989401 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:03.046355 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:03.046321 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:03.368044 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:03.367940 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:35:03.954086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:03.954052 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:04.099411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:04.099378 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:04.783815 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:04.783769 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" podUID="23593f20-8dc6-4ff6-93f4-d9b1fc859341" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:35:04.784185 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:04.784155 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" podUID="609d560e-9146-4fbf-9aeb-1ca65e5d3971" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:35:14.784528 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:14.784486 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" podUID="23593f20-8dc6-4ff6-93f4-d9b1fc859341" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:35:14.784528 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:14.784511 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" podUID="609d560e-9146-4fbf-9aeb-1ca65e5d3971" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:35:16.181036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.181007 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" Nov 02 18:35:16.183810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.183788 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" Nov 02 18:35:16.328490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.328458 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23593f20-8dc6-4ff6-93f4-d9b1fc859341-kube-api-access-zqlg4\" (UniqueName: \"kubernetes.io/projected/23593f20-8dc6-4ff6-93f4-d9b1fc859341-kube-api-access-zqlg4\") pod \"23593f20-8dc6-4ff6-93f4-d9b1fc859341\" (UID: \"23593f20-8dc6-4ff6-93f4-d9b1fc859341\") " Nov 02 18:35:16.328661 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.328552 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/609d560e-9146-4fbf-9aeb-1ca65e5d3971-kube-api-access-zb2w9\" (UniqueName: \"kubernetes.io/projected/609d560e-9146-4fbf-9aeb-1ca65e5d3971-kube-api-access-zb2w9\") pod \"609d560e-9146-4fbf-9aeb-1ca65e5d3971\" (UID: \"609d560e-9146-4fbf-9aeb-1ca65e5d3971\") " Nov 02 18:35:16.328661 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.328612 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/609d560e-9146-4fbf-9aeb-1ca65e5d3971-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609d560e-9146-4fbf-9aeb-1ca65e5d3971-plugins-config-volume\") pod \"609d560e-9146-4fbf-9aeb-1ca65e5d3971\" (UID: \"609d560e-9146-4fbf-9aeb-1ca65e5d3971\") " Nov 02 18:35:16.328661 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.328639 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23593f20-8dc6-4ff6-93f4-d9b1fc859341-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23593f20-8dc6-4ff6-93f4-d9b1fc859341-plugins-config-volume\") pod \"23593f20-8dc6-4ff6-93f4-d9b1fc859341\" (UID: \"23593f20-8dc6-4ff6-93f4-d9b1fc859341\") " Nov 02 18:35:16.329052 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.329023 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609d560e-9146-4fbf-9aeb-1ca65e5d3971-plugins-config-volume" pod "609d560e-9146-4fbf-9aeb-1ca65e5d3971" (UID: "609d560e-9146-4fbf-9aeb-1ca65e5d3971"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:35:16.329181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.329049 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23593f20-8dc6-4ff6-93f4-d9b1fc859341-plugins-config-volume" pod "23593f20-8dc6-4ff6-93f4-d9b1fc859341" (UID: "23593f20-8dc6-4ff6-93f4-d9b1fc859341"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:35:16.330528 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.330511 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23593f20-8dc6-4ff6-93f4-d9b1fc859341-kube-api-access-zqlg4" pod "23593f20-8dc6-4ff6-93f4-d9b1fc859341" (UID: "23593f20-8dc6-4ff6-93f4-d9b1fc859341"). InnerVolumeSpecName "kube-api-access-zqlg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:35:16.330583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.330569 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609d560e-9146-4fbf-9aeb-1ca65e5d3971-kube-api-access-zb2w9" pod "609d560e-9146-4fbf-9aeb-1ca65e5d3971" (UID: "609d560e-9146-4fbf-9aeb-1ca65e5d3971"). InnerVolumeSpecName "kube-api-access-zb2w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:35:16.429773 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.429678 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb2w9\" (UniqueName: \"kubernetes.io/projected/609d560e-9146-4fbf-9aeb-1ca65e5d3971-kube-api-access-zb2w9\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:16.429773 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.429710 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609d560e-9146-4fbf-9aeb-1ca65e5d3971-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:16.429773 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.429723 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23593f20-8dc6-4ff6-93f4-d9b1fc859341-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:16.429773 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.429736 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqlg4\" (UniqueName: \"kubernetes.io/projected/23593f20-8dc6-4ff6-93f4-d9b1fc859341-kube-api-access-zqlg4\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:16.656983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.656436 5549 generic.go:356] "Generic (PLEG): container finished" podID="23593f20-8dc6-4ff6-93f4-d9b1fc859341" containerID="5f289c32be11f7a00c9333de065ac38f7e617d07ce8b83ef8a784416e9ba1e09" exitCode=0 Nov 02 18:35:16.656983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.656638 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" event={"ID":"23593f20-8dc6-4ff6-93f4-d9b1fc859341","Type":"ContainerDied","Data":"5f289c32be11f7a00c9333de065ac38f7e617d07ce8b83ef8a784416e9ba1e09"} Nov 02 18:35:16.656983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.656692 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" event={"ID":"23593f20-8dc6-4ff6-93f4-d9b1fc859341","Type":"ContainerDied","Data":"c2fa95881d1db0c0a69fe0fb2da77ea7fcc496941295709074d62adcfe05092e"} Nov 02 18:35:16.656983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.656721 5549 scope.go:122] "RemoveContainer" containerID="5f289c32be11f7a00c9333de065ac38f7e617d07ce8b83ef8a784416e9ba1e09" Nov 02 18:35:16.656983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.656859 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl" Nov 02 18:35:16.659123 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.659094 5549 generic.go:356] "Generic (PLEG): container finished" podID="609d560e-9146-4fbf-9aeb-1ca65e5d3971" containerID="047efcc3b279a6bb8b7ba49bba4bea599c56776e417d1abc5ac90de17358d6cc" exitCode=0 Nov 02 18:35:16.659266 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.659198 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" Nov 02 18:35:16.659346 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.659172 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" event={"ID":"609d560e-9146-4fbf-9aeb-1ca65e5d3971","Type":"ContainerDied","Data":"047efcc3b279a6bb8b7ba49bba4bea599c56776e417d1abc5ac90de17358d6cc"} Nov 02 18:35:16.659478 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.659459 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8" event={"ID":"609d560e-9146-4fbf-9aeb-1ca65e5d3971","Type":"ContainerDied","Data":"7b1b479c8b63dfce36fcf10c4a37f302f047c6bf6118f9e82e67e797b4d522aa"} Nov 02 18:35:16.667657 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.667634 5549 scope.go:122] "RemoveContainer" containerID="5f289c32be11f7a00c9333de065ac38f7e617d07ce8b83ef8a784416e9ba1e09" Nov 02 18:35:16.668095 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:35:16.667939 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f289c32be11f7a00c9333de065ac38f7e617d07ce8b83ef8a784416e9ba1e09\": container with ID starting with 5f289c32be11f7a00c9333de065ac38f7e617d07ce8b83ef8a784416e9ba1e09 not found: ID does not exist" containerID="5f289c32be11f7a00c9333de065ac38f7e617d07ce8b83ef8a784416e9ba1e09" Nov 02 18:35:16.668095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.667995 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f289c32be11f7a00c9333de065ac38f7e617d07ce8b83ef8a784416e9ba1e09"} err="failed to get container status \"5f289c32be11f7a00c9333de065ac38f7e617d07ce8b83ef8a784416e9ba1e09\": rpc error: code = NotFound desc = could not find container \"5f289c32be11f7a00c9333de065ac38f7e617d07ce8b83ef8a784416e9ba1e09\": container with ID starting with 5f289c32be11f7a00c9333de065ac38f7e617d07ce8b83ef8a784416e9ba1e09 not found: ID does not exist" Nov 02 18:35:16.668095 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.668019 5549 scope.go:122] "RemoveContainer" containerID="047efcc3b279a6bb8b7ba49bba4bea599c56776e417d1abc5ac90de17358d6cc" Nov 02 18:35:16.676290 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.676274 5549 scope.go:122] "RemoveContainer" containerID="047efcc3b279a6bb8b7ba49bba4bea599c56776e417d1abc5ac90de17358d6cc" Nov 02 18:35:16.676527 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:35:16.676511 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"047efcc3b279a6bb8b7ba49bba4bea599c56776e417d1abc5ac90de17358d6cc\": container with ID starting with 047efcc3b279a6bb8b7ba49bba4bea599c56776e417d1abc5ac90de17358d6cc not found: ID does not exist" containerID="047efcc3b279a6bb8b7ba49bba4bea599c56776e417d1abc5ac90de17358d6cc" Nov 02 18:35:16.676583 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.676533 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"047efcc3b279a6bb8b7ba49bba4bea599c56776e417d1abc5ac90de17358d6cc"} err="failed to get container status \"047efcc3b279a6bb8b7ba49bba4bea599c56776e417d1abc5ac90de17358d6cc\": rpc error: code = NotFound desc = could not find container \"047efcc3b279a6bb8b7ba49bba4bea599c56776e417d1abc5ac90de17358d6cc\": container with ID starting with 047efcc3b279a6bb8b7ba49bba4bea599c56776e417d1abc5ac90de17358d6cc not found: ID does not exist" Nov 02 18:35:16.682205 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.682157 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl"] Nov 02 18:35:16.685443 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.685422 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-77f103-6d75f74b87-96cvl"] Nov 02 18:35:16.696436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.696413 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8"] Nov 02 18:35:16.699695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:16.699674 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-77f103-5b4b55b64c-j8tj8"] Nov 02 18:35:17.586356 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:35:17.586322 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:35:18.317439 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:18.317403 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23593f20-8dc6-4ff6-93f4-d9b1fc859341" path="/var/lib/kubelet/pods/23593f20-8dc6-4ff6-93f4-d9b1fc859341/volumes" Nov 02 18:35:18.317795 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:18.317781 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="609d560e-9146-4fbf-9aeb-1ca65e5d3971" path="/var/lib/kubelet/pods/609d560e-9146-4fbf-9aeb-1ca65e5d3971/volumes" Nov 02 18:35:20.777171 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:20.777131 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:20.926010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:20.925976 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:21.129007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:21.128908 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:21.266830 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:21.266788 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:22.557936 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:22.557901 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:23.883824 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:23.883775 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:24.111737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:24.111691 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw"] Nov 02 18:35:24.111943 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:24.111907 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" podUID="f199938d-ac35-4850-bb3b-85cb08feab80" containerName="istio-proxy" containerID="cri-o://ea156511272c1f497a586c2c628e925129034bd532bedc68da72b7cef7ad6e84" gracePeriod=30 Nov 02 18:35:24.201618 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:24.201571 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:25.607608 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:25.607577 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-f74t5_db3845da-3e81-491c-9e2d-09f220ea1b9d/kuadrant-console-plugin/0.log" Nov 02 18:35:26.081491 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:26.081460 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-99bp7_0de90549-6c86-4ce1-9d45-1f3b136fa6e7/limitador/0.log" Nov 02 18:35:27.156930 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:27.156897 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-56bcd69c95-g7ngq_b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54/manager/0.log" Nov 02 18:35:28.120150 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:28.120115 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:28.250887 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:28.250852 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:28.681856 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:28.681826 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4q597_585a619b-ddf1-497a-9fa0-40d105881005/server/0.log" Nov 02 18:35:28.845818 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:28.845789 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2wms8_13885bf3-7cdc-40aa-8e58-01eced7a6249/manager/0.log" Nov 02 18:35:29.058626 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.058596 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-xswfx_aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4/proxy/0.log" Nov 02 18:35:29.373361 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.373337 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 18:35:29.468353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.468319 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-workload-certs\") pod \"f199938d-ac35-4850-bb3b-85cb08feab80\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " Nov 02 18:35:29.468353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.468356 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-credential-socket\") pod \"f199938d-ac35-4850-bb3b-85cb08feab80\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " Nov 02 18:35:29.468611 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.468412 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-workload-socket\") pod \"f199938d-ac35-4850-bb3b-85cb08feab80\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " Nov 02 18:35:29.468611 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.468490 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f199938d-ac35-4850-bb3b-85cb08feab80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f199938d-ac35-4850-bb3b-85cb08feab80-istiod-ca-cert\") pod \"f199938d-ac35-4850-bb3b-85cb08feab80\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " Nov 02 18:35:29.468611 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.468527 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f199938d-ac35-4850-bb3b-85cb08feab80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f199938d-ac35-4850-bb3b-85cb08feab80-istio-podinfo\") pod \"f199938d-ac35-4850-bb3b-85cb08feab80\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " Nov 02 18:35:29.468611 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.468573 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-istio-data\") pod \"f199938d-ac35-4850-bb3b-85cb08feab80\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " Nov 02 18:35:29.468611 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.468605 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f199938d-ac35-4850-bb3b-85cb08feab80-istio-token\" (UniqueName: \"kubernetes.io/projected/f199938d-ac35-4850-bb3b-85cb08feab80-istio-token\") pod \"f199938d-ac35-4850-bb3b-85cb08feab80\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " Nov 02 18:35:29.468894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.468639 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-workload-certs" pod "f199938d-ac35-4850-bb3b-85cb08feab80" (UID: "f199938d-ac35-4850-bb3b-85cb08feab80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:29.468894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.468722 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f199938d-ac35-4850-bb3b-85cb08feab80-kube-api-access-69qzh\" (UniqueName: \"kubernetes.io/projected/f199938d-ac35-4850-bb3b-85cb08feab80-kube-api-access-69qzh\") pod \"f199938d-ac35-4850-bb3b-85cb08feab80\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " Nov 02 18:35:29.468894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.468786 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-credential-socket" pod "f199938d-ac35-4850-bb3b-85cb08feab80" (UID: "f199938d-ac35-4850-bb3b-85cb08feab80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:29.468894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.468825 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-workload-socket" pod "f199938d-ac35-4850-bb3b-85cb08feab80" (UID: "f199938d-ac35-4850-bb3b-85cb08feab80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:29.469082 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.468946 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f199938d-ac35-4850-bb3b-85cb08feab80-istiod-ca-cert" pod "f199938d-ac35-4850-bb3b-85cb08feab80" (UID: "f199938d-ac35-4850-bb3b-85cb08feab80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:35:29.469082 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.468992 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-istio-data" pod "f199938d-ac35-4850-bb3b-85cb08feab80" (UID: "f199938d-ac35-4850-bb3b-85cb08feab80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:29.469156 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.469133 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-istio-envoy\") pod \"f199938d-ac35-4850-bb3b-85cb08feab80\" (UID: \"f199938d-ac35-4850-bb3b-85cb08feab80\") " Nov 02 18:35:29.469533 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.469512 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:29.469622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.469541 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:29.469622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.469556 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:29.469622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.469572 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f199938d-ac35-4850-bb3b-85cb08feab80-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:29.469622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.469588 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:29.471036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.471006 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-istio-envoy" pod "f199938d-ac35-4850-bb3b-85cb08feab80" (UID: "f199938d-ac35-4850-bb3b-85cb08feab80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:29.471036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.471019 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f199938d-ac35-4850-bb3b-85cb08feab80-istio-token" pod "f199938d-ac35-4850-bb3b-85cb08feab80" (UID: "f199938d-ac35-4850-bb3b-85cb08feab80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:35:29.471274 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.471249 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f199938d-ac35-4850-bb3b-85cb08feab80-kube-api-access-69qzh" pod "f199938d-ac35-4850-bb3b-85cb08feab80" (UID: "f199938d-ac35-4850-bb3b-85cb08feab80"). InnerVolumeSpecName "kube-api-access-69qzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:35:29.471346 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.471324 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f199938d-ac35-4850-bb3b-85cb08feab80-istio-podinfo" pod "f199938d-ac35-4850-bb3b-85cb08feab80" (UID: "f199938d-ac35-4850-bb3b-85cb08feab80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:35:29.571086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.571050 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f199938d-ac35-4850-bb3b-85cb08feab80-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:29.571086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.571080 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69qzh\" (UniqueName: \"kubernetes.io/projected/f199938d-ac35-4850-bb3b-85cb08feab80-kube-api-access-69qzh\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:29.571300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.571092 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f199938d-ac35-4850-bb3b-85cb08feab80-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:29.571300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.571103 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f199938d-ac35-4850-bb3b-85cb08feab80-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:29.721484 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.721446 5549 generic.go:356] "Generic (PLEG): container finished" podID="f199938d-ac35-4850-bb3b-85cb08feab80" containerID="ea156511272c1f497a586c2c628e925129034bd532bedc68da72b7cef7ad6e84" exitCode=0 Nov 02 18:35:29.721689 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.721496 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" Nov 02 18:35:29.721689 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.721531 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" event={"ID":"f199938d-ac35-4850-bb3b-85cb08feab80","Type":"ContainerDied","Data":"ea156511272c1f497a586c2c628e925129034bd532bedc68da72b7cef7ad6e84"} Nov 02 18:35:29.721689 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.721626 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw" event={"ID":"f199938d-ac35-4850-bb3b-85cb08feab80","Type":"ContainerDied","Data":"b9a7f593f0ff13a47737cd87d122debcc854de45f2f3c1386149d74f14fe2219"} Nov 02 18:35:29.721689 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.721654 5549 scope.go:122] "RemoveContainer" containerID="ea156511272c1f497a586c2c628e925129034bd532bedc68da72b7cef7ad6e84" Nov 02 18:35:29.731059 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.731041 5549 scope.go:122] "RemoveContainer" containerID="ea156511272c1f497a586c2c628e925129034bd532bedc68da72b7cef7ad6e84" Nov 02 18:35:29.731294 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:35:29.731276 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea156511272c1f497a586c2c628e925129034bd532bedc68da72b7cef7ad6e84\": container with ID starting with ea156511272c1f497a586c2c628e925129034bd532bedc68da72b7cef7ad6e84 not found: ID does not exist" containerID="ea156511272c1f497a586c2c628e925129034bd532bedc68da72b7cef7ad6e84" Nov 02 18:35:29.731352 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.731301 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea156511272c1f497a586c2c628e925129034bd532bedc68da72b7cef7ad6e84"} err="failed to get container status \"ea156511272c1f497a586c2c628e925129034bd532bedc68da72b7cef7ad6e84\": rpc error: code = NotFound desc = could not find container \"ea156511272c1f497a586c2c628e925129034bd532bedc68da72b7cef7ad6e84\": container with ID starting with ea156511272c1f497a586c2c628e925129034bd532bedc68da72b7cef7ad6e84 not found: ID does not exist" Nov 02 18:35:29.744709 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.744684 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw"] Nov 02 18:35:29.749210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:29.749188 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-4j6lw"] Nov 02 18:35:30.317632 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:30.317551 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f199938d-ac35-4850-bb3b-85cb08feab80" path="/var/lib/kubelet/pods/f199938d-ac35-4850-bb3b-85cb08feab80/volumes" Nov 02 18:35:33.692526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.692489 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:33.971596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.971558 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh"] Nov 02 18:35:33.972006 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.971988 5549 cpu_manager.go:414] "containerMap: removing container" podUID="bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" containerName="istio-proxy" Nov 02 18:35:33.972006 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972006 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" containerName="istio-proxy" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972020 5549 cpu_manager.go:414] "containerMap: removing container" podUID="5700c62a-902c-4366-8a4e-8d8ca71b3eef" containerName="payload-pre-processing" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972026 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5700c62a-902c-4366-8a4e-8d8ca71b3eef" containerName="payload-pre-processing" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972043 5549 cpu_manager.go:414] "containerMap: removing container" podUID="31758614-1d84-4256-ba23-aed7dd3dd328" containerName="payload-pre-processing" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972048 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31758614-1d84-4256-ba23-aed7dd3dd328" containerName="payload-pre-processing" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972055 5549 cpu_manager.go:414] "containerMap: removing container" podUID="c2b8094f-68c0-461b-b5c9-c6a431267bf2" containerName="payload-processing" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972059 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b8094f-68c0-461b-b5c9-c6a431267bf2" containerName="payload-processing" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972065 5549 cpu_manager.go:414] "containerMap: removing container" podUID="23593f20-8dc6-4ff6-93f4-d9b1fc859341" containerName="payload-pre-processing" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972070 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23593f20-8dc6-4ff6-93f4-d9b1fc859341" containerName="payload-pre-processing" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972077 5549 cpu_manager.go:414] "containerMap: removing container" podUID="821f0afb-87af-4690-a871-6244febd580d" containerName="payload-pre-processing" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972082 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="821f0afb-87af-4690-a871-6244febd580d" containerName="payload-pre-processing" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972090 5549 cpu_manager.go:414] "containerMap: removing container" podUID="fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9" containerName="payload-pre-processing" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972095 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9" containerName="payload-pre-processing" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972102 5549 cpu_manager.go:414] "containerMap: removing container" podUID="f2ab9120-6ff3-489c-8dd2-bfb0f1201172" containerName="payload-pre-processing" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972107 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ab9120-6ff3-489c-8dd2-bfb0f1201172" containerName="payload-pre-processing" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972113 5549 cpu_manager.go:414] "containerMap: removing container" podUID="a575e3e4-796c-4183-9755-8d0f73d233ca" containerName="main" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972118 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a575e3e4-796c-4183-9755-8d0f73d233ca" containerName="main" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972123 5549 cpu_manager.go:414] "containerMap: removing container" podUID="55ebe7be-c46e-4fb3-b7df-cf2a522b9f70" containerName="payload-processing" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972127 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ebe7be-c46e-4fb3-b7df-cf2a522b9f70" containerName="payload-processing" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972135 5549 cpu_manager.go:414] "containerMap: removing container" podUID="4603b3a2-b9a8-4a06-97c0-a97d08f96470" containerName="istio-proxy" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972139 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4603b3a2-b9a8-4a06-97c0-a97d08f96470" containerName="istio-proxy" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972149 5549 cpu_manager.go:414] "containerMap: removing container" podUID="c15281fd-8907-4d15-a06e-85a772e3c837" containerName="maas-api" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972154 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15281fd-8907-4d15-a06e-85a772e3c837" containerName="maas-api" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972162 5549 cpu_manager.go:414] "containerMap: removing container" podUID="12621cb6-273e-4730-85c7-101d872c1b71" containerName="test-curl-7350-2ca161" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972166 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12621cb6-273e-4730-85c7-101d872c1b71" containerName="test-curl-7350-2ca161" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972174 5549 cpu_manager.go:414] "containerMap: removing container" podUID="3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" containerName="istio-proxy" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972179 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" containerName="istio-proxy" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972188 5549 cpu_manager.go:414] "containerMap: removing container" podUID="da89e636-cb83-41a2-ae31-cfbbaff79e79" containerName="istio-proxy" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972192 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da89e636-cb83-41a2-ae31-cfbbaff79e79" containerName="istio-proxy" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972200 5549 cpu_manager.go:414] "containerMap: removing container" podUID="3190ca5d-41b1-474c-aaf6-8859ef3d2dfd" containerName="payload-pre-processing" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972205 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3190ca5d-41b1-474c-aaf6-8859ef3d2dfd" containerName="payload-pre-processing" Nov 02 18:35:33.972191 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972213 5549 cpu_manager.go:414] "containerMap: removing container" podUID="bc67d335-3268-420d-ab7b-e63af8d1165f" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972218 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc67d335-3268-420d-ab7b-e63af8d1165f" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972225 5549 cpu_manager.go:414] "containerMap: removing container" podUID="3ea5ef1e-e3a6-47cf-a908-843442f86e83" containerName="payload-pre-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972229 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea5ef1e-e3a6-47cf-a908-843442f86e83" containerName="payload-pre-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972237 5549 cpu_manager.go:414] "containerMap: removing container" podUID="609d560e-9146-4fbf-9aeb-1ca65e5d3971" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972242 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="609d560e-9146-4fbf-9aeb-1ca65e5d3971" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972249 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7fe6f607-ad08-4d0c-b92f-1618ce0aa86b" containerName="maas-api" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972253 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe6f607-ad08-4d0c-b92f-1618ce0aa86b" containerName="maas-api" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972263 5549 cpu_manager.go:414] "containerMap: removing container" podUID="b612d0da-2500-4091-a03e-89faeb9c435d" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972268 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b612d0da-2500-4091-a03e-89faeb9c435d" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972274 5549 cpu_manager.go:414] "containerMap: removing container" podUID="521107be-a789-485f-8827-235aae17ce7e" containerName="main" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972278 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521107be-a789-485f-8827-235aae17ce7e" containerName="main" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972284 5549 cpu_manager.go:414] "containerMap: removing container" podUID="a011b9f5-8629-4514-b5db-52b58316d975" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972288 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a011b9f5-8629-4514-b5db-52b58316d975" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972295 5549 cpu_manager.go:414] "containerMap: removing container" podUID="8062774b-5045-48ff-9846-c20c1ea18860" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972299 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8062774b-5045-48ff-9846-c20c1ea18860" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972305 5549 cpu_manager.go:414] "containerMap: removing container" podUID="dca8565f-3d0f-4154-ba3f-9e99a38af3fe" containerName="maas-api" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972309 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dca8565f-3d0f-4154-ba3f-9e99a38af3fe" containerName="maas-api" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972316 5549 cpu_manager.go:414] "containerMap: removing container" podUID="95748775-ad15-4a15-8031-a0486a2cf317" containerName="payload-pre-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972320 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95748775-ad15-4a15-8031-a0486a2cf317" containerName="payload-pre-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972329 5549 cpu_manager.go:414] "containerMap: removing container" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972333 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972339 5549 cpu_manager.go:414] "containerMap: removing container" podUID="ed3785d4-1670-4f59-8f60-20e66fc3210b" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972343 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3785d4-1670-4f59-8f60-20e66fc3210b" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972349 5549 cpu_manager.go:414] "containerMap: removing container" podUID="f199938d-ac35-4850-bb3b-85cb08feab80" containerName="istio-proxy" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972353 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f199938d-ac35-4850-bb3b-85cb08feab80" containerName="istio-proxy" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972362 5549 cpu_manager.go:414] "containerMap: removing container" podUID="d132c2c1-5fab-4ab4-82f1-892f5aad2771" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972366 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d132c2c1-5fab-4ab4-82f1-892f5aad2771" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972373 5549 cpu_manager.go:414] "containerMap: removing container" podUID="189c356c-d893-4df2-b800-97fd592fbe62" containerName="maas-api" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972377 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189c356c-d893-4df2-b800-97fd592fbe62" containerName="maas-api" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972431 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d132c2c1-5fab-4ab4-82f1-892f5aad2771" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972438 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="189c356c-d893-4df2-b800-97fd592fbe62" containerName="maas-api" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972445 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfcfb5bd-0bbe-4347-9eaa-89bbef082fdb" containerName="istio-proxy" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972452 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5700c62a-902c-4366-8a4e-8d8ca71b3eef" containerName="payload-pre-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972462 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="31758614-1d84-4256-ba23-aed7dd3dd328" containerName="payload-pre-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972467 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b8094f-68c0-461b-b5c9-c6a431267bf2" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972474 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="23593f20-8dc6-4ff6-93f4-d9b1fc859341" containerName="payload-pre-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972480 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="821f0afb-87af-4690-a871-6244febd580d" containerName="payload-pre-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972486 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa32712b-4cd8-4ce5-99a1-bbd2e00da2f9" containerName="payload-pre-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972491 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ab9120-6ff3-489c-8dd2-bfb0f1201172" containerName="payload-pre-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972496 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a575e3e4-796c-4183-9755-8d0f73d233ca" containerName="main" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972501 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ebe7be-c46e-4fb3-b7df-cf2a522b9f70" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972506 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="4603b3a2-b9a8-4a06-97c0-a97d08f96470" containerName="istio-proxy" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972513 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15281fd-8907-4d15-a06e-85a772e3c837" containerName="maas-api" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972518 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="12621cb6-273e-4730-85c7-101d872c1b71" containerName="test-curl-7350-2ca161" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972523 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" containerName="istio-proxy" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972528 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="da89e636-cb83-41a2-ae31-cfbbaff79e79" containerName="istio-proxy" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972532 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3190ca5d-41b1-474c-aaf6-8859ef3d2dfd" containerName="payload-pre-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972537 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc67d335-3268-420d-ab7b-e63af8d1165f" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972542 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea5ef1e-e3a6-47cf-a908-843442f86e83" containerName="payload-pre-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972548 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="609d560e-9146-4fbf-9aeb-1ca65e5d3971" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972552 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe6f607-ad08-4d0c-b92f-1618ce0aa86b" containerName="maas-api" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972556 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="b612d0da-2500-4091-a03e-89faeb9c435d" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972563 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" containerName="istio-proxy" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972569 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="521107be-a789-485f-8827-235aae17ce7e" containerName="main" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972574 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a011b9f5-8629-4514-b5db-52b58316d975" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972578 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8062774b-5045-48ff-9846-c20c1ea18860" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972583 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="dca8565f-3d0f-4154-ba3f-9e99a38af3fe" containerName="maas-api" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972588 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="95748775-ad15-4a15-8031-a0486a2cf317" containerName="payload-pre-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972595 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3043cd28-bbab-4205-872c-2428bc46bbdc" containerName="main" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972601 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3785d4-1670-4f59-8f60-20e66fc3210b" containerName="payload-processing" Nov 02 18:35:33.974002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.972606 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="f199938d-ac35-4850-bb3b-85cb08feab80" containerName="istio-proxy" Nov 02 18:35:33.977456 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.977245 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:33.987019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.986400 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f8e1abfd-openshift-default-dockercfg-mqqc7\"" Nov 02 18:35:33.987019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:33.986723 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh"] Nov 02 18:35:34.010650 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.010621 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-podinfo\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.010783 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.010657 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-workload-socket\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.010783 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.010677 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-envoy\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.010783 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.010698 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-workload-certs\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.010902 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.010812 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-data\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.010902 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.010843 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istiod-ca-cert\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.010902 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.010862 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dckb\" (UniqueName: \"kubernetes.io/projected/253eb7e3-34ea-4cad-bf30-84667ff95d7f-kube-api-access-8dckb\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.011040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.010988 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-token\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.011040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.011029 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-credential-socket\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.055393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.055365 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:34.111911 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.111877 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-token\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.111911 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.111919 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-credential-socket\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.112186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.112010 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-podinfo\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.112186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.112036 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-workload-socket\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.112186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.112066 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-envoy\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.112186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.112096 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-workload-certs\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.112419 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.112262 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-data\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.112419 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.112322 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istiod-ca-cert\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.112419 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.112355 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dckb\" (UniqueName: \"kubernetes.io/projected/253eb7e3-34ea-4cad-bf30-84667ff95d7f-kube-api-access-8dckb\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.112993 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.112969 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-credential-socket\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.113124 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.113093 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-workload-socket\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.113186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.112986 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-workload-certs\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.113400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.113373 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-data\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.113484 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.113394 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istiod-ca-cert\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.115648 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.115619 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-podinfo\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.121174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.121146 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dckb\" (UniqueName: \"kubernetes.io/projected/253eb7e3-34ea-4cad-bf30-84667ff95d7f-kube-api-access-8dckb\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.121617 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.121594 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-token\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.135145 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.135120 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-envoy\") pod \"e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.212110 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.212072 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:34.292822 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.292737 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:34.567329 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.567299 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh"] Nov 02 18:35:34.570235 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:35:34.570204 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253eb7e3_34ea_4cad_bf30_84667ff95d7f.slice/crio-a4e6bef13a4055ead919f6717dfe22eb6e2bbeea1cc52c4fa55720b5aef3ce00 WatchSource:0}: Error finding container a4e6bef13a4055ead919f6717dfe22eb6e2bbeea1cc52c4fa55720b5aef3ce00: Status 404 returned error can't find the container with id a4e6bef13a4055ead919f6717dfe22eb6e2bbeea1cc52c4fa55720b5aef3ce00 Nov 02 18:35:34.572407 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.572369 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:35:34.572496 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.572436 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:35:34.572496 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.572463 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:35:34.750731 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.750694 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" event={"ID":"253eb7e3-34ea-4cad-bf30-84667ff95d7f","Type":"ContainerStarted","Data":"3b70c9df078f15101e623baa78248b428a83167aceb96a5a6e851270074752d1"} Nov 02 18:35:34.751159 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.750751 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" event={"ID":"253eb7e3-34ea-4cad-bf30-84667ff95d7f","Type":"ContainerStarted","Data":"a4e6bef13a4055ead919f6717dfe22eb6e2bbeea1cc52c4fa55720b5aef3ce00"} Nov 02 18:35:34.770341 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:34.770289 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" podStartSLOduration=1.770272275 podStartE2EDuration="1.770272275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:35:34.769100053 +0000 UTC m=+3706.557605619" watchObservedRunningTime="2026-11-02 18:35:34.770272275 +0000 UTC m=+3706.558777802" Nov 02 18:35:35.293712 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:35.293674 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:35.298625 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:35.298598 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:35.755757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:35.755725 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:35.756724 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:35.756703 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:36.239567 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:36.239525 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:36.380599 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:36.380563 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:36.732052 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:36.732023 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:37.403743 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.403698 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8"] Nov 02 18:35:37.405358 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.405321 5549 cpu_manager.go:414] "containerMap: removing container" podUID="3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" containerName="istio-proxy" Nov 02 18:35:37.405571 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.405547 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c9fbbb3-23a8-4f3f-a954-085d739a4ca2" containerName="istio-proxy" Nov 02 18:35:37.414921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.411207 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" Nov 02 18:35:37.416739 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.416711 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8"] Nov 02 18:35:37.418817 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.418767 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f8e1abfd-serving-cert\"" Nov 02 18:35:37.442728 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.442697 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:37.444011 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.443983 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19d9d63c-48cf-4818-87d4-302400a2c5e4-metrics-certs\") pod \"maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8\" (UID: \"19d9d63c-48cf-4818-87d4-302400a2c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" Nov 02 18:35:37.444148 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.444020 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19d9d63c-48cf-4818-87d4-302400a2c5e4-maas-api-tls\") pod \"maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8\" (UID: \"19d9d63c-48cf-4818-87d4-302400a2c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" Nov 02 18:35:37.444148 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.444108 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d9d63c-48cf-4818-87d4-302400a2c5e4-openshift-service-ca\") pod \"maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8\" (UID: \"19d9d63c-48cf-4818-87d4-302400a2c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" Nov 02 18:35:37.444290 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.444196 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txzq6\" (UniqueName: \"kubernetes.io/projected/19d9d63c-48cf-4818-87d4-302400a2c5e4-kube-api-access-txzq6\") pod \"maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8\" (UID: \"19d9d63c-48cf-4818-87d4-302400a2c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" Nov 02 18:35:37.448553 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.448530 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw"] Nov 02 18:35:37.451999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.451981 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw" Nov 02 18:35:37.459695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.459596 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f8e1abfd\"" Nov 02 18:35:37.461268 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.461244 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw"] Nov 02 18:35:37.519693 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.519666 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f8e1abfd-dockercfg-q5s4t\"" Nov 02 18:35:37.545437 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.545384 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d9d63c-48cf-4818-87d4-302400a2c5e4-openshift-service-ca\") pod \"maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8\" (UID: \"19d9d63c-48cf-4818-87d4-302400a2c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" Nov 02 18:35:37.545643 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.545503 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txzq6\" (UniqueName: \"kubernetes.io/projected/19d9d63c-48cf-4818-87d4-302400a2c5e4-kube-api-access-txzq6\") pod \"maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8\" (UID: \"19d9d63c-48cf-4818-87d4-302400a2c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" Nov 02 18:35:37.545758 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.545734 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wb2x\" (UniqueName: \"kubernetes.io/projected/8e916e35-eeb6-45cc-9764-4827410a03f6-kube-api-access-4wb2x\") pod \"payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw\" (UID: \"8e916e35-eeb6-45cc-9764-4827410a03f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw" Nov 02 18:35:37.546340 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.546245 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19d9d63c-48cf-4818-87d4-302400a2c5e4-metrics-certs\") pod \"maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8\" (UID: \"19d9d63c-48cf-4818-87d4-302400a2c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" Nov 02 18:35:37.546340 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.546325 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19d9d63c-48cf-4818-87d4-302400a2c5e4-maas-api-tls\") pod \"maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8\" (UID: \"19d9d63c-48cf-4818-87d4-302400a2c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" Nov 02 18:35:37.546579 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.546403 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e916e35-eeb6-45cc-9764-4827410a03f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw\" (UID: \"8e916e35-eeb6-45cc-9764-4827410a03f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw" Nov 02 18:35:37.546579 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.546421 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d9d63c-48cf-4818-87d4-302400a2c5e4-openshift-service-ca\") pod \"maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8\" (UID: \"19d9d63c-48cf-4818-87d4-302400a2c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" Nov 02 18:35:37.549465 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.549435 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19d9d63c-48cf-4818-87d4-302400a2c5e4-maas-api-tls\") pod \"maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8\" (UID: \"19d9d63c-48cf-4818-87d4-302400a2c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" Nov 02 18:35:37.549617 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.549586 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19d9d63c-48cf-4818-87d4-302400a2c5e4-metrics-certs\") pod \"maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8\" (UID: \"19d9d63c-48cf-4818-87d4-302400a2c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" Nov 02 18:35:37.560730 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.560630 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txzq6\" (UniqueName: \"kubernetes.io/projected/19d9d63c-48cf-4818-87d4-302400a2c5e4-kube-api-access-txzq6\") pod \"maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8\" (UID: \"19d9d63c-48cf-4818-87d4-302400a2c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" Nov 02 18:35:37.581750 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.581713 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:37.647863 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.647821 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e916e35-eeb6-45cc-9764-4827410a03f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw\" (UID: \"8e916e35-eeb6-45cc-9764-4827410a03f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw" Nov 02 18:35:37.648074 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.647897 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wb2x\" (UniqueName: \"kubernetes.io/projected/8e916e35-eeb6-45cc-9764-4827410a03f6-kube-api-access-4wb2x\") pod \"payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw\" (UID: \"8e916e35-eeb6-45cc-9764-4827410a03f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw" Nov 02 18:35:37.648593 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.648574 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e916e35-eeb6-45cc-9764-4827410a03f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw\" (UID: \"8e916e35-eeb6-45cc-9764-4827410a03f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw" Nov 02 18:35:37.656172 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.656111 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wb2x\" (UniqueName: \"kubernetes.io/projected/8e916e35-eeb6-45cc-9764-4827410a03f6-kube-api-access-4wb2x\") pod \"payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw\" (UID: \"8e916e35-eeb6-45cc-9764-4827410a03f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw" Nov 02 18:35:37.708472 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.708444 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:37.733160 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.733130 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" Nov 02 18:35:37.766130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:37.766099 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw" Nov 02 18:35:38.015984 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:38.015938 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8"] Nov 02 18:35:38.017502 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:35:38.017460 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d9d63c_48cf_4818_87d4_302400a2c5e4.slice/crio-eb0ffc8891ee3b05bb1ea144658b038fe34767da9833864431f6174309db406e WatchSource:0}: Error finding container eb0ffc8891ee3b05bb1ea144658b038fe34767da9833864431f6174309db406e: Status 404 returned error can't find the container with id eb0ffc8891ee3b05bb1ea144658b038fe34767da9833864431f6174309db406e Nov 02 18:35:38.018935 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:38.018905 5549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:35:38.050213 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:38.050067 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw"] Nov 02 18:35:38.052612 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:35:38.052582 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e916e35_eeb6_45cc_9764_4827410a03f6.slice/crio-cf1c09f7ddbc9d256d5fb9aa09a80be8fcdbbaf6c6172d5dfc8c3a014b26e0fa WatchSource:0}: Error finding container cf1c09f7ddbc9d256d5fb9aa09a80be8fcdbbaf6c6172d5dfc8c3a014b26e0fa: Status 404 returned error can't find the container with id cf1c09f7ddbc9d256d5fb9aa09a80be8fcdbbaf6c6172d5dfc8c3a014b26e0fa Nov 02 18:35:38.317502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:38.317414 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:38.771250 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:38.771206 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" event={"ID":"19d9d63c-48cf-4818-87d4-302400a2c5e4","Type":"ContainerStarted","Data":"7318f5cc6e6d746364112fbccab8445cf5ad7bd894f87decc101b1b28648b27f"} Nov 02 18:35:38.771738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:38.771274 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" event={"ID":"19d9d63c-48cf-4818-87d4-302400a2c5e4","Type":"ContainerStarted","Data":"eb0ffc8891ee3b05bb1ea144658b038fe34767da9833864431f6174309db406e"} Nov 02 18:35:38.771738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:38.771311 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" Nov 02 18:35:38.772782 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:38.772752 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw" event={"ID":"8e916e35-eeb6-45cc-9764-4827410a03f6","Type":"ContainerStarted","Data":"1d81e4a5ad80c687f4b4292a37932addff6367b423cee7bf99c5f0325ee499ff"} Nov 02 18:35:38.772909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:38.772795 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw" event={"ID":"8e916e35-eeb6-45cc-9764-4827410a03f6","Type":"ContainerStarted","Data":"cf1c09f7ddbc9d256d5fb9aa09a80be8fcdbbaf6c6172d5dfc8c3a014b26e0fa"} Nov 02 18:35:38.772909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:38.772829 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw" Nov 02 18:35:38.789200 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:38.789158 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" podStartSLOduration=1.2480337910000001 podStartE2EDuration="1.78914421s" totalImagesPullingTime="541.110419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:35:38.787941593 +0000 UTC m=+3710.576447113" watchObservedRunningTime="2026-11-02 18:35:38.78914421 +0000 UTC m=+3710.577649738" Nov 02 18:35:38.804692 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:38.804645 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw" podStartSLOduration=1.804632402 podStartE2EDuration="1.804632402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:35:38.80307701 +0000 UTC m=+3710.591582537" watchObservedRunningTime="2026-11-02 18:35:38.804632402 +0000 UTC m=+3710.593137950" Nov 02 18:35:39.010481 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:39.010423 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:39.163482 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:39.163445 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:35:39.722062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:39.722028 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:40.421430 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:40.421395 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:41.119590 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:41.119551 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:41.807932 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:41.807869 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:42.423289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:42.423251 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:43.206118 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:43.204528 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:43.810708 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:43.810672 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:44.409515 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:44.409477 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:44.804636 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:44.804599 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" Nov 02 18:35:45.016788 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:45.016752 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:45.720235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:45.720195 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:45.855427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:45.855348 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:46.664143 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:46.664106 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:47.352908 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:47.352863 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:47.586753 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:35:47.586711 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:35:47.997986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:47.997925 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:48.698077 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:48.698044 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:48.846609 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:48.846572 5549 generic.go:356] "Generic (PLEG): container finished" podID="253eb7e3-34ea-4cad-bf30-84667ff95d7f" containerID="3b70c9df078f15101e623baa78248b428a83167aceb96a5a6e851270074752d1" exitCode=0 Nov 02 18:35:48.846802 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:48.846654 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" event={"ID":"253eb7e3-34ea-4cad-bf30-84667ff95d7f","Type":"ContainerDied","Data":"3b70c9df078f15101e623baa78248b428a83167aceb96a5a6e851270074752d1"} Nov 02 18:35:48.847130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:48.847112 5549 scope.go:122] "RemoveContainer" containerID="3b70c9df078f15101e623baa78248b428a83167aceb96a5a6e851270074752d1" Nov 02 18:35:48.848307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:48.848273 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:35:48.848392 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:48.848350 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:35:48.848433 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:48.848400 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:35:49.314117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:49.314063 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:49.805168 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:49.805131 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw" Nov 02 18:35:49.853348 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:49.853311 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" event={"ID":"253eb7e3-34ea-4cad-bf30-84667ff95d7f","Type":"ContainerStarted","Data":"447755ed39518523e20ec87273716b8d740003f763f820cd1f9ca0f3d9d27fb4"} Nov 02 18:35:50.119772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:50.119645 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:50.298976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:50.298307 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:50.305026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:50.304990 5549 patch_prober.go:29] interesting pod/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Nov 02 18:35:50.305192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:50.305047 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" podUID="253eb7e3-34ea-4cad-bf30-84667ff95d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Nov 02 18:35:50.708324 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:50.707834 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:50.756202 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:50.756167 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:51.304174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:51.304135 5549 patch_prober.go:29] interesting pod/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Nov 02 18:35:51.305189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:51.305151 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" podUID="253eb7e3-34ea-4cad-bf30-84667ff95d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Nov 02 18:35:51.315233 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:51.315188 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:51.923243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:51.923206 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:52.303980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:52.301050 5549 patch_prober.go:29] interesting pod/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Nov 02 18:35:52.303980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:52.301129 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" podUID="253eb7e3-34ea-4cad-bf30-84667ff95d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Nov 02 18:35:52.903981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:52.901153 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:53.299475 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:53.299428 5549 patch_prober.go:29] interesting pod/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Nov 02 18:35:53.299810 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:53.299775 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" podUID="253eb7e3-34ea-4cad-bf30-84667ff95d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Nov 02 18:35:53.512402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:53.512360 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:54.118201 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:54.118160 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:54.295988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:54.295766 5549 patch_prober.go:29] interesting pod/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Nov 02 18:35:54.295988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:54.295841 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" podUID="253eb7e3-34ea-4cad-bf30-84667ff95d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Nov 02 18:35:54.922872 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:54.922837 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:55.297569 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:55.297539 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:55.298499 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:55.298476 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:35:55.506381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:55.506347 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:56.117525 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:56.117491 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:56.804143 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:56.804101 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:57.417096 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:57.417065 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:58.107391 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:58.107352 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:58.781862 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:58.781824 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:35:59.422093 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:35:59.422029 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:36:00.010759 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:00.010716 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:36:00.622340 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:00.622310 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:36:01.262793 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:01.262750 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:36:01.905490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:01.905450 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:36:02.161844 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:02.161735 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw"] Nov 02 18:36:02.383189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:02.383155 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:36:02.520279 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:02.520228 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:36:02.652502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:02.652464 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:36:02.941893 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:02.941800 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8"] Nov 02 18:36:02.944277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:02.942036 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" podUID="19d9d63c-48cf-4818-87d4-302400a2c5e4" containerName="maas-api" containerID="cri-o://7318f5cc6e6d746364112fbccab8445cf5ad7bd894f87decc101b1b28648b27f" gracePeriod=30 Nov 02 18:36:03.064287 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.064174 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:36:03.196773 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.196745 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:36:03.198182 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.198159 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" Nov 02 18:36:03.326242 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.326206 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:36:03.338212 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.338188 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19d9d63c-48cf-4818-87d4-302400a2c5e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/19d9d63c-48cf-4818-87d4-302400a2c5e4-metrics-certs\") pod \"19d9d63c-48cf-4818-87d4-302400a2c5e4\" (UID: \"19d9d63c-48cf-4818-87d4-302400a2c5e4\") " Nov 02 18:36:03.338308 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.338281 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19d9d63c-48cf-4818-87d4-302400a2c5e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19d9d63c-48cf-4818-87d4-302400a2c5e4-maas-api-tls\") pod \"19d9d63c-48cf-4818-87d4-302400a2c5e4\" (UID: \"19d9d63c-48cf-4818-87d4-302400a2c5e4\") " Nov 02 18:36:03.338445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.338428 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19d9d63c-48cf-4818-87d4-302400a2c5e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d9d63c-48cf-4818-87d4-302400a2c5e4-openshift-service-ca\") pod \"19d9d63c-48cf-4818-87d4-302400a2c5e4\" (UID: \"19d9d63c-48cf-4818-87d4-302400a2c5e4\") " Nov 02 18:36:03.338502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.338476 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d9d63c-48cf-4818-87d4-302400a2c5e4-kube-api-access-txzq6\" (UniqueName: \"kubernetes.io/projected/19d9d63c-48cf-4818-87d4-302400a2c5e4-kube-api-access-txzq6\") pod \"19d9d63c-48cf-4818-87d4-302400a2c5e4\" (UID: \"19d9d63c-48cf-4818-87d4-302400a2c5e4\") " Nov 02 18:36:03.338785 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.338761 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d9d63c-48cf-4818-87d4-302400a2c5e4-openshift-service-ca" pod "19d9d63c-48cf-4818-87d4-302400a2c5e4" (UID: "19d9d63c-48cf-4818-87d4-302400a2c5e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:36:03.340503 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.340477 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d9d63c-48cf-4818-87d4-302400a2c5e4-kube-api-access-txzq6" pod "19d9d63c-48cf-4818-87d4-302400a2c5e4" (UID: "19d9d63c-48cf-4818-87d4-302400a2c5e4"). InnerVolumeSpecName "kube-api-access-txzq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:36:03.340884 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.340857 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19d9d63c-48cf-4818-87d4-302400a2c5e4-maas-api-tls" pod "19d9d63c-48cf-4818-87d4-302400a2c5e4" (UID: "19d9d63c-48cf-4818-87d4-302400a2c5e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:36:03.340938 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.340871 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19d9d63c-48cf-4818-87d4-302400a2c5e4-metrics-certs" pod "19d9d63c-48cf-4818-87d4-302400a2c5e4" (UID: "19d9d63c-48cf-4818-87d4-302400a2c5e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:36:03.440150 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.440110 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txzq6\" (UniqueName: \"kubernetes.io/projected/19d9d63c-48cf-4818-87d4-302400a2c5e4-kube-api-access-txzq6\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:03.440150 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.440142 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19d9d63c-48cf-4818-87d4-302400a2c5e4-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:03.440150 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.440152 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19d9d63c-48cf-4818-87d4-302400a2c5e4-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:03.440150 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.440161 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d9d63c-48cf-4818-87d4-302400a2c5e4-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:03.920016 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.919974 5549 generic.go:356] "Generic (PLEG): container finished" podID="19d9d63c-48cf-4818-87d4-302400a2c5e4" containerID="7318f5cc6e6d746364112fbccab8445cf5ad7bd894f87decc101b1b28648b27f" exitCode=0 Nov 02 18:36:03.920264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.920029 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" Nov 02 18:36:03.920264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.920063 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" event={"ID":"19d9d63c-48cf-4818-87d4-302400a2c5e4","Type":"ContainerDied","Data":"7318f5cc6e6d746364112fbccab8445cf5ad7bd894f87decc101b1b28648b27f"} Nov 02 18:36:03.920264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.920122 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8" event={"ID":"19d9d63c-48cf-4818-87d4-302400a2c5e4","Type":"ContainerDied","Data":"eb0ffc8891ee3b05bb1ea144658b038fe34767da9833864431f6174309db406e"} Nov 02 18:36:03.920264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.920144 5549 scope.go:122] "RemoveContainer" containerID="7318f5cc6e6d746364112fbccab8445cf5ad7bd894f87decc101b1b28648b27f" Nov 02 18:36:03.929659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.929635 5549 scope.go:122] "RemoveContainer" containerID="7318f5cc6e6d746364112fbccab8445cf5ad7bd894f87decc101b1b28648b27f" Nov 02 18:36:03.929904 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:36:03.929886 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7318f5cc6e6d746364112fbccab8445cf5ad7bd894f87decc101b1b28648b27f\": container with ID starting with 7318f5cc6e6d746364112fbccab8445cf5ad7bd894f87decc101b1b28648b27f not found: ID does not exist" containerID="7318f5cc6e6d746364112fbccab8445cf5ad7bd894f87decc101b1b28648b27f" Nov 02 18:36:03.930087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.929910 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7318f5cc6e6d746364112fbccab8445cf5ad7bd894f87decc101b1b28648b27f"} err="failed to get container status \"7318f5cc6e6d746364112fbccab8445cf5ad7bd894f87decc101b1b28648b27f\": rpc error: code = NotFound desc = could not find container \"7318f5cc6e6d746364112fbccab8445cf5ad7bd894f87decc101b1b28648b27f\": container with ID starting with 7318f5cc6e6d746364112fbccab8445cf5ad7bd894f87decc101b1b28648b27f not found: ID does not exist" Nov 02 18:36:03.943858 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.943818 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8"] Nov 02 18:36:03.945961 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:03.945926 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f8e1abfd-78595fbc7d-slhl8"] Nov 02 18:36:04.318089 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:04.318053 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19d9d63c-48cf-4818-87d4-302400a2c5e4" path="/var/lib/kubelet/pods/19d9d63c-48cf-4818-87d4-302400a2c5e4/volumes" Nov 02 18:36:04.351117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:04.351084 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:36:07.162701 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:07.162656 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw" podUID="8e916e35-eeb6-45cc-9764-4827410a03f6" containerName="payload-pre-processing" containerID="cri-o://1d81e4a5ad80c687f4b4292a37932addff6367b423cee7bf99c5f0325ee499ff" gracePeriod=25 Nov 02 18:36:07.411378 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:07.411351 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw" Nov 02 18:36:07.579430 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:07.579395 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e916e35-eeb6-45cc-9764-4827410a03f6-kube-api-access-4wb2x\" (UniqueName: \"kubernetes.io/projected/8e916e35-eeb6-45cc-9764-4827410a03f6-kube-api-access-4wb2x\") pod \"8e916e35-eeb6-45cc-9764-4827410a03f6\" (UID: \"8e916e35-eeb6-45cc-9764-4827410a03f6\") " Nov 02 18:36:07.579632 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:07.579455 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e916e35-eeb6-45cc-9764-4827410a03f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e916e35-eeb6-45cc-9764-4827410a03f6-plugins-config-volume\") pod \"8e916e35-eeb6-45cc-9764-4827410a03f6\" (UID: \"8e916e35-eeb6-45cc-9764-4827410a03f6\") " Nov 02 18:36:07.579859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:07.579836 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e916e35-eeb6-45cc-9764-4827410a03f6-plugins-config-volume" pod "8e916e35-eeb6-45cc-9764-4827410a03f6" (UID: "8e916e35-eeb6-45cc-9764-4827410a03f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:36:07.581523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:07.581496 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e916e35-eeb6-45cc-9764-4827410a03f6-kube-api-access-4wb2x" pod "8e916e35-eeb6-45cc-9764-4827410a03f6" (UID: "8e916e35-eeb6-45cc-9764-4827410a03f6"). InnerVolumeSpecName "kube-api-access-4wb2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:36:07.680219 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:07.680170 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wb2x\" (UniqueName: \"kubernetes.io/projected/8e916e35-eeb6-45cc-9764-4827410a03f6-kube-api-access-4wb2x\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:07.680219 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:07.680216 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e916e35-eeb6-45cc-9764-4827410a03f6-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:07.941625 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:07.941534 5549 generic.go:356] "Generic (PLEG): container finished" podID="8e916e35-eeb6-45cc-9764-4827410a03f6" containerID="1d81e4a5ad80c687f4b4292a37932addff6367b423cee7bf99c5f0325ee499ff" exitCode=0 Nov 02 18:36:07.941625 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:07.941579 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw" event={"ID":"8e916e35-eeb6-45cc-9764-4827410a03f6","Type":"ContainerDied","Data":"1d81e4a5ad80c687f4b4292a37932addff6367b423cee7bf99c5f0325ee499ff"} Nov 02 18:36:07.941625 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:07.941588 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw" Nov 02 18:36:07.941625 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:07.941620 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw" event={"ID":"8e916e35-eeb6-45cc-9764-4827410a03f6","Type":"ContainerDied","Data":"cf1c09f7ddbc9d256d5fb9aa09a80be8fcdbbaf6c6172d5dfc8c3a014b26e0fa"} Nov 02 18:36:07.941945 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:07.941641 5549 scope.go:122] "RemoveContainer" containerID="1d81e4a5ad80c687f4b4292a37932addff6367b423cee7bf99c5f0325ee499ff" Nov 02 18:36:07.951696 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:07.951676 5549 scope.go:122] "RemoveContainer" containerID="1d81e4a5ad80c687f4b4292a37932addff6367b423cee7bf99c5f0325ee499ff" Nov 02 18:36:07.952123 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:36:07.952094 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d81e4a5ad80c687f4b4292a37932addff6367b423cee7bf99c5f0325ee499ff\": container with ID starting with 1d81e4a5ad80c687f4b4292a37932addff6367b423cee7bf99c5f0325ee499ff not found: ID does not exist" containerID="1d81e4a5ad80c687f4b4292a37932addff6367b423cee7bf99c5f0325ee499ff" Nov 02 18:36:07.952221 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:07.952131 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d81e4a5ad80c687f4b4292a37932addff6367b423cee7bf99c5f0325ee499ff"} err="failed to get container status \"1d81e4a5ad80c687f4b4292a37932addff6367b423cee7bf99c5f0325ee499ff\": rpc error: code = NotFound desc = could not find container \"1d81e4a5ad80c687f4b4292a37932addff6367b423cee7bf99c5f0325ee499ff\": container with ID starting with 1d81e4a5ad80c687f4b4292a37932addff6367b423cee7bf99c5f0325ee499ff not found: ID does not exist" Nov 02 18:36:07.966040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:07.966009 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw"] Nov 02 18:36:07.969378 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:07.969357 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f8e1abfd-5d48457bf5-f7sqw"] Nov 02 18:36:08.319494 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:08.319457 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e916e35-eeb6-45cc-9764-4827410a03f6" path="/var/lib/kubelet/pods/8e916e35-eeb6-45cc-9764-4827410a03f6/volumes" Nov 02 18:36:08.340301 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:08.340268 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh"] Nov 02 18:36:08.340535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:08.340512 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" podUID="253eb7e3-34ea-4cad-bf30-84667ff95d7f" containerName="istio-proxy" containerID="cri-o://447755ed39518523e20ec87273716b8d740003f763f820cd1f9ca0f3d9d27fb4" gracePeriod=30 Nov 02 18:36:08.433397 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:08.433359 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:36:10.300597 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:10.300558 5549 patch_prober.go:29] interesting pod/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:36:10.301000 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:10.300632 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" podUID="253eb7e3-34ea-4cad-bf30-84667ff95d7f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:36:11.070973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.070918 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56bc685b8f-jx88t"] Nov 02 18:36:11.071387 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.071367 5549 cpu_manager.go:414] "containerMap: removing container" podUID="8e916e35-eeb6-45cc-9764-4827410a03f6" containerName="payload-pre-processing" Nov 02 18:36:11.071387 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.071386 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e916e35-eeb6-45cc-9764-4827410a03f6" containerName="payload-pre-processing" Nov 02 18:36:11.071571 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.071396 5549 cpu_manager.go:414] "containerMap: removing container" podUID="19d9d63c-48cf-4818-87d4-302400a2c5e4" containerName="maas-api" Nov 02 18:36:11.071571 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.071401 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19d9d63c-48cf-4818-87d4-302400a2c5e4" containerName="maas-api" Nov 02 18:36:11.071571 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.071467 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e916e35-eeb6-45cc-9764-4827410a03f6" containerName="payload-pre-processing" Nov 02 18:36:11.071571 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.071474 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="19d9d63c-48cf-4818-87d4-302400a2c5e4" containerName="maas-api" Nov 02 18:36:11.075886 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.075863 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bc685b8f-jx88t" Nov 02 18:36:11.085566 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.085543 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 18:36:11.085716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.085618 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5hbfs\"" Nov 02 18:36:11.085716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.085676 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 18:36:11.085850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.085783 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 18:36:11.087538 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.087511 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56bc685b8f-jx88t"] Nov 02 18:36:11.213548 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.213500 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-metrics-certs\") pod \"maas-controller-56bc685b8f-jx88t\" (UID: \"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1\") " pod="opendatahub/maas-controller-56bc685b8f-jx88t" Nov 02 18:36:11.213548 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.213549 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-webhook-cert\") pod \"maas-controller-56bc685b8f-jx88t\" (UID: \"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1\") " pod="opendatahub/maas-controller-56bc685b8f-jx88t" Nov 02 18:36:11.213833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.213698 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7j6k\" (UniqueName: \"kubernetes.io/projected/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-kube-api-access-w7j6k\") pod \"maas-controller-56bc685b8f-jx88t\" (UID: \"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1\") " pod="opendatahub/maas-controller-56bc685b8f-jx88t" Nov 02 18:36:11.315049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.314982 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-metrics-certs\") pod \"maas-controller-56bc685b8f-jx88t\" (UID: \"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1\") " pod="opendatahub/maas-controller-56bc685b8f-jx88t" Nov 02 18:36:11.315049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.315050 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-webhook-cert\") pod \"maas-controller-56bc685b8f-jx88t\" (UID: \"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1\") " pod="opendatahub/maas-controller-56bc685b8f-jx88t" Nov 02 18:36:11.315672 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.315112 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7j6k\" (UniqueName: \"kubernetes.io/projected/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-kube-api-access-w7j6k\") pod \"maas-controller-56bc685b8f-jx88t\" (UID: \"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1\") " pod="opendatahub/maas-controller-56bc685b8f-jx88t" Nov 02 18:36:11.317632 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.317601 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-webhook-cert\") pod \"maas-controller-56bc685b8f-jx88t\" (UID: \"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1\") " pod="opendatahub/maas-controller-56bc685b8f-jx88t" Nov 02 18:36:11.317775 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.317748 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-metrics-certs\") pod \"maas-controller-56bc685b8f-jx88t\" (UID: \"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1\") " pod="opendatahub/maas-controller-56bc685b8f-jx88t" Nov 02 18:36:11.323247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.323176 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7j6k\" (UniqueName: \"kubernetes.io/projected/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-kube-api-access-w7j6k\") pod \"maas-controller-56bc685b8f-jx88t\" (UID: \"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1\") " pod="opendatahub/maas-controller-56bc685b8f-jx88t" Nov 02 18:36:11.394381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.394332 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bc685b8f-jx88t" Nov 02 18:36:11.669980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.669930 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56bc685b8f-jx88t"] Nov 02 18:36:11.672856 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:36:11.672818 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a1f80a_fb7d_4ea6_acf7_a3ce5d5aa5f1.slice/crio-29d58e35f6e4c7f7200e3c8a42a2278817e3ca78f37e39ebe04374670fe31145 WatchSource:0}: Error finding container 29d58e35f6e4c7f7200e3c8a42a2278817e3ca78f37e39ebe04374670fe31145: Status 404 returned error can't find the container with id 29d58e35f6e4c7f7200e3c8a42a2278817e3ca78f37e39ebe04374670fe31145 Nov 02 18:36:11.962423 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:11.962382 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bc685b8f-jx88t" event={"ID":"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1","Type":"ContainerStarted","Data":"29d58e35f6e4c7f7200e3c8a42a2278817e3ca78f37e39ebe04374670fe31145"} Nov 02 18:36:13.613732 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.613661 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:36:13.759097 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.759065 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-credential-socket\") pod \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " Nov 02 18:36:13.759240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.759125 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253eb7e3-34ea-4cad-bf30-84667ff95d7f-kube-api-access-8dckb\" (UniqueName: \"kubernetes.io/projected/253eb7e3-34ea-4cad-bf30-84667ff95d7f-kube-api-access-8dckb\") pod \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " Nov 02 18:36:13.759240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.759205 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-workload-socket\") pod \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " Nov 02 18:36:13.759367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.759243 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-data\") pod \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " Nov 02 18:36:13.759367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.759290 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istiod-ca-cert\") pod \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " Nov 02 18:36:13.759367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.759328 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-workload-certs\") pod \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " Nov 02 18:36:13.759367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.759346 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-credential-socket" pod "253eb7e3-34ea-4cad-bf30-84667ff95d7f" (UID: "253eb7e3-34ea-4cad-bf30-84667ff95d7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:36:13.759607 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.759467 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-envoy\") pod \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " Nov 02 18:36:13.759607 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.759502 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-token\" (UniqueName: \"kubernetes.io/projected/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-token\") pod \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " Nov 02 18:36:13.759607 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.759553 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-podinfo\") pod \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\" (UID: \"253eb7e3-34ea-4cad-bf30-84667ff95d7f\") " Nov 02 18:36:13.759775 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.759605 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-data" pod "253eb7e3-34ea-4cad-bf30-84667ff95d7f" (UID: "253eb7e3-34ea-4cad-bf30-84667ff95d7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:36:13.759775 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.759759 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-workload-certs" pod "253eb7e3-34ea-4cad-bf30-84667ff95d7f" (UID: "253eb7e3-34ea-4cad-bf30-84667ff95d7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:36:13.759899 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.759832 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-workload-socket" pod "253eb7e3-34ea-4cad-bf30-84667ff95d7f" (UID: "253eb7e3-34ea-4cad-bf30-84667ff95d7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:36:13.759967 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.759931 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:13.760117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.759969 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:13.760117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.759987 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:13.760117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.760001 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:13.760296 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.760162 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istiod-ca-cert" pod "253eb7e3-34ea-4cad-bf30-84667ff95d7f" (UID: "253eb7e3-34ea-4cad-bf30-84667ff95d7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:36:13.761895 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.761868 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-token" pod "253eb7e3-34ea-4cad-bf30-84667ff95d7f" (UID: "253eb7e3-34ea-4cad-bf30-84667ff95d7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:36:13.762146 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.762121 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253eb7e3-34ea-4cad-bf30-84667ff95d7f-kube-api-access-8dckb" pod "253eb7e3-34ea-4cad-bf30-84667ff95d7f" (UID: "253eb7e3-34ea-4cad-bf30-84667ff95d7f"). InnerVolumeSpecName "kube-api-access-8dckb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:36:13.762209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.762160 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-envoy" pod "253eb7e3-34ea-4cad-bf30-84667ff95d7f" (UID: "253eb7e3-34ea-4cad-bf30-84667ff95d7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:36:13.762209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.762180 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-podinfo" pod "253eb7e3-34ea-4cad-bf30-84667ff95d7f" (UID: "253eb7e3-34ea-4cad-bf30-84667ff95d7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:36:13.861331 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.861292 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:13.861331 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.861324 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:13.861331 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.861336 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:13.861605 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.861346 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253eb7e3-34ea-4cad-bf30-84667ff95d7f-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:13.861605 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.861361 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dckb\" (UniqueName: \"kubernetes.io/projected/253eb7e3-34ea-4cad-bf30-84667ff95d7f-kube-api-access-8dckb\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:13.974101 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.974061 5549 generic.go:356] "Generic (PLEG): container finished" podID="253eb7e3-34ea-4cad-bf30-84667ff95d7f" containerID="447755ed39518523e20ec87273716b8d740003f763f820cd1f9ca0f3d9d27fb4" exitCode=0 Nov 02 18:36:13.974286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.974115 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" Nov 02 18:36:13.974286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.974142 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" event={"ID":"253eb7e3-34ea-4cad-bf30-84667ff95d7f","Type":"ContainerDied","Data":"447755ed39518523e20ec87273716b8d740003f763f820cd1f9ca0f3d9d27fb4"} Nov 02 18:36:13.974286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.974196 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh" event={"ID":"253eb7e3-34ea-4cad-bf30-84667ff95d7f","Type":"ContainerDied","Data":"a4e6bef13a4055ead919f6717dfe22eb6e2bbeea1cc52c4fa55720b5aef3ce00"} Nov 02 18:36:13.974286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.974219 5549 scope.go:122] "RemoveContainer" containerID="447755ed39518523e20ec87273716b8d740003f763f820cd1f9ca0f3d9d27fb4" Nov 02 18:36:13.975987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.975907 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bc685b8f-jx88t" event={"ID":"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1","Type":"ContainerStarted","Data":"ec1669e0e64cc8614010319ee425cf7a51ab7b8ea77f2263f01dc959b6c0dda9"} Nov 02 18:36:13.975987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.975968 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56bc685b8f-jx88t" Nov 02 18:36:13.983393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.983378 5549 scope.go:122] "RemoveContainer" containerID="3b70c9df078f15101e623baa78248b428a83167aceb96a5a6e851270074752d1" Nov 02 18:36:13.991000 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.990982 5549 scope.go:122] "RemoveContainer" containerID="447755ed39518523e20ec87273716b8d740003f763f820cd1f9ca0f3d9d27fb4" Nov 02 18:36:13.991280 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:36:13.991259 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447755ed39518523e20ec87273716b8d740003f763f820cd1f9ca0f3d9d27fb4\": container with ID starting with 447755ed39518523e20ec87273716b8d740003f763f820cd1f9ca0f3d9d27fb4 not found: ID does not exist" containerID="447755ed39518523e20ec87273716b8d740003f763f820cd1f9ca0f3d9d27fb4" Nov 02 18:36:13.991327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.991289 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447755ed39518523e20ec87273716b8d740003f763f820cd1f9ca0f3d9d27fb4"} err="failed to get container status \"447755ed39518523e20ec87273716b8d740003f763f820cd1f9ca0f3d9d27fb4\": rpc error: code = NotFound desc = could not find container \"447755ed39518523e20ec87273716b8d740003f763f820cd1f9ca0f3d9d27fb4\": container with ID starting with 447755ed39518523e20ec87273716b8d740003f763f820cd1f9ca0f3d9d27fb4 not found: ID does not exist" Nov 02 18:36:13.991327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.991307 5549 scope.go:122] "RemoveContainer" containerID="3b70c9df078f15101e623baa78248b428a83167aceb96a5a6e851270074752d1" Nov 02 18:36:13.991521 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:36:13.991503 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b70c9df078f15101e623baa78248b428a83167aceb96a5a6e851270074752d1\": container with ID starting with 3b70c9df078f15101e623baa78248b428a83167aceb96a5a6e851270074752d1 not found: ID does not exist" containerID="3b70c9df078f15101e623baa78248b428a83167aceb96a5a6e851270074752d1" Nov 02 18:36:13.991581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:13.991530 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b70c9df078f15101e623baa78248b428a83167aceb96a5a6e851270074752d1"} err="failed to get container status \"3b70c9df078f15101e623baa78248b428a83167aceb96a5a6e851270074752d1\": rpc error: code = NotFound desc = could not find container \"3b70c9df078f15101e623baa78248b428a83167aceb96a5a6e851270074752d1\": container with ID starting with 3b70c9df078f15101e623baa78248b428a83167aceb96a5a6e851270074752d1 not found: ID does not exist" Nov 02 18:36:14.009055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:14.009009 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-56bc685b8f-jx88t" podStartSLOduration=0.96358207 podStartE2EDuration="3.008997447s" totalImagesPullingTime="2.045415377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:36:14.007880731 +0000 UTC m=+3745.796386321" watchObservedRunningTime="2026-11-02 18:36:14.008997447 +0000 UTC m=+3745.797502972" Nov 02 18:36:14.024242 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:14.024213 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh"] Nov 02 18:36:14.029196 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:14.029168 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f8e1abfd-openshift-default-779df9d59f-bq2gh"] Nov 02 18:36:14.318478 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:14.318394 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="253eb7e3-34ea-4cad-bf30-84667ff95d7f" path="/var/lib/kubelet/pods/253eb7e3-34ea-4cad-bf30-84667ff95d7f/volumes" Nov 02 18:36:17.585493 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:36:17.585446 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:36:17.906250 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:17.906165 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:36:18.043769 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:18.043726 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:36:24.985367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:24.985331 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56bc685b8f-jx88t" Nov 02 18:36:26.243858 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.243822 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:36:26.393967 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.393929 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:36:26.524183 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.524102 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg"] Nov 02 18:36:26.524552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.524538 5549 cpu_manager.go:414] "containerMap: removing container" podUID="253eb7e3-34ea-4cad-bf30-84667ff95d7f" containerName="istio-proxy" Nov 02 18:36:26.524598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.524554 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253eb7e3-34ea-4cad-bf30-84667ff95d7f" containerName="istio-proxy" Nov 02 18:36:26.524598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.524569 5549 cpu_manager.go:414] "containerMap: removing container" podUID="253eb7e3-34ea-4cad-bf30-84667ff95d7f" containerName="istio-proxy" Nov 02 18:36:26.524598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.524575 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253eb7e3-34ea-4cad-bf30-84667ff95d7f" containerName="istio-proxy" Nov 02 18:36:26.524705 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.524643 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="253eb7e3-34ea-4cad-bf30-84667ff95d7f" containerName="istio-proxy" Nov 02 18:36:26.524705 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.524652 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="253eb7e3-34ea-4cad-bf30-84667ff95d7f" containerName="istio-proxy" Nov 02 18:36:26.527776 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.527754 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.541261 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.541232 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg"] Nov 02 18:36:26.552118 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.552084 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c"] Nov 02 18:36:26.555809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.555784 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.567802 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.567773 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c"] Nov 02 18:36:26.591961 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.591897 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.591961 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.591947 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.592182 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.592063 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.592182 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.592117 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.592263 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.592239 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p247x\" (UniqueName: \"kubernetes.io/projected/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-kube-api-access-p247x\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.592307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.592281 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.592307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.592299 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.592375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.592316 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.592375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.592348 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.693343 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.693288 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.693535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.693375 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.693535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.693418 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c208756b-40ff-4691-aad8-9008058e4291-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.693535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.693447 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c208756b-40ff-4691-aad8-9008058e4291-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.693535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.693523 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c208756b-40ff-4691-aad8-9008058e4291-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.693715 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.693578 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p247x\" (UniqueName: \"kubernetes.io/projected/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-kube-api-access-p247x\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.693715 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.693598 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c208756b-40ff-4691-aad8-9008058e4291-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.693715 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.693619 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c208756b-40ff-4691-aad8-9008058e4291-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.693897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.693705 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.693897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.693743 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.693897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.693770 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.693897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.693796 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.693897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.693828 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c208756b-40ff-4691-aad8-9008058e4291-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.693897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.693874 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c208756b-40ff-4691-aad8-9008058e4291-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.694303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.693987 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c208756b-40ff-4691-aad8-9008058e4291-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.694303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.694093 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdww6\" (UniqueName: \"kubernetes.io/projected/c208756b-40ff-4691-aad8-9008058e4291-kube-api-access-zdww6\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.694303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.694119 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.694303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.694146 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.694303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.694153 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.694303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.694154 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.694303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.694194 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.694303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.694225 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.694624 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.694359 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.696322 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.696290 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.696504 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.696486 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.701166 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.701145 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p247x\" (UniqueName: \"kubernetes.io/projected/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-kube-api-access-p247x\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.701275 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.701215 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a7bea79-f2bb-489d-b861-28fa66d0ec4d-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-r2rbg\" (UID: \"6a7bea79-f2bb-489d-b861-28fa66d0ec4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.794942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.794843 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c208756b-40ff-4691-aad8-9008058e4291-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.794942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.794888 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c208756b-40ff-4691-aad8-9008058e4291-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.794942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.794917 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c208756b-40ff-4691-aad8-9008058e4291-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.794942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.794943 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c208756b-40ff-4691-aad8-9008058e4291-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.795315 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.795003 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c208756b-40ff-4691-aad8-9008058e4291-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.795315 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.795026 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c208756b-40ff-4691-aad8-9008058e4291-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.795315 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.795049 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c208756b-40ff-4691-aad8-9008058e4291-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.795315 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.795121 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdww6\" (UniqueName: \"kubernetes.io/projected/c208756b-40ff-4691-aad8-9008058e4291-kube-api-access-zdww6\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.795315 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.795204 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c208756b-40ff-4691-aad8-9008058e4291-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.795575 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.795557 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c208756b-40ff-4691-aad8-9008058e4291-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.795641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.795571 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c208756b-40ff-4691-aad8-9008058e4291-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.795641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.795617 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c208756b-40ff-4691-aad8-9008058e4291-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.795726 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.795678 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c208756b-40ff-4691-aad8-9008058e4291-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.795726 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.795714 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c208756b-40ff-4691-aad8-9008058e4291-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.797446 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.797420 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c208756b-40ff-4691-aad8-9008058e4291-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.797601 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.797585 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c208756b-40ff-4691-aad8-9008058e4291-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.802610 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.802582 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c208756b-40ff-4691-aad8-9008058e4291-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.803266 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.803247 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdww6\" (UniqueName: \"kubernetes.io/projected/c208756b-40ff-4691-aad8-9008058e4291-kube-api-access-zdww6\") pod \"maas-default-gateway-openshift-default-65c448b547-kgx7c\" (UID: \"c208756b-40ff-4691-aad8-9008058e4291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:26.841972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.841925 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:26.871538 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:26.871506 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:27.120167 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:27.120137 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg"] Nov 02 18:36:27.122274 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:36:27.122247 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7bea79_f2bb_489d_b861_28fa66d0ec4d.slice/crio-236176acc86960776b3bf6c60e127e2dc8af0e32d5b567ff8ed18a0615c6398f WatchSource:0}: Error finding container 236176acc86960776b3bf6c60e127e2dc8af0e32d5b567ff8ed18a0615c6398f: Status 404 returned error can't find the container with id 236176acc86960776b3bf6c60e127e2dc8af0e32d5b567ff8ed18a0615c6398f Nov 02 18:36:27.124585 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:27.124546 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:36:27.124684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:27.124626 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:36:27.124684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:27.124653 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:36:27.166300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:27.166274 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c"] Nov 02 18:36:27.168754 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:36:27.168723 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc208756b_40ff_4691_aad8_9008058e4291.slice/crio-fbfed54e2d9c5ae015b533033c5783a105c19049e68121cdcf44a539935799c2 WatchSource:0}: Error finding container fbfed54e2d9c5ae015b533033c5783a105c19049e68121cdcf44a539935799c2: Status 404 returned error can't find the container with id fbfed54e2d9c5ae015b533033c5783a105c19049e68121cdcf44a539935799c2 Nov 02 18:36:27.171172 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:27.171136 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:36:27.171244 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:27.171214 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:36:27.171293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:27.171248 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:36:28.049037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:28.048987 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" event={"ID":"6a7bea79-f2bb-489d-b861-28fa66d0ec4d","Type":"ContainerStarted","Data":"8481a7d1b84bc4192e6e116f814d732b7e6e26b698690d65c6f9093c94265c5c"} Nov 02 18:36:28.049556 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:28.049061 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" event={"ID":"6a7bea79-f2bb-489d-b861-28fa66d0ec4d","Type":"ContainerStarted","Data":"236176acc86960776b3bf6c60e127e2dc8af0e32d5b567ff8ed18a0615c6398f"} Nov 02 18:36:28.050623 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:28.050592 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" event={"ID":"c208756b-40ff-4691-aad8-9008058e4291","Type":"ContainerStarted","Data":"94fa1857e6a1cf245f31e5fed425afd0c9a4e29977c2b1da366177380687d552"} Nov 02 18:36:28.050753 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:28.050647 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" event={"ID":"c208756b-40ff-4691-aad8-9008058e4291","Type":"ContainerStarted","Data":"fbfed54e2d9c5ae015b533033c5783a105c19049e68121cdcf44a539935799c2"} Nov 02 18:36:28.070097 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:28.070041 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" podStartSLOduration=2.070026955 podStartE2EDuration="2.070026955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:36:28.068430452 +0000 UTC m=+3759.856935983" watchObservedRunningTime="2026-11-02 18:36:28.070026955 +0000 UTC m=+3759.858532480" Nov 02 18:36:28.842837 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:28.842787 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:28.844299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:28.844267 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-r2rbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body= Nov 02 18:36:28.844421 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:28.844319 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" podUID="6a7bea79-f2bb-489d-b861-28fa66d0ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" Nov 02 18:36:28.872502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:28.872466 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:28.873803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:28.873777 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-kgx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Nov 02 18:36:28.873980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:28.873831 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" podUID="c208756b-40ff-4691-aad8-9008058e4291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Nov 02 18:36:29.843157 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:29.843111 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-r2rbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body= Nov 02 18:36:29.843581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:29.843185 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" podUID="6a7bea79-f2bb-489d-b861-28fa66d0ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" Nov 02 18:36:29.872741 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:29.872711 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-kgx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Nov 02 18:36:29.872913 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:29.872754 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" podUID="c208756b-40ff-4691-aad8-9008058e4291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Nov 02 18:36:30.843266 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:30.843216 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-r2rbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body= Nov 02 18:36:30.843693 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:30.843283 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" podUID="6a7bea79-f2bb-489d-b861-28fa66d0ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" Nov 02 18:36:30.872717 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:30.872682 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-kgx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Nov 02 18:36:30.872831 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:30.872750 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" podUID="c208756b-40ff-4691-aad8-9008058e4291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Nov 02 18:36:31.842783 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:31.842737 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-r2rbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body= Nov 02 18:36:31.842976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:31.842806 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" podUID="6a7bea79-f2bb-489d-b861-28fa66d0ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" Nov 02 18:36:31.871967 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:31.871915 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-kgx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Nov 02 18:36:31.872430 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:31.872408 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" podUID="c208756b-40ff-4691-aad8-9008058e4291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Nov 02 18:36:32.843278 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:32.843233 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-r2rbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body= Nov 02 18:36:32.843456 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:32.843312 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" podUID="6a7bea79-f2bb-489d-b861-28fa66d0ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" Nov 02 18:36:32.871855 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:32.871822 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-kgx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Nov 02 18:36:32.872026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:32.871878 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" podUID="c208756b-40ff-4691-aad8-9008058e4291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Nov 02 18:36:33.842858 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:33.842816 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-r2rbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body= Nov 02 18:36:33.843103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:33.842874 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" podUID="6a7bea79-f2bb-489d-b861-28fa66d0ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" Nov 02 18:36:33.872933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:33.872894 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-kgx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Nov 02 18:36:33.873337 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:33.872978 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" podUID="c208756b-40ff-4691-aad8-9008058e4291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Nov 02 18:36:35.844023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:35.843983 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-r2rbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:36:35.844479 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:35.844051 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" podUID="6a7bea79-f2bb-489d-b861-28fa66d0ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:36:35.872977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:35.872919 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-kgx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:36:35.873149 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:35.872992 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" podUID="c208756b-40ff-4691-aad8-9008058e4291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:36:36.885247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:36.885197 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-r2rbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:36:36.885795 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:36.885277 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" podUID="6a7bea79-f2bb-489d-b861-28fa66d0ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:36:36.915131 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:36.915089 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-kgx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:36:36.915427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:36.915392 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" podUID="c208756b-40ff-4691-aad8-9008058e4291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:36:37.882559 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:37.882524 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:37.882872 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:37.882841 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:37.883608 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:37.883585 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" Nov 02 18:36:37.904631 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:37.904575 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" podStartSLOduration=11.904562063 podStartE2EDuration="11.904562063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:36:28.102146588 +0000 UTC m=+3759.890652117" watchObservedRunningTime="2026-11-02 18:36:37.904562063 +0000 UTC m=+3769.693067646" Nov 02 18:36:37.916132 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:37.916078 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-kgx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:36:37.916392 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:37.916359 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" podUID="c208756b-40ff-4691-aad8-9008058e4291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:36:37.924731 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:37.924699 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:37.925156 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:37.925129 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:37.925814 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:37.925794 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" Nov 02 18:36:42.878893 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:42.878854 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:36:42.942570 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:42.942536 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56bc685b8f-jx88t"] Nov 02 18:36:42.942775 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:42.942738 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-56bc685b8f-jx88t" podUID="53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1" containerName="manager" containerID="cri-o://ec1669e0e64cc8614010319ee425cf7a51ab7b8ea77f2263f01dc959b6c0dda9" gracePeriod=30 Nov 02 18:36:43.160578 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:43.160483 5549 generic.go:356] "Generic (PLEG): container finished" podID="53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1" containerID="ec1669e0e64cc8614010319ee425cf7a51ab7b8ea77f2263f01dc959b6c0dda9" exitCode=0 Nov 02 18:36:43.160578 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:43.160559 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bc685b8f-jx88t" event={"ID":"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1","Type":"ContainerDied","Data":"ec1669e0e64cc8614010319ee425cf7a51ab7b8ea77f2263f01dc959b6c0dda9"} Nov 02 18:36:43.196633 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:43.196607 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bc685b8f-jx88t" Nov 02 18:36:43.308026 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:43.307985 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-webhook-cert\" (UniqueName: \"kubernetes.io/secret/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-webhook-cert\") pod \"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1\" (UID: \"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1\") " Nov 02 18:36:43.308232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:43.308067 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-kube-api-access-w7j6k\" (UniqueName: \"kubernetes.io/projected/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-kube-api-access-w7j6k\") pod \"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1\" (UID: \"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1\") " Nov 02 18:36:43.308232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:43.308117 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-metrics-certs\") pod \"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1\" (UID: \"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1\") " Nov 02 18:36:43.310101 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:43.310064 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-webhook-cert" pod "53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1" (UID: "53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:36:43.310258 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:43.310177 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-metrics-certs" pod "53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1" (UID: "53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:36:43.310258 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:43.310246 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-kube-api-access-w7j6k" pod "53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1" (UID: "53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1"). InnerVolumeSpecName "kube-api-access-w7j6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:36:43.409223 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:43.409184 5549 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-webhook-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:43.409223 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:43.409216 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7j6k\" (UniqueName: \"kubernetes.io/projected/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-kube-api-access-w7j6k\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:43.409223 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:43.409227 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:44.167274 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:44.167241 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bc685b8f-jx88t" Nov 02 18:36:44.167274 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:44.167253 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bc685b8f-jx88t" event={"ID":"53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1","Type":"ContainerDied","Data":"29d58e35f6e4c7f7200e3c8a42a2278817e3ca78f37e39ebe04374670fe31145"} Nov 02 18:36:44.167869 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:44.167339 5549 scope.go:122] "RemoveContainer" containerID="ec1669e0e64cc8614010319ee425cf7a51ab7b8ea77f2263f01dc959b6c0dda9" Nov 02 18:36:44.189787 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:44.189756 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56bc685b8f-jx88t"] Nov 02 18:36:44.194032 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:44.194004 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-56bc685b8f-jx88t"] Nov 02 18:36:44.318788 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:44.318750 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1" path="/var/lib/kubelet/pods/53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1/volumes" Nov 02 18:36:45.249165 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.249126 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:36:45.357258 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.357221 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56bc685b8f-tzvbj"] Nov 02 18:36:45.357707 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.357687 5549 cpu_manager.go:414] "containerMap: removing container" podUID="53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1" containerName="manager" Nov 02 18:36:45.357816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.357709 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1" containerName="manager" Nov 02 18:36:45.357816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.357792 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a1f80a-fb7d-4ea6-acf7-a3ce5d5aa5f1" containerName="manager" Nov 02 18:36:45.362189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.362166 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bc685b8f-tzvbj" Nov 02 18:36:45.369604 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.369572 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 18:36:45.369802 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.369640 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 18:36:45.370008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.369725 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 18:36:45.370008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.369798 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5hbfs\"" Nov 02 18:36:45.371963 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.371932 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56bc685b8f-tzvbj"] Nov 02 18:36:45.531647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.531554 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/248b5aef-8f56-4496-9bc2-521bc36176ba-webhook-cert\") pod \"maas-controller-56bc685b8f-tzvbj\" (UID: \"248b5aef-8f56-4496-9bc2-521bc36176ba\") " pod="opendatahub/maas-controller-56bc685b8f-tzvbj" Nov 02 18:36:45.531647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.531621 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/248b5aef-8f56-4496-9bc2-521bc36176ba-metrics-certs\") pod \"maas-controller-56bc685b8f-tzvbj\" (UID: \"248b5aef-8f56-4496-9bc2-521bc36176ba\") " pod="opendatahub/maas-controller-56bc685b8f-tzvbj" Nov 02 18:36:45.531852 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.531707 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkvxw\" (UniqueName: \"kubernetes.io/projected/248b5aef-8f56-4496-9bc2-521bc36176ba-kube-api-access-gkvxw\") pod \"maas-controller-56bc685b8f-tzvbj\" (UID: \"248b5aef-8f56-4496-9bc2-521bc36176ba\") " pod="opendatahub/maas-controller-56bc685b8f-tzvbj" Nov 02 18:36:45.632562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.632524 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/248b5aef-8f56-4496-9bc2-521bc36176ba-webhook-cert\") pod \"maas-controller-56bc685b8f-tzvbj\" (UID: \"248b5aef-8f56-4496-9bc2-521bc36176ba\") " pod="opendatahub/maas-controller-56bc685b8f-tzvbj" Nov 02 18:36:45.632746 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.632587 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/248b5aef-8f56-4496-9bc2-521bc36176ba-metrics-certs\") pod \"maas-controller-56bc685b8f-tzvbj\" (UID: \"248b5aef-8f56-4496-9bc2-521bc36176ba\") " pod="opendatahub/maas-controller-56bc685b8f-tzvbj" Nov 02 18:36:45.632746 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.632618 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkvxw\" (UniqueName: \"kubernetes.io/projected/248b5aef-8f56-4496-9bc2-521bc36176ba-kube-api-access-gkvxw\") pod \"maas-controller-56bc685b8f-tzvbj\" (UID: \"248b5aef-8f56-4496-9bc2-521bc36176ba\") " pod="opendatahub/maas-controller-56bc685b8f-tzvbj" Nov 02 18:36:45.635034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.634999 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/248b5aef-8f56-4496-9bc2-521bc36176ba-webhook-cert\") pod \"maas-controller-56bc685b8f-tzvbj\" (UID: \"248b5aef-8f56-4496-9bc2-521bc36176ba\") " pod="opendatahub/maas-controller-56bc685b8f-tzvbj" Nov 02 18:36:45.635168 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.635037 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/248b5aef-8f56-4496-9bc2-521bc36176ba-metrics-certs\") pod \"maas-controller-56bc685b8f-tzvbj\" (UID: \"248b5aef-8f56-4496-9bc2-521bc36176ba\") " pod="opendatahub/maas-controller-56bc685b8f-tzvbj" Nov 02 18:36:45.641134 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.641108 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkvxw\" (UniqueName: \"kubernetes.io/projected/248b5aef-8f56-4496-9bc2-521bc36176ba-kube-api-access-gkvxw\") pod \"maas-controller-56bc685b8f-tzvbj\" (UID: \"248b5aef-8f56-4496-9bc2-521bc36176ba\") " pod="opendatahub/maas-controller-56bc685b8f-tzvbj" Nov 02 18:36:45.677017 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.676972 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bc685b8f-tzvbj" Nov 02 18:36:45.949529 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:45.949501 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56bc685b8f-tzvbj"] Nov 02 18:36:45.951724 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:36:45.951698 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248b5aef_8f56_4496_9bc2_521bc36176ba.slice/crio-067e596a52604e3b9b89ed1368e4533b650a5089adf8ec621afefab1b8c1d5d4 WatchSource:0}: Error finding container 067e596a52604e3b9b89ed1368e4533b650a5089adf8ec621afefab1b8c1d5d4: Status 404 returned error can't find the container with id 067e596a52604e3b9b89ed1368e4533b650a5089adf8ec621afefab1b8c1d5d4 Nov 02 18:36:46.178158 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:46.178117 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bc685b8f-tzvbj" event={"ID":"248b5aef-8f56-4496-9bc2-521bc36176ba","Type":"ContainerStarted","Data":"067e596a52604e3b9b89ed1368e4533b650a5089adf8ec621afefab1b8c1d5d4"} Nov 02 18:36:47.184419 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:47.184380 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bc685b8f-tzvbj" event={"ID":"248b5aef-8f56-4496-9bc2-521bc36176ba","Type":"ContainerStarted","Data":"f07d77ae382e1a46e2783bc2ad844300252e4d4448a9295bedcfeea0b6259f14"} Nov 02 18:36:47.184852 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:47.184480 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56bc685b8f-tzvbj" Nov 02 18:36:47.203910 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:47.203856 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-56bc685b8f-tzvbj" podStartSLOduration=1.767790476 podStartE2EDuration="2.203843216s" totalImagesPullingTime="436.05274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:36:47.202578892 +0000 UTC m=+3778.991084420" watchObservedRunningTime="2026-11-02 18:36:47.203843216 +0000 UTC m=+3778.992348772" Nov 02 18:36:47.586219 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:36:47.586184 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:36:58.191365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:36:58.191323 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56bc685b8f-tzvbj" Nov 02 18:37:03.158273 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:37:03.158185 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:37:04.670867 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:37:04.670828 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:37:17.586779 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:37:17.586734 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:37:20.524753 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:37:20.524707 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:37:24.273875 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:37:24.273796 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:37:27.578823 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:37:27.578784 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:37:42.773394 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:37:42.773360 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:37:47.649973 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:37:47.649153 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:38:01.042129 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:01.042093 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:38:03.197608 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:03.197573 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:38:09.436553 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:09.436511 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:38:10.372977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.370671 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp"] Nov 02 18:38:10.424350 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.424255 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp"] Nov 02 18:38:10.424762 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.424684 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:10.432172 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.432142 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-3c449aaf-kserve-self-signed-certs\"" Nov 02 18:38:10.464010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.461820 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:38:10.523152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.523116 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-home\") pod \"e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:10.523487 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.523457 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-model-cache\") pod \"e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:10.523739 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.523718 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-dshm\") pod \"e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:10.523939 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.523909 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c12e1dd-4e0d-416a-a772-bd00f96f892b-tls-certs\") pod \"e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:10.524218 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.524190 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-tmp-dir\") pod \"e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:10.524378 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.524358 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c77wz\" (UniqueName: \"kubernetes.io/projected/1c12e1dd-4e0d-416a-a772-bd00f96f892b-kube-api-access-c77wz\") pod \"e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:10.627497 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.627452 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-home\") pod \"e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:10.636001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.628166 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-home\") pod \"e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:10.636306 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.636281 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-model-cache\") pod \"e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:10.640064 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.636752 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-model-cache\") pod \"e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:10.657867 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.657834 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-dshm\") pod \"e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:10.658151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.658122 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c12e1dd-4e0d-416a-a772-bd00f96f892b-tls-certs\") pod \"e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:10.658318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.658299 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-tmp-dir\") pod \"e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:10.658452 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.658435 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c77wz\" (UniqueName: \"kubernetes.io/projected/1c12e1dd-4e0d-416a-a772-bd00f96f892b-kube-api-access-c77wz\") pod \"e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:10.660335 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.660309 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-tmp-dir\") pod \"e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:10.698018 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.694678 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c77wz\" (UniqueName: \"kubernetes.io/projected/1c12e1dd-4e0d-416a-a772-bd00f96f892b-kube-api-access-c77wz\") pod \"e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:10.719181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.715519 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-dshm\") pod \"e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:10.807977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.807871 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:38:10.838977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:10.692127 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c12e1dd-4e0d-416a-a772-bd00f96f892b-tls-certs\") pod \"e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:11.124219 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:11.124133 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:11.728965 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:11.728790 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp"] Nov 02 18:38:11.783717 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:11.783637 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" event={"ID":"1c12e1dd-4e0d-416a-a772-bd00f96f892b","Type":"ContainerStarted","Data":"b39d678f9512e19476f4a294ffd93052e23df5246d294f65841d46866d0f981a"} Nov 02 18:38:12.355196 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:12.355140 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:38:12.792251 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:12.792209 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" event={"ID":"1c12e1dd-4e0d-416a-a772-bd00f96f892b","Type":"ContainerStarted","Data":"916b9986a79ac8508890de494c8db04d8fa1c2929c8faf15e514a31db1bb70af"} Nov 02 18:38:12.809560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:12.809509 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" podStartSLOduration=2.80949567 podStartE2EDuration="2.80949567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:38:12.808013947 +0000 UTC m=+3864.596519475" watchObservedRunningTime="2026-11-02 18:38:12.80949567 +0000 UTC m=+3864.598001197" Nov 02 18:38:13.125070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:13.124925 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:16.139020 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:16.138980 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:38:17.586509 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:38:17.586469 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:38:21.125001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:21.124965 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:21.137423 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:21.137397 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:21.864628 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:21.864595 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:38:47.586464 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:38:47.586384 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:38:50.065277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:50.065247 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:38:50.284114 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:50.284076 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:38:53.134399 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:38:53.134363 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:39:03.283963 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:03.283913 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:06.560544 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:06.560502 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj"] Nov 02 18:39:06.561100 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:06.560691 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" podUID="8cb1b264-9e1e-4838-a719-d962ad2f35a9" containerName="discovery" containerID="cri-o://a61b851a8ab4a5c371f9537f663914999bebd1cc2062ccb2f488a726309d48fe" gracePeriod=30 Nov 02 18:39:06.813628 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:06.813566 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 18:39:06.909573 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:06.909543 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-token\" (UniqueName: \"kubernetes.io/projected/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-token\") pod \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " Nov 02 18:39:06.909764 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:06.909582 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-csr-dns-cert\") pod \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " Nov 02 18:39:06.909764 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:06.909602 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cb1b264-9e1e-4838-a719-d962ad2f35a9-kube-api-access-tz6d5\" (UniqueName: \"kubernetes.io/projected/8cb1b264-9e1e-4838-a719-d962ad2f35a9-kube-api-access-tz6d5\") pod \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " Nov 02 18:39:06.909764 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:06.909657 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cb1b264-9e1e-4838-a719-d962ad2f35a9-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cb1b264-9e1e-4838-a719-d962ad2f35a9-local-certs\") pod \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " Nov 02 18:39:06.909764 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:06.909684 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-csr-ca-configmap\") pod \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " Nov 02 18:39:06.909764 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:06.909726 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-cacerts\" (UniqueName: \"kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-cacerts\") pod \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " Nov 02 18:39:06.910100 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:06.909787 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-kubeconfig\") pod \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\" (UID: \"8cb1b264-9e1e-4838-a719-d962ad2f35a9\") " Nov 02 18:39:06.910226 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:06.910190 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-csr-ca-configmap" pod "8cb1b264-9e1e-4838-a719-d962ad2f35a9" (UID: "8cb1b264-9e1e-4838-a719-d962ad2f35a9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:39:06.911988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:06.911921 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb1b264-9e1e-4838-a719-d962ad2f35a9-local-certs" pod "8cb1b264-9e1e-4838-a719-d962ad2f35a9" (UID: "8cb1b264-9e1e-4838-a719-d962ad2f35a9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:39:06.912124 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:06.912002 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-csr-dns-cert" pod "8cb1b264-9e1e-4838-a719-d962ad2f35a9" (UID: "8cb1b264-9e1e-4838-a719-d962ad2f35a9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:39:06.912385 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:06.912358 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb1b264-9e1e-4838-a719-d962ad2f35a9-kube-api-access-tz6d5" pod "8cb1b264-9e1e-4838-a719-d962ad2f35a9" (UID: "8cb1b264-9e1e-4838-a719-d962ad2f35a9"). InnerVolumeSpecName "kube-api-access-tz6d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:39:06.912505 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:06.912485 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-token" pod "8cb1b264-9e1e-4838-a719-d962ad2f35a9" (UID: "8cb1b264-9e1e-4838-a719-d962ad2f35a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:39:06.912605 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:06.912503 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-cacerts" pod "8cb1b264-9e1e-4838-a719-d962ad2f35a9" (UID: "8cb1b264-9e1e-4838-a719-d962ad2f35a9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:39:06.912605 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:06.912547 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-kubeconfig" pod "8cb1b264-9e1e-4838-a719-d962ad2f35a9" (UID: "8cb1b264-9e1e-4838-a719-d962ad2f35a9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:39:07.011108 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:07.011075 5549 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-cacerts\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:07.011108 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:07.011107 5549 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-kubeconfig\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:07.011307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:07.011119 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:07.011307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:07.011129 5549 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-csr-dns-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:07.011307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:07.011140 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz6d5\" (UniqueName: \"kubernetes.io/projected/8cb1b264-9e1e-4838-a719-d962ad2f35a9-kube-api-access-tz6d5\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:07.011307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:07.011151 5549 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cb1b264-9e1e-4838-a719-d962ad2f35a9-local-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:07.011307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:07.011160 5549 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8cb1b264-9e1e-4838-a719-d962ad2f35a9-istio-csr-ca-configmap\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:07.095163 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:07.095074 5549 generic.go:356] "Generic (PLEG): container finished" podID="8cb1b264-9e1e-4838-a719-d962ad2f35a9" containerID="a61b851a8ab4a5c371f9537f663914999bebd1cc2062ccb2f488a726309d48fe" exitCode=0 Nov 02 18:39:07.095163 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:07.095132 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" event={"ID":"8cb1b264-9e1e-4838-a719-d962ad2f35a9","Type":"ContainerDied","Data":"a61b851a8ab4a5c371f9537f663914999bebd1cc2062ccb2f488a726309d48fe"} Nov 02 18:39:07.095163 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:07.095147 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" Nov 02 18:39:07.095463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:07.095171 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj" event={"ID":"8cb1b264-9e1e-4838-a719-d962ad2f35a9","Type":"ContainerDied","Data":"c0ba8e45760af4fea54d727b5fd48a83a497b9303f3c2fd62c9bf5fb295132f1"} Nov 02 18:39:07.095463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:07.095190 5549 scope.go:122] "RemoveContainer" containerID="a61b851a8ab4a5c371f9537f663914999bebd1cc2062ccb2f488a726309d48fe" Nov 02 18:39:07.110336 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:07.110318 5549 scope.go:122] "RemoveContainer" containerID="a61b851a8ab4a5c371f9537f663914999bebd1cc2062ccb2f488a726309d48fe" Nov 02 18:39:07.110598 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:39:07.110574 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61b851a8ab4a5c371f9537f663914999bebd1cc2062ccb2f488a726309d48fe\": container with ID starting with a61b851a8ab4a5c371f9537f663914999bebd1cc2062ccb2f488a726309d48fe not found: ID does not exist" containerID="a61b851a8ab4a5c371f9537f663914999bebd1cc2062ccb2f488a726309d48fe" Nov 02 18:39:07.110670 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:07.110613 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61b851a8ab4a5c371f9537f663914999bebd1cc2062ccb2f488a726309d48fe"} err="failed to get container status \"a61b851a8ab4a5c371f9537f663914999bebd1cc2062ccb2f488a726309d48fe\": rpc error: code = NotFound desc = could not find container \"a61b851a8ab4a5c371f9537f663914999bebd1cc2062ccb2f488a726309d48fe\": container with ID starting with a61b851a8ab4a5c371f9537f663914999bebd1cc2062ccb2f488a726309d48fe not found: ID does not exist" Nov 02 18:39:07.116722 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:07.116696 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj"] Nov 02 18:39:07.120929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:07.120905 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tlkmj"] Nov 02 18:39:08.318595 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:08.318561 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cb1b264-9e1e-4838-a719-d962ad2f35a9" path="/var/lib/kubelet/pods/8cb1b264-9e1e-4838-a719-d962ad2f35a9/volumes" Nov 02 18:39:11.775217 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:11.775177 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:39:12.519446 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:12.519407 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp"] Nov 02 18:39:12.542310 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:12.542281 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:12.676723 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:12.676690 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:15.941816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:15.941778 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:39:17.726310 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:39:17.726258 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:39:27.539982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.539890 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" podUID="1c12e1dd-4e0d-416a-a772-bd00f96f892b" containerName="main" containerID="cri-o://916b9986a79ac8508890de494c8db04d8fa1c2929c8faf15e514a31db1bb70af" gracePeriod=45 Nov 02 18:39:27.792366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.792301 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:39:27.895588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.895551 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-home\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-home\") pod \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " Nov 02 18:39:27.895790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.895597 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c12e1dd-4e0d-416a-a772-bd00f96f892b-kube-api-access-c77wz\" (UniqueName: \"kubernetes.io/projected/1c12e1dd-4e0d-416a-a772-bd00f96f892b-kube-api-access-c77wz\") pod \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " Nov 02 18:39:27.895790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.895640 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c12e1dd-4e0d-416a-a772-bd00f96f892b-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c12e1dd-4e0d-416a-a772-bd00f96f892b-tls-certs\") pod \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " Nov 02 18:39:27.895790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.895661 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-tmp-dir\") pod \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " Nov 02 18:39:27.895790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.895692 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-dshm\") pod \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " Nov 02 18:39:27.895790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.895777 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-model-cache\") pod \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\" (UID: \"1c12e1dd-4e0d-416a-a772-bd00f96f892b\") " Nov 02 18:39:27.896146 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.895898 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-home" pod "1c12e1dd-4e0d-416a-a772-bd00f96f892b" (UID: "1c12e1dd-4e0d-416a-a772-bd00f96f892b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:39:27.896146 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.896034 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-tmp-dir" pod "1c12e1dd-4e0d-416a-a772-bd00f96f892b" (UID: "1c12e1dd-4e0d-416a-a772-bd00f96f892b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:39:27.896146 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.896108 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:27.896146 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.896121 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-model-cache" pod "1c12e1dd-4e0d-416a-a772-bd00f96f892b" (UID: "1c12e1dd-4e0d-416a-a772-bd00f96f892b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:39:27.898002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.897982 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c12e1dd-4e0d-416a-a772-bd00f96f892b-tls-certs" pod "1c12e1dd-4e0d-416a-a772-bd00f96f892b" (UID: "1c12e1dd-4e0d-416a-a772-bd00f96f892b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:39:27.898076 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.897995 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-dshm" pod "1c12e1dd-4e0d-416a-a772-bd00f96f892b" (UID: "1c12e1dd-4e0d-416a-a772-bd00f96f892b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:39:27.898122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.898091 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c12e1dd-4e0d-416a-a772-bd00f96f892b-kube-api-access-c77wz" pod "1c12e1dd-4e0d-416a-a772-bd00f96f892b" (UID: "1c12e1dd-4e0d-416a-a772-bd00f96f892b"). InnerVolumeSpecName "kube-api-access-c77wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:39:27.996896 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.996865 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c77wz\" (UniqueName: \"kubernetes.io/projected/1c12e1dd-4e0d-416a-a772-bd00f96f892b-kube-api-access-c77wz\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:27.996896 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.996892 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c12e1dd-4e0d-416a-a772-bd00f96f892b-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:27.996896 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.996905 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:27.997153 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.996914 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:27.997153 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:27.996923 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c12e1dd-4e0d-416a-a772-bd00f96f892b-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:28.025138 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.025106 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:28.307577 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.307543 5549 generic.go:356] "Generic (PLEG): container finished" podID="1c12e1dd-4e0d-416a-a772-bd00f96f892b" containerID="916b9986a79ac8508890de494c8db04d8fa1c2929c8faf15e514a31db1bb70af" exitCode=0 Nov 02 18:39:28.307577 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.307583 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" Nov 02 18:39:28.307577 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.307604 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" event={"ID":"1c12e1dd-4e0d-416a-a772-bd00f96f892b","Type":"ContainerDied","Data":"916b9986a79ac8508890de494c8db04d8fa1c2929c8faf15e514a31db1bb70af"} Nov 02 18:39:28.307922 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.307639 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp" event={"ID":"1c12e1dd-4e0d-416a-a772-bd00f96f892b","Type":"ContainerDied","Data":"b39d678f9512e19476f4a294ffd93052e23df5246d294f65841d46866d0f981a"} Nov 02 18:39:28.307922 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.307661 5549 scope.go:122] "RemoveContainer" containerID="916b9986a79ac8508890de494c8db04d8fa1c2929c8faf15e514a31db1bb70af" Nov 02 18:39:28.317212 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.317196 5549 scope.go:122] "RemoveContainer" containerID="916b9986a79ac8508890de494c8db04d8fa1c2929c8faf15e514a31db1bb70af" Nov 02 18:39:28.317532 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:39:28.317464 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916b9986a79ac8508890de494c8db04d8fa1c2929c8faf15e514a31db1bb70af\": container with ID starting with 916b9986a79ac8508890de494c8db04d8fa1c2929c8faf15e514a31db1bb70af not found: ID does not exist" containerID="916b9986a79ac8508890de494c8db04d8fa1c2929c8faf15e514a31db1bb70af" Nov 02 18:39:28.317532 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.317514 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916b9986a79ac8508890de494c8db04d8fa1c2929c8faf15e514a31db1bb70af"} err="failed to get container status \"916b9986a79ac8508890de494c8db04d8fa1c2929c8faf15e514a31db1bb70af\": rpc error: code = NotFound desc = could not find container \"916b9986a79ac8508890de494c8db04d8fa1c2929c8faf15e514a31db1bb70af\": container with ID starting with 916b9986a79ac8508890de494c8db04d8fa1c2929c8faf15e514a31db1bb70af not found: ID does not exist" Nov 02 18:39:28.340876 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.340838 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp"] Nov 02 18:39:28.342232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.342205 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb"] Nov 02 18:39:28.342640 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.342624 5549 cpu_manager.go:414] "containerMap: removing container" podUID="8cb1b264-9e1e-4838-a719-d962ad2f35a9" containerName="discovery" Nov 02 18:39:28.342728 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.342640 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb1b264-9e1e-4838-a719-d962ad2f35a9" containerName="discovery" Nov 02 18:39:28.342728 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.342656 5549 cpu_manager.go:414] "containerMap: removing container" podUID="1c12e1dd-4e0d-416a-a772-bd00f96f892b" containerName="main" Nov 02 18:39:28.342728 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.342661 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c12e1dd-4e0d-416a-a772-bd00f96f892b" containerName="main" Nov 02 18:39:28.342911 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.342737 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb1b264-9e1e-4838-a719-d962ad2f35a9" containerName="discovery" Nov 02 18:39:28.342911 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.342750 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c12e1dd-4e0d-416a-a772-bd00f96f892b" containerName="main" Nov 02 18:39:28.347114 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.347096 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-3c449aaf-kserve-b5dcbd876-v2lqp"] Nov 02 18:39:28.347198 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.347173 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.356296 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.356263 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-c3df6f-openshift-default-dockercfg-4v754\"" Nov 02 18:39:28.356870 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.356827 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb"] Nov 02 18:39:28.400394 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.400361 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-token\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.400394 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.400399 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-envoy\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.400596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.400423 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-credential-socket\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.400596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.400488 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-data\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.400596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.400584 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcm8w\" (UniqueName: \"kubernetes.io/projected/59120275-fc6b-49f6-b250-5eedcf17fc5d-kube-api-access-bcm8w\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.400728 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.400612 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-workload-socket\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.400728 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.400642 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59120275-fc6b-49f6-b250-5eedcf17fc5d-istiod-ca-cert\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.400813 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.400761 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-podinfo\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.400813 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.400794 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-workload-certs\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.427022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.426983 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:28.501317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.501280 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-data\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.501496 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.501349 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcm8w\" (UniqueName: \"kubernetes.io/projected/59120275-fc6b-49f6-b250-5eedcf17fc5d-kube-api-access-bcm8w\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.501496 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.501387 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-workload-socket\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.501496 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.501429 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59120275-fc6b-49f6-b250-5eedcf17fc5d-istiod-ca-cert\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.501496 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.501490 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-podinfo\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.501722 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.501532 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-workload-certs\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.501722 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.501691 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-token\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.501839 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.501728 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-envoy\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.501839 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.501760 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-credential-socket\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.501839 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.501820 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-workload-socket\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.502062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.501759 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-data\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.502062 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.502017 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-credential-socket\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.502176 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.502104 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-workload-certs\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.502325 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.502307 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59120275-fc6b-49f6-b250-5eedcf17fc5d-istiod-ca-cert\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.503914 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.503897 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-envoy\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.504060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.504041 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-podinfo\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.509232 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.509210 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-token\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.509334 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.509237 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcm8w\" (UniqueName: \"kubernetes.io/projected/59120275-fc6b-49f6-b250-5eedcf17fc5d-kube-api-access-bcm8w\") pod \"e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.564445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.564357 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:28.662252 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.662218 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:28.943559 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.943482 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb"] Nov 02 18:39:28.947225 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:39:28.947186 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59120275_fc6b_49f6_b250_5eedcf17fc5d.slice/crio-7df3d756fe152fdc3659ee44445e1d424498726d80efcb4afa5a86859163d5dd WatchSource:0}: Error finding container 7df3d756fe152fdc3659ee44445e1d424498726d80efcb4afa5a86859163d5dd: Status 404 returned error can't find the container with id 7df3d756fe152fdc3659ee44445e1d424498726d80efcb4afa5a86859163d5dd Nov 02 18:39:28.949459 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.949426 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:39:28.949533 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.949490 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:39:28.949533 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:28.949520 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:39:29.313664 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:29.313629 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" event={"ID":"59120275-fc6b-49f6-b250-5eedcf17fc5d","Type":"ContainerStarted","Data":"9573e694980df8b35246c8052d24dda12bc0f141321ff358e4c1e3625a261ce6"} Nov 02 18:39:29.313894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:29.313680 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" event={"ID":"59120275-fc6b-49f6-b250-5eedcf17fc5d","Type":"ContainerStarted","Data":"7df3d756fe152fdc3659ee44445e1d424498726d80efcb4afa5a86859163d5dd"} Nov 02 18:39:29.662369 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:29.662277 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:30.319057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:30.319020 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c12e1dd-4e0d-416a-a772-bd00f96f892b" path="/var/lib/kubelet/pods/1c12e1dd-4e0d-416a-a772-bd00f96f892b/volumes" Nov 02 18:39:30.668451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:30.668377 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:30.690366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:30.690298 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" podStartSLOduration=2.690279221 podStartE2EDuration="2.690279221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:39:29.335196053 +0000 UTC m=+3941.123701581" watchObservedRunningTime="2026-11-02 18:39:30.690279221 +0000 UTC m=+3942.478784749" Nov 02 18:39:31.324361 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:31.324328 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:31.325258 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:31.325236 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:31.576266 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:31.576185 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:31.708057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:31.708018 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:32.656003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.655934 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d"] Nov 02 18:39:32.660159 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.660124 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" Nov 02 18:39:32.669070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.669037 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-c3df6f-serving-cert\"" Nov 02 18:39:32.672114 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.672086 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d"] Nov 02 18:39:32.745448 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.745417 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f0a9f98-54f8-4124-aa44-18dc43d9766f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d\" (UID: \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" Nov 02 18:39:32.745448 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.745453 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f0a9f98-54f8-4124-aa44-18dc43d9766f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d\" (UID: \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" Nov 02 18:39:32.745874 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.745474 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4bcq\" (UniqueName: \"kubernetes.io/projected/5f0a9f98-54f8-4124-aa44-18dc43d9766f-kube-api-access-x4bcq\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d\" (UID: \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" Nov 02 18:39:32.745874 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.745642 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f0a9f98-54f8-4124-aa44-18dc43d9766f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d\" (UID: \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" Nov 02 18:39:32.770409 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.770374 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:32.793070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.793038 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8"] Nov 02 18:39:32.796679 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.796661 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" Nov 02 18:39:32.804184 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.804161 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-c3df6f\"" Nov 02 18:39:32.804313 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.804281 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-88893f71-dockercfg-lknwl\"" Nov 02 18:39:32.806235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.806205 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8"] Nov 02 18:39:32.830347 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.830320 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj"] Nov 02 18:39:32.833867 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.833853 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" Nov 02 18:39:32.843593 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.843125 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj"] Nov 02 18:39:32.846767 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.846741 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khkmt\" (UniqueName: \"kubernetes.io/projected/6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3-kube-api-access-khkmt\") pod \"payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8\" (UID: \"6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" Nov 02 18:39:32.846873 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.846854 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8\" (UID: \"6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" Nov 02 18:39:32.846946 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.846905 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f0a9f98-54f8-4124-aa44-18dc43d9766f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d\" (UID: \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" Nov 02 18:39:32.847074 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.847054 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f0a9f98-54f8-4124-aa44-18dc43d9766f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d\" (UID: \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" Nov 02 18:39:32.847140 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.847088 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f0a9f98-54f8-4124-aa44-18dc43d9766f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d\" (UID: \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" Nov 02 18:39:32.847140 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.847135 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4bcq\" (UniqueName: \"kubernetes.io/projected/5f0a9f98-54f8-4124-aa44-18dc43d9766f-kube-api-access-x4bcq\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d\" (UID: \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" Nov 02 18:39:32.847799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.847774 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f0a9f98-54f8-4124-aa44-18dc43d9766f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d\" (UID: \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" Nov 02 18:39:32.849923 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.849897 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f0a9f98-54f8-4124-aa44-18dc43d9766f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d\" (UID: \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" Nov 02 18:39:32.850050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.849902 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f0a9f98-54f8-4124-aa44-18dc43d9766f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d\" (UID: \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" Nov 02 18:39:32.855911 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.855889 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4bcq\" (UniqueName: \"kubernetes.io/projected/5f0a9f98-54f8-4124-aa44-18dc43d9766f-kube-api-access-x4bcq\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d\" (UID: \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" Nov 02 18:39:32.905977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.904667 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:32.947718 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.947681 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5jtr\" (UniqueName: \"kubernetes.io/projected/9f038914-39b0-401a-a3be-69ec22982014-kube-api-access-z5jtr\") pod \"payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj\" (UID: \"9f038914-39b0-401a-a3be-69ec22982014\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" Nov 02 18:39:32.947890 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.947731 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkmt\" (UniqueName: \"kubernetes.io/projected/6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3-kube-api-access-khkmt\") pod \"payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8\" (UID: \"6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" Nov 02 18:39:32.947890 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.947822 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f038914-39b0-401a-a3be-69ec22982014-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj\" (UID: \"9f038914-39b0-401a-a3be-69ec22982014\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" Nov 02 18:39:32.947890 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.947877 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8\" (UID: \"6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" Nov 02 18:39:32.948607 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.948588 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8\" (UID: \"6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" Nov 02 18:39:32.957417 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.957387 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khkmt\" (UniqueName: \"kubernetes.io/projected/6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3-kube-api-access-khkmt\") pod \"payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8\" (UID: \"6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" Nov 02 18:39:32.977222 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:32.977196 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" Nov 02 18:39:33.049077 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:33.049033 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5jtr\" (UniqueName: \"kubernetes.io/projected/9f038914-39b0-401a-a3be-69ec22982014-kube-api-access-z5jtr\") pod \"payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj\" (UID: \"9f038914-39b0-401a-a3be-69ec22982014\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" Nov 02 18:39:33.049255 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:33.049122 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f038914-39b0-401a-a3be-69ec22982014-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj\" (UID: \"9f038914-39b0-401a-a3be-69ec22982014\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" Nov 02 18:39:33.050209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:33.050177 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f038914-39b0-401a-a3be-69ec22982014-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj\" (UID: \"9f038914-39b0-401a-a3be-69ec22982014\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" Nov 02 18:39:33.062633 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:33.061331 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5jtr\" (UniqueName: \"kubernetes.io/projected/9f038914-39b0-401a-a3be-69ec22982014-kube-api-access-z5jtr\") pod \"payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj\" (UID: \"9f038914-39b0-401a-a3be-69ec22982014\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" Nov 02 18:39:33.111879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:33.111841 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" Nov 02 18:39:33.148397 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:33.148361 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" Nov 02 18:39:33.256976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:33.256937 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d"] Nov 02 18:39:33.259025 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:39:33.258995 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0a9f98_54f8_4124_aa44_18dc43d9766f.slice/crio-6b09e4a6b583f81942b2f8544c489a4888cfcdf876c0c2f5ddd3bdb8f9e9bf82 WatchSource:0}: Error finding container 6b09e4a6b583f81942b2f8544c489a4888cfcdf876c0c2f5ddd3bdb8f9e9bf82: Status 404 returned error can't find the container with id 6b09e4a6b583f81942b2f8544c489a4888cfcdf876c0c2f5ddd3bdb8f9e9bf82 Nov 02 18:39:33.334475 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:33.334434 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" event={"ID":"5f0a9f98-54f8-4124-aa44-18dc43d9766f","Type":"ContainerStarted","Data":"6b09e4a6b583f81942b2f8544c489a4888cfcdf876c0c2f5ddd3bdb8f9e9bf82"} Nov 02 18:39:33.387185 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:33.387157 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8"] Nov 02 18:39:33.389246 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:39:33.389219 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e2c7ceb_4e60_4eb2_86c9_e402a0d2ffa3.slice/crio-3bd49dcc329cbb859f759057116b3a66887d909fc5680697b85c6ec292e8d5ff WatchSource:0}: Error finding container 3bd49dcc329cbb859f759057116b3a66887d909fc5680697b85c6ec292e8d5ff: Status 404 returned error can't find the container with id 3bd49dcc329cbb859f759057116b3a66887d909fc5680697b85c6ec292e8d5ff Nov 02 18:39:33.431539 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:33.431514 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj"] Nov 02 18:39:33.437041 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:39:33.437009 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f038914_39b0_401a_a3be_69ec22982014.slice/crio-fd752f58aa5a716af8ec94c15a44024ec32ce00c5f8df628e2ce2c53ccffd015 WatchSource:0}: Error finding container fd752f58aa5a716af8ec94c15a44024ec32ce00c5f8df628e2ce2c53ccffd015: Status 404 returned error can't find the container with id fd752f58aa5a716af8ec94c15a44024ec32ce00c5f8df628e2ce2c53ccffd015 Nov 02 18:39:33.613148 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:33.613114 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:34.174897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:34.174803 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:34.309092 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:34.309060 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:34.340600 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:34.340559 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" event={"ID":"9f038914-39b0-401a-a3be-69ec22982014","Type":"ContainerStarted","Data":"ccd908d4bc03265ad7bcdca9038a5a48cdaa9be4c377a243ff36ab37efca0565"} Nov 02 18:39:34.340784 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:34.340617 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" event={"ID":"9f038914-39b0-401a-a3be-69ec22982014","Type":"ContainerStarted","Data":"fd752f58aa5a716af8ec94c15a44024ec32ce00c5f8df628e2ce2c53ccffd015"} Nov 02 18:39:34.340784 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:34.340650 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" Nov 02 18:39:34.342214 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:34.342189 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" event={"ID":"5f0a9f98-54f8-4124-aa44-18dc43d9766f","Type":"ContainerStarted","Data":"773ea284525c0704caad2742bd974142bc53b5be39a557534bfe2c06b20e18f9"} Nov 02 18:39:34.342328 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:34.342248 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" Nov 02 18:39:34.343840 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:34.343815 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" event={"ID":"6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3","Type":"ContainerStarted","Data":"6bcf4c671bf17010022dbb669caef548b82e916a157c63ec46fc798abc741356"} Nov 02 18:39:34.343932 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:34.343859 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" event={"ID":"6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3","Type":"ContainerStarted","Data":"3bd49dcc329cbb859f759057116b3a66887d909fc5680697b85c6ec292e8d5ff"} Nov 02 18:39:34.344002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:34.343947 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" Nov 02 18:39:34.356608 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:34.356565 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" podStartSLOduration=2.356554738 podStartE2EDuration="2.356554738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:39:34.356049065 +0000 UTC m=+3946.144554590" watchObservedRunningTime="2026-11-02 18:39:34.356554738 +0000 UTC m=+3946.145060266" Nov 02 18:39:34.375733 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:34.375695 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" podStartSLOduration=1.928087463 podStartE2EDuration="2.375683214s" totalImagesPullingTime="447.595751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:39:34.374612761 +0000 UTC m=+3946.163118289" watchObservedRunningTime="2026-11-02 18:39:34.375683214 +0000 UTC m=+3946.164188738" Nov 02 18:39:34.391676 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:34.391641 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" podStartSLOduration=2.391631573 podStartE2EDuration="2.391631573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:39:34.390511998 +0000 UTC m=+3946.179017527" watchObservedRunningTime="2026-11-02 18:39:34.391631573 +0000 UTC m=+3946.180137100" Nov 02 18:39:34.963448 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:34.963407 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:35.051852 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:35.051816 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:39:35.756568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:35.756533 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:36.379976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:36.376724 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:37.086977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:37.079373 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:37.796978 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:37.784624 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:38.471073 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:38.471027 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:39.161087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:39.161055 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:39.869636 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:39.869600 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:40.353968 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:40.353931 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" Nov 02 18:39:40.575683 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:40.575644 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:41.274303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:41.274263 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:42.062146 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:42.062107 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:42.735086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:42.735049 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:43.448548 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:43.448498 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:44.073646 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:44.073606 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:44.746272 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:44.746138 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:45.352223 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:45.352195 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" Nov 02 18:39:45.352628 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:45.352261 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" Nov 02 18:39:45.402965 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:45.402920 5549 generic.go:356] "Generic (PLEG): container finished" podID="59120275-fc6b-49f6-b250-5eedcf17fc5d" containerID="9573e694980df8b35246c8052d24dda12bc0f141321ff358e4c1e3625a261ce6" exitCode=0 Nov 02 18:39:45.403165 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:45.402998 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" event={"ID":"59120275-fc6b-49f6-b250-5eedcf17fc5d","Type":"ContainerDied","Data":"9573e694980df8b35246c8052d24dda12bc0f141321ff358e4c1e3625a261ce6"} Nov 02 18:39:45.403416 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:45.403399 5549 scope.go:122] "RemoveContainer" containerID="9573e694980df8b35246c8052d24dda12bc0f141321ff358e4c1e3625a261ce6" Nov 02 18:39:45.404465 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:45.404339 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:45.404934 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:45.404901 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:39:45.405048 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:45.405023 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:39:45.405113 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:45.405086 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:39:46.277791 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:46.277730 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:46.324242 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:46.324209 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:46.409849 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:46.409808 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" event={"ID":"59120275-fc6b-49f6-b250-5eedcf17fc5d","Type":"ContainerStarted","Data":"1aecf0f84ab6bf45b1b88b2ae8d1805f34e2a69298d4f1eedb5554c0e52b6e34"} Nov 02 18:39:46.602145 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:46.597226 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:46.663415 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:46.663382 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:46.664185 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:46.664164 5549 patch_prober.go:29] interesting pod/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body= Nov 02 18:39:46.664248 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:46.664210 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" podUID="59120275-fc6b-49f6-b250-5eedcf17fc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" Nov 02 18:39:47.011098 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:47.011055 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:47.169516 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:47.169466 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:47.319071 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:47.318946 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:47.464198 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:47.464160 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:47.616604 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:39:47.616509 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:39:47.663213 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:47.663179 5549 patch_prober.go:29] interesting pod/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body= Nov 02 18:39:47.663412 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:47.663235 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" podUID="59120275-fc6b-49f6-b250-5eedcf17fc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" Nov 02 18:39:47.869988 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:47.867489 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj"] Nov 02 18:39:47.911467 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:47.911426 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:47.917863 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:47.917831 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj"] Nov 02 18:39:47.918113 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:47.917986 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:47.923459 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:47.923419 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c3df6f\"/\"default-dockercfg-czfhs\"" Nov 02 18:39:47.924338 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:47.924316 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c3df6f\"/\"simulator-single-main-c3df6f-kserve-self-signed-certs\"" Nov 02 18:39:47.999976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:47.998215 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c3df6f\"/\"openshift-service-ca.crt\"" Nov 02 18:39:47.999976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:47.998367 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c3df6f\"/\"kube-root-ca.crt\"" Nov 02 18:39:48.034307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.033608 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-tmp-dir\") pod \"simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.034307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.033672 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-dshm\") pod \"simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.034307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.033700 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-home\") pod \"simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.034307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.033755 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-model-cache\") pod \"simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.034307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.033854 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8qrv\" (UniqueName: \"kubernetes.io/projected/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-kube-api-access-p8qrv\") pod \"simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.034307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.033928 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-tls-certs\") pod \"simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.054790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.054755 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:48.135032 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.134931 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-model-cache\") pod \"simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.135201 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.135030 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8qrv\" (UniqueName: \"kubernetes.io/projected/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-kube-api-access-p8qrv\") pod \"simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.135201 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.135071 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-tls-certs\") pod \"simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.135201 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.135139 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-tmp-dir\") pod \"simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.135201 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.135167 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-dshm\") pod \"simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.135446 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.135266 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-model-cache\") pod \"simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.135446 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.135319 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-home\") pod \"simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.135569 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.135537 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-tmp-dir\") pod \"simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.135627 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.135605 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-home\") pod \"simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.137425 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.137396 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-dshm\") pod \"simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.137845 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.137820 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-tls-certs\") pod \"simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.144248 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.144225 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8qrv\" (UniqueName: \"kubernetes.io/projected/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-kube-api-access-p8qrv\") pod \"simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.238612 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.238575 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:48.538298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.538170 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj"] Nov 02 18:39:48.540837 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:39:48.540812 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0293e7b9_4dbc_4edd_bf82_c1ef9b79d2f5.slice/crio-6c616b3b2c7c2023db3cd3d169bfeff5131f14857cc8344c35620297f2ebb802 WatchSource:0}: Error finding container 6c616b3b2c7c2023db3cd3d169bfeff5131f14857cc8344c35620297f2ebb802: Status 404 returned error can't find the container with id 6c616b3b2c7c2023db3cd3d169bfeff5131f14857cc8344c35620297f2ebb802 Nov 02 18:39:48.663859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.663824 5549 patch_prober.go:29] interesting pod/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body= Nov 02 18:39:48.664033 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.663877 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" podUID="59120275-fc6b-49f6-b250-5eedcf17fc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" Nov 02 18:39:48.756891 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:48.756853 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:49.427355 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:49.427314 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" event={"ID":"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5","Type":"ContainerStarted","Data":"8cee4f299c2a995760c88aac5ae04ff0d48abfd4c6619f76098e8d239f34b3c8"} Nov 02 18:39:49.427562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:49.427367 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" event={"ID":"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5","Type":"ContainerStarted","Data":"6c616b3b2c7c2023db3cd3d169bfeff5131f14857cc8344c35620297f2ebb802"} Nov 02 18:39:49.434605 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:49.434571 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:39:49.445833 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:49.445784 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" podStartSLOduration=2.44577108 podStartE2EDuration="2.44577108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:39:49.444044692 +0000 UTC m=+3961.232550253" watchObservedRunningTime="2026-11-02 18:39:49.44577108 +0000 UTC m=+3961.234276607" Nov 02 18:39:49.504477 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:49.504437 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:49.663224 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:49.663190 5549 patch_prober.go:29] interesting pod/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body= Nov 02 18:39:49.663630 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:49.663251 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" podUID="59120275-fc6b-49f6-b250-5eedcf17fc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" Nov 02 18:39:50.238902 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:50.238863 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:50.367977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:50.367908 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:50.662716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:50.662636 5549 patch_prober.go:29] interesting pod/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body= Nov 02 18:39:50.662886 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:50.662695 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" podUID="59120275-fc6b-49f6-b250-5eedcf17fc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" Nov 02 18:39:51.094977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:51.085815 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:51.877568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:51.877523 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:51.927256 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:51.927226 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:51.927713 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:51.927692 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:51.928779 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:51.928759 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:39:52.650668 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:52.650625 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:53.332680 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:53.332637 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:53.954775 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:53.954730 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:54.566552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:54.566510 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:55.230519 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:55.230483 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:55.857230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:55.857176 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:56.492285 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:56.492246 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:57.247622 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:57.247579 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:57.855389 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:57.855346 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:58.238698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:58.238660 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:58.251470 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:58.251439 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:58.472456 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:58.472423 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:58.485871 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:58.485846 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:39:59.148635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:59.148598 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:39:59.770753 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:39:59.770715 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:00.458462 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:00.458368 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:01.056612 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:01.056555 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:01.662573 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:01.662535 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:02.345917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:02.345875 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:02.967507 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:02.967468 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:03.289826 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:03.289733 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:03.744557 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:03.744519 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:04.520097 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:04.520059 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:05.156629 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:05.156596 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:05.757165 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:05.757122 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:05.869616 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:05.869578 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:40:06.554835 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:06.554798 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:07.261509 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:07.261471 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:08.024085 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:08.024030 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:08.652605 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:08.652561 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:09.360609 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:09.360574 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:10.068084 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:10.068050 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:10.750811 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:10.750752 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:11.365356 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:11.365317 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:12.007127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:12.007092 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:12.654584 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:12.654535 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:13.265879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:13.265819 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:13.944849 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:13.944813 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:14.574286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:14.574256 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:15.164219 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:15.164183 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:15.742450 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:15.742416 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:16.370353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:16.370320 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:17.047817 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:17.047781 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:17.587413 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:40:17.587377 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:40:17.666417 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:17.666380 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:18.277681 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:18.277644 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:18.939781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:18.939746 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:19.557225 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:19.557190 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:20.342247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:20.342207 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:20.965073 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:20.965035 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:21.652763 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:21.652726 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:22.260525 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:22.260479 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:22.716676 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:22.716638 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:22.853349 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:22.853311 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:22.990019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:22.989913 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:23.457827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:23.457789 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:24.054872 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:24.054828 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:24.749683 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:24.749645 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:25.352180 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:25.352139 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:25.966495 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:25.966438 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:26.560731 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:26.560695 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:26.686241 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:26.686206 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:27.152278 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:27.152234 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:27.743202 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:27.743163 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:28.430446 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:28.430379 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:29.049307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:29.049267 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:29.331102 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:29.331019 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:29.625748 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:29.625650 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:40:29.758609 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:29.758577 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:29.888348 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:29.888272 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:30.463142 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:30.463107 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:31.169558 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:31.169522 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:31.815556 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:31.815516 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:32.557233 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:32.557196 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:33.150497 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:33.150454 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:33.761422 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:33.761372 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:34.243617 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:34.243578 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj"] Nov 02 18:40:34.355602 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:34.355561 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:34.485169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:34.485136 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:35.150994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:35.150940 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:35.599131 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:35.599086 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj"] Nov 02 18:40:35.611344 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:35.611312 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8"] Nov 02 18:40:35.730187 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:35.730149 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:35.857865 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:35.857741 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:36.348217 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:36.348181 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d"] Nov 02 18:40:36.348456 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:36.348428 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" podUID="5f0a9f98-54f8-4124-aa44-18dc43d9766f" containerName="maas-api" containerID="cri-o://773ea284525c0704caad2742bd974142bc53b5be39a557534bfe2c06b20e18f9" gracePeriod=30 Nov 02 18:40:36.403900 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:36.403863 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:36.538851 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:36.538814 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:37.599690 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.599666 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" Nov 02 18:40:37.690898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.690819 5549 generic.go:356] "Generic (PLEG): container finished" podID="5f0a9f98-54f8-4124-aa44-18dc43d9766f" containerID="773ea284525c0704caad2742bd974142bc53b5be39a557534bfe2c06b20e18f9" exitCode=0 Nov 02 18:40:37.690898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.690864 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" event={"ID":"5f0a9f98-54f8-4124-aa44-18dc43d9766f","Type":"ContainerDied","Data":"773ea284525c0704caad2742bd974142bc53b5be39a557534bfe2c06b20e18f9"} Nov 02 18:40:37.690898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.690869 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" Nov 02 18:40:37.691177 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.690941 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d" event={"ID":"5f0a9f98-54f8-4124-aa44-18dc43d9766f","Type":"ContainerDied","Data":"6b09e4a6b583f81942b2f8544c489a4888cfcdf876c0c2f5ddd3bdb8f9e9bf82"} Nov 02 18:40:37.691177 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.690990 5549 scope.go:122] "RemoveContainer" containerID="773ea284525c0704caad2742bd974142bc53b5be39a557534bfe2c06b20e18f9" Nov 02 18:40:37.699738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.699720 5549 scope.go:122] "RemoveContainer" containerID="773ea284525c0704caad2742bd974142bc53b5be39a557534bfe2c06b20e18f9" Nov 02 18:40:37.700040 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:40:37.700020 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"773ea284525c0704caad2742bd974142bc53b5be39a557534bfe2c06b20e18f9\": container with ID starting with 773ea284525c0704caad2742bd974142bc53b5be39a557534bfe2c06b20e18f9 not found: ID does not exist" containerID="773ea284525c0704caad2742bd974142bc53b5be39a557534bfe2c06b20e18f9" Nov 02 18:40:37.700104 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.700050 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"773ea284525c0704caad2742bd974142bc53b5be39a557534bfe2c06b20e18f9"} err="failed to get container status \"773ea284525c0704caad2742bd974142bc53b5be39a557534bfe2c06b20e18f9\": rpc error: code = NotFound desc = could not find container \"773ea284525c0704caad2742bd974142bc53b5be39a557534bfe2c06b20e18f9\": container with ID starting with 773ea284525c0704caad2742bd974142bc53b5be39a557534bfe2c06b20e18f9 not found: ID does not exist" Nov 02 18:40:37.763633 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.763591 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f0a9f98-54f8-4124-aa44-18dc43d9766f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f0a9f98-54f8-4124-aa44-18dc43d9766f-metrics-certs\") pod \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\" (UID: \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\") " Nov 02 18:40:37.763802 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.763692 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f0a9f98-54f8-4124-aa44-18dc43d9766f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f0a9f98-54f8-4124-aa44-18dc43d9766f-maas-api-tls\") pod \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\" (UID: \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\") " Nov 02 18:40:37.763802 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.763714 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0a9f98-54f8-4124-aa44-18dc43d9766f-kube-api-access-x4bcq\" (UniqueName: \"kubernetes.io/projected/5f0a9f98-54f8-4124-aa44-18dc43d9766f-kube-api-access-x4bcq\") pod \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\" (UID: \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\") " Nov 02 18:40:37.763915 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.763824 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f0a9f98-54f8-4124-aa44-18dc43d9766f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f0a9f98-54f8-4124-aa44-18dc43d9766f-openshift-service-ca\") pod \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\" (UID: \"5f0a9f98-54f8-4124-aa44-18dc43d9766f\") " Nov 02 18:40:37.764291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.764265 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0a9f98-54f8-4124-aa44-18dc43d9766f-openshift-service-ca" pod "5f0a9f98-54f8-4124-aa44-18dc43d9766f" (UID: "5f0a9f98-54f8-4124-aa44-18dc43d9766f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:40:37.766182 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.766155 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f0a9f98-54f8-4124-aa44-18dc43d9766f-metrics-certs" pod "5f0a9f98-54f8-4124-aa44-18dc43d9766f" (UID: "5f0a9f98-54f8-4124-aa44-18dc43d9766f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:40:37.766295 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.766177 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f0a9f98-54f8-4124-aa44-18dc43d9766f-maas-api-tls" pod "5f0a9f98-54f8-4124-aa44-18dc43d9766f" (UID: "5f0a9f98-54f8-4124-aa44-18dc43d9766f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:40:37.766295 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.766222 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0a9f98-54f8-4124-aa44-18dc43d9766f-kube-api-access-x4bcq" pod "5f0a9f98-54f8-4124-aa44-18dc43d9766f" (UID: "5f0a9f98-54f8-4124-aa44-18dc43d9766f"). InnerVolumeSpecName "kube-api-access-x4bcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:40:37.865089 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.865051 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f0a9f98-54f8-4124-aa44-18dc43d9766f-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:40:37.865089 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.865083 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f0a9f98-54f8-4124-aa44-18dc43d9766f-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:40:37.865089 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.865094 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f0a9f98-54f8-4124-aa44-18dc43d9766f-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:40:37.865331 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.865103 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4bcq\" (UniqueName: \"kubernetes.io/projected/5f0a9f98-54f8-4124-aa44-18dc43d9766f-kube-api-access-x4bcq\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:40:37.955972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:37.955923 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:40:38.015794 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:38.015705 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d"] Nov 02 18:40:38.017083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:38.017019 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-57g5d"] Nov 02 18:40:38.317676 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:38.317594 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f0a9f98-54f8-4124-aa44-18dc43d9766f" path="/var/lib/kubelet/pods/5f0a9f98-54f8-4124-aa44-18dc43d9766f/volumes" Nov 02 18:40:40.600425 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:40.600368 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" podUID="9f038914-39b0-401a-a3be-69ec22982014" containerName="payload-processing" containerID="cri-o://ccd908d4bc03265ad7bcdca9038a5a48cdaa9be4c377a243ff36ab37efca0565" gracePeriod=25 Nov 02 18:40:40.612057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:40.612022 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" podUID="6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3" containerName="payload-pre-processing" containerID="cri-o://6bcf4c671bf17010022dbb669caef548b82e916a157c63ec46fc798abc741356" gracePeriod=25 Nov 02 18:40:42.283323 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:42.283288 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:42.446933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:42.446851 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:45.350467 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:45.350428 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" podUID="9f038914-39b0-401a-a3be-69ec22982014" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:40:45.350874 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:45.350531 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" podUID="6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:40:47.586241 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:40:47.586202 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:40:49.263981 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.263929 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" podUID="0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5" containerName="main" containerID="cri-o://8cee4f299c2a995760c88aac5ae04ff0d48abfd4c6619f76098e8d239f34b3c8" gracePeriod=45 Nov 02 18:40:49.505465 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.505437 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:40:49.580989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.580881 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-home\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-home\") pod \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " Nov 02 18:40:49.581153 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.581017 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-dshm\") pod \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " Nov 02 18:40:49.581153 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.581073 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-tls-certs\") pod \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " Nov 02 18:40:49.581234 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.581192 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-tmp-dir\") pod \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " Nov 02 18:40:49.581234 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.581196 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-home" pod "0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5" (UID: "0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:40:49.581317 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.581261 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-model-cache\") pod \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " Nov 02 18:40:49.581372 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.581328 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-kube-api-access-p8qrv\" (UniqueName: \"kubernetes.io/projected/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-kube-api-access-p8qrv\") pod \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\" (UID: \"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5\") " Nov 02 18:40:49.581461 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.581435 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-tmp-dir" pod "0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5" (UID: "0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:40:49.581616 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.581590 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-model-cache" pod "0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5" (UID: "0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:40:49.581809 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.581781 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:40:49.581902 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.581810 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:40:49.581902 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.581825 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:40:49.583349 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.583317 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-dshm" pod "0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5" (UID: "0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:40:49.583349 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.583335 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-tls-certs" pod "0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5" (UID: "0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:40:49.583494 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.583407 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-kube-api-access-p8qrv" pod "0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5" (UID: "0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5"). InnerVolumeSpecName "kube-api-access-p8qrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:40:49.683247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.683215 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8qrv\" (UniqueName: \"kubernetes.io/projected/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-kube-api-access-p8qrv\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:40:49.683247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.683243 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:40:49.683247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.683252 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:40:49.752001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.751965 5549 generic.go:356] "Generic (PLEG): container finished" podID="0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5" containerID="8cee4f299c2a995760c88aac5ae04ff0d48abfd4c6619f76098e8d239f34b3c8" exitCode=0 Nov 02 18:40:49.752187 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.752018 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" Nov 02 18:40:49.752187 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.752047 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" event={"ID":"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5","Type":"ContainerDied","Data":"8cee4f299c2a995760c88aac5ae04ff0d48abfd4c6619f76098e8d239f34b3c8"} Nov 02 18:40:49.752187 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.752089 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" event={"ID":"0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5","Type":"ContainerDied","Data":"6c616b3b2c7c2023db3cd3d169bfeff5131f14857cc8344c35620297f2ebb802"} Nov 02 18:40:49.752187 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.752108 5549 scope.go:122] "RemoveContainer" containerID="8cee4f299c2a995760c88aac5ae04ff0d48abfd4c6619f76098e8d239f34b3c8" Nov 02 18:40:49.762347 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.762331 5549 scope.go:122] "RemoveContainer" containerID="8cee4f299c2a995760c88aac5ae04ff0d48abfd4c6619f76098e8d239f34b3c8" Nov 02 18:40:49.762624 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:40:49.762604 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cee4f299c2a995760c88aac5ae04ff0d48abfd4c6619f76098e8d239f34b3c8\": container with ID starting with 8cee4f299c2a995760c88aac5ae04ff0d48abfd4c6619f76098e8d239f34b3c8 not found: ID does not exist" containerID="8cee4f299c2a995760c88aac5ae04ff0d48abfd4c6619f76098e8d239f34b3c8" Nov 02 18:40:49.762694 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.762635 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cee4f299c2a995760c88aac5ae04ff0d48abfd4c6619f76098e8d239f34b3c8"} err="failed to get container status \"8cee4f299c2a995760c88aac5ae04ff0d48abfd4c6619f76098e8d239f34b3c8\": rpc error: code = NotFound desc = could not find container \"8cee4f299c2a995760c88aac5ae04ff0d48abfd4c6619f76098e8d239f34b3c8\": container with ID starting with 8cee4f299c2a995760c88aac5ae04ff0d48abfd4c6619f76098e8d239f34b3c8 not found: ID does not exist" Nov 02 18:40:49.774103 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.774079 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj"] Nov 02 18:40:49.777541 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:49.777521 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj"] Nov 02 18:40:50.318150 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:50.318110 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5" path="/var/lib/kubelet/pods/0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5/volumes" Nov 02 18:40:50.475935 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:50.475892 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-c3df6f/simulator-single-main-c3df6f-kserve-775dbc7b6b-l7kbj" podUID="0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.1.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:40:54.176616 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:54.176574 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:55.277288 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:55.277248 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:40:55.350061 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:55.350022 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" podUID="9f038914-39b0-401a-a3be-69ec22982014" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:40:55.350248 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:40:55.350038 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" podUID="6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:41:00.804724 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:00.804679 5549 generic.go:356] "Generic (PLEG): container finished" podID="6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3" containerID="6bcf4c671bf17010022dbb669caef548b82e916a157c63ec46fc798abc741356" exitCode=0 Nov 02 18:41:00.805220 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:00.804742 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" event={"ID":"6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3","Type":"ContainerDied","Data":"6bcf4c671bf17010022dbb669caef548b82e916a157c63ec46fc798abc741356"} Nov 02 18:41:00.806909 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:00.806879 5549 generic.go:356] "Generic (PLEG): container finished" podID="9f038914-39b0-401a-a3be-69ec22982014" containerID="ccd908d4bc03265ad7bcdca9038a5a48cdaa9be4c377a243ff36ab37efca0565" exitCode=0 Nov 02 18:41:00.807040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:00.806924 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" event={"ID":"9f038914-39b0-401a-a3be-69ec22982014","Type":"ContainerDied","Data":"ccd908d4bc03265ad7bcdca9038a5a48cdaa9be4c377a243ff36ab37efca0565"} Nov 02 18:41:00.867001 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:00.866886 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" Nov 02 18:41:00.892199 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:00.892168 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" Nov 02 18:41:00.997233 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:00.997189 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f038914-39b0-401a-a3be-69ec22982014-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f038914-39b0-401a-a3be-69ec22982014-plugins-config-volume\") pod \"9f038914-39b0-401a-a3be-69ec22982014\" (UID: \"9f038914-39b0-401a-a3be-69ec22982014\") " Nov 02 18:41:00.997454 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:00.997249 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3-plugins-config-volume\") pod \"6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3\" (UID: \"6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3\") " Nov 02 18:41:00.997454 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:00.997315 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3-kube-api-access-khkmt\" (UniqueName: \"kubernetes.io/projected/6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3-kube-api-access-khkmt\") pod \"6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3\" (UID: \"6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3\") " Nov 02 18:41:00.997454 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:00.997361 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f038914-39b0-401a-a3be-69ec22982014-kube-api-access-z5jtr\" (UniqueName: \"kubernetes.io/projected/9f038914-39b0-401a-a3be-69ec22982014-kube-api-access-z5jtr\") pod \"9f038914-39b0-401a-a3be-69ec22982014\" (UID: \"9f038914-39b0-401a-a3be-69ec22982014\") " Nov 02 18:41:00.997683 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:00.997650 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f038914-39b0-401a-a3be-69ec22982014-plugins-config-volume" pod "9f038914-39b0-401a-a3be-69ec22982014" (UID: "9f038914-39b0-401a-a3be-69ec22982014"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:41:00.997751 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:00.997657 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3-plugins-config-volume" pod "6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3" (UID: "6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:41:00.999606 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:00.999585 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3-kube-api-access-khkmt" pod "6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3" (UID: "6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3"). InnerVolumeSpecName "kube-api-access-khkmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:41:00.999677 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:00.999618 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f038914-39b0-401a-a3be-69ec22982014-kube-api-access-z5jtr" pod "9f038914-39b0-401a-a3be-69ec22982014" (UID: "9f038914-39b0-401a-a3be-69ec22982014"). InnerVolumeSpecName "kube-api-access-z5jtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:41:01.098326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:01.098286 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5jtr\" (UniqueName: \"kubernetes.io/projected/9f038914-39b0-401a-a3be-69ec22982014-kube-api-access-z5jtr\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:01.098326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:01.098321 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f038914-39b0-401a-a3be-69ec22982014-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:01.098326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:01.098332 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:01.098584 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:01.098343 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khkmt\" (UniqueName: \"kubernetes.io/projected/6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3-kube-api-access-khkmt\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:01.812938 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:01.812907 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" Nov 02 18:41:01.813467 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:01.812969 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj" event={"ID":"9f038914-39b0-401a-a3be-69ec22982014","Type":"ContainerDied","Data":"fd752f58aa5a716af8ec94c15a44024ec32ce00c5f8df628e2ce2c53ccffd015"} Nov 02 18:41:01.813467 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:01.813038 5549 scope.go:122] "RemoveContainer" containerID="ccd908d4bc03265ad7bcdca9038a5a48cdaa9be4c377a243ff36ab37efca0565" Nov 02 18:41:01.814393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:01.814376 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" Nov 02 18:41:01.814474 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:01.814368 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8" event={"ID":"6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3","Type":"ContainerDied","Data":"3bd49dcc329cbb859f759057116b3a66887d909fc5680697b85c6ec292e8d5ff"} Nov 02 18:41:01.823588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:01.823564 5549 scope.go:122] "RemoveContainer" containerID="6bcf4c671bf17010022dbb669caef548b82e916a157c63ec46fc798abc741356" Nov 02 18:41:01.836675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:01.836644 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj"] Nov 02 18:41:01.840604 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:01.840577 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cchqcj"] Nov 02 18:41:01.850287 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:01.850263 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8"] Nov 02 18:41:01.854698 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:01.854672 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65slqq8"] Nov 02 18:41:02.318376 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:02.318340 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3" path="/var/lib/kubelet/pods/6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3/volumes" Nov 02 18:41:02.318736 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:02.318723 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f038914-39b0-401a-a3be-69ec22982014" path="/var/lib/kubelet/pods/9f038914-39b0-401a-a3be-69ec22982014/volumes" Nov 02 18:41:03.331355 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:03.331318 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:06.607300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:06.607257 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:06.738609 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:06.738573 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:07.659440 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.659401 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc"] Nov 02 18:41:07.660199 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.660174 5549 cpu_manager.go:414] "containerMap: removing container" podUID="5f0a9f98-54f8-4124-aa44-18dc43d9766f" containerName="maas-api" Nov 02 18:41:07.660302 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.660204 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0a9f98-54f8-4124-aa44-18dc43d9766f" containerName="maas-api" Nov 02 18:41:07.660302 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.660229 5549 cpu_manager.go:414] "containerMap: removing container" podUID="0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5" containerName="main" Nov 02 18:41:07.660302 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.660237 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5" containerName="main" Nov 02 18:41:07.660302 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.660268 5549 cpu_manager.go:414] "containerMap: removing container" podUID="9f038914-39b0-401a-a3be-69ec22982014" containerName="payload-processing" Nov 02 18:41:07.660302 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.660277 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f038914-39b0-401a-a3be-69ec22982014" containerName="payload-processing" Nov 02 18:41:07.660302 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.660290 5549 cpu_manager.go:414] "containerMap: removing container" podUID="6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3" containerName="payload-pre-processing" Nov 02 18:41:07.660302 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.660299 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3" containerName="payload-pre-processing" Nov 02 18:41:07.660551 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.660423 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0a9f98-54f8-4124-aa44-18dc43d9766f" containerName="maas-api" Nov 02 18:41:07.660551 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.660440 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="0293e7b9-4dbc-4edd-bf82-c1ef9b79d2f5" containerName="main" Nov 02 18:41:07.660551 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.660455 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f038914-39b0-401a-a3be-69ec22982014" containerName="payload-processing" Nov 02 18:41:07.660551 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.660466 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2c7ceb-4e60-4eb2-86c9-e402a0d2ffa3" containerName="payload-pre-processing" Nov 02 18:41:07.665781 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.665757 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" Nov 02 18:41:07.670976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.670731 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-c3df6f-serving-cert\"" Nov 02 18:41:07.673658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.673633 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc"] Nov 02 18:41:07.757608 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.757575 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:07.765771 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.765735 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlxdh\" (UniqueName: \"kubernetes.io/projected/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-kube-api-access-qlxdh\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc\" (UID: \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" Nov 02 18:41:07.765977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.765890 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc\" (UID: \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" Nov 02 18:41:07.765977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.765944 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc\" (UID: \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" Nov 02 18:41:07.766087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.766025 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc\" (UID: \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" Nov 02 18:41:07.802890 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.802852 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78"] Nov 02 18:41:07.806527 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.806509 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl"] Nov 02 18:41:07.806642 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.806623 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78" Nov 02 18:41:07.809859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.809841 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl" Nov 02 18:41:07.814493 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.814460 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl"] Nov 02 18:41:07.816291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.816183 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-88893f71-dockercfg-flvf2\"" Nov 02 18:41:07.816291 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.816267 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-c3df6f\"" Nov 02 18:41:07.817483 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.817427 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78"] Nov 02 18:41:07.867502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.867468 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxdh\" (UniqueName: \"kubernetes.io/projected/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-kube-api-access-qlxdh\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc\" (UID: \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" Nov 02 18:41:07.867682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.867571 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc\" (UID: \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" Nov 02 18:41:07.867682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.867617 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc\" (UID: \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" Nov 02 18:41:07.867682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.867660 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc\" (UID: \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" Nov 02 18:41:07.868383 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.868354 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc\" (UID: \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" Nov 02 18:41:07.870173 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.870149 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc\" (UID: \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" Nov 02 18:41:07.870289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.870158 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc\" (UID: \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" Nov 02 18:41:07.875483 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.875457 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlxdh\" (UniqueName: \"kubernetes.io/projected/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-kube-api-access-qlxdh\") pod \"maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc\" (UID: \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" Nov 02 18:41:07.909556 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.909454 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:07.969097 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.969053 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn656\" (UniqueName: \"kubernetes.io/projected/998d4416-2115-40e0-b81f-6d14aaa8cbea-kube-api-access-jn656\") pod \"payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78\" (UID: \"998d4416-2115-40e0-b81f-6d14aaa8cbea\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78" Nov 02 18:41:07.969275 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.969218 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998d4416-2115-40e0-b81f-6d14aaa8cbea-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78\" (UID: \"998d4416-2115-40e0-b81f-6d14aaa8cbea\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78" Nov 02 18:41:07.969327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.969290 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hcsf\" (UniqueName: \"kubernetes.io/projected/797b3fdb-77db-4bd5-8371-5cfe85f7c1c6-kube-api-access-9hcsf\") pod \"payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl\" (UID: \"797b3fdb-77db-4bd5-8371-5cfe85f7c1c6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl" Nov 02 18:41:07.969396 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.969372 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797b3fdb-77db-4bd5-8371-5cfe85f7c1c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl\" (UID: \"797b3fdb-77db-4bd5-8371-5cfe85f7c1c6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl" Nov 02 18:41:07.982877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:07.982847 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" Nov 02 18:41:08.070880 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.070841 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998d4416-2115-40e0-b81f-6d14aaa8cbea-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78\" (UID: \"998d4416-2115-40e0-b81f-6d14aaa8cbea\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78" Nov 02 18:41:08.071109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.070911 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hcsf\" (UniqueName: \"kubernetes.io/projected/797b3fdb-77db-4bd5-8371-5cfe85f7c1c6-kube-api-access-9hcsf\") pod \"payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl\" (UID: \"797b3fdb-77db-4bd5-8371-5cfe85f7c1c6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl" Nov 02 18:41:08.071109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.070944 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797b3fdb-77db-4bd5-8371-5cfe85f7c1c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl\" (UID: \"797b3fdb-77db-4bd5-8371-5cfe85f7c1c6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl" Nov 02 18:41:08.071109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.070994 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn656\" (UniqueName: \"kubernetes.io/projected/998d4416-2115-40e0-b81f-6d14aaa8cbea-kube-api-access-jn656\") pod \"payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78\" (UID: \"998d4416-2115-40e0-b81f-6d14aaa8cbea\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78" Nov 02 18:41:08.072080 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.072051 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998d4416-2115-40e0-b81f-6d14aaa8cbea-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78\" (UID: \"998d4416-2115-40e0-b81f-6d14aaa8cbea\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78" Nov 02 18:41:08.072588 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.072565 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797b3fdb-77db-4bd5-8371-5cfe85f7c1c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl\" (UID: \"797b3fdb-77db-4bd5-8371-5cfe85f7c1c6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl" Nov 02 18:41:08.079491 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.079461 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hcsf\" (UniqueName: \"kubernetes.io/projected/797b3fdb-77db-4bd5-8371-5cfe85f7c1c6-kube-api-access-9hcsf\") pod \"payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl\" (UID: \"797b3fdb-77db-4bd5-8371-5cfe85f7c1c6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl" Nov 02 18:41:08.079741 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.079720 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn656\" (UniqueName: \"kubernetes.io/projected/998d4416-2115-40e0-b81f-6d14aaa8cbea-kube-api-access-jn656\") pod \"payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78\" (UID: \"998d4416-2115-40e0-b81f-6d14aaa8cbea\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78" Nov 02 18:41:08.122651 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.122612 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78" Nov 02 18:41:08.131339 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.131148 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl" Nov 02 18:41:08.262547 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.262501 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc"] Nov 02 18:41:08.264373 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:41:08.264336 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b90fc8_37f6_48fa_b3f0_0787b75f0f41.slice/crio-0e38e744f70f1c7b0dc4721d227d7864d2c15b3a414dc2ce8f538dbab603d58d WatchSource:0}: Error finding container 0e38e744f70f1c7b0dc4721d227d7864d2c15b3a414dc2ce8f538dbab603d58d: Status 404 returned error can't find the container with id 0e38e744f70f1c7b0dc4721d227d7864d2c15b3a414dc2ce8f538dbab603d58d Nov 02 18:41:08.265601 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.265581 5549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:41:08.403725 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.403691 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78"] Nov 02 18:41:08.405673 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:41:08.405643 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998d4416_2115_40e0_b81f_6d14aaa8cbea.slice/crio-9fc1ac0ab9c3f6452d919036be30e029c706a36f8664fcc3039d450b5344a72b WatchSource:0}: Error finding container 9fc1ac0ab9c3f6452d919036be30e029c706a36f8664fcc3039d450b5344a72b: Status 404 returned error can't find the container with id 9fc1ac0ab9c3f6452d919036be30e029c706a36f8664fcc3039d450b5344a72b Nov 02 18:41:08.414439 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.414416 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl"] Nov 02 18:41:08.416892 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:41:08.416865 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797b3fdb_77db_4bd5_8371_5cfe85f7c1c6.slice/crio-bf3a6cb785feea904ebb49a205927c68a562a4beaafe1511b293e7da97e14b23 WatchSource:0}: Error finding container bf3a6cb785feea904ebb49a205927c68a562a4beaafe1511b293e7da97e14b23: Status 404 returned error can't find the container with id bf3a6cb785feea904ebb49a205927c68a562a4beaafe1511b293e7da97e14b23 Nov 02 18:41:08.546626 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.545716 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:08.849102 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.849059 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78" event={"ID":"998d4416-2115-40e0-b81f-6d14aaa8cbea","Type":"ContainerStarted","Data":"09552fef72ba8a319d677402fe0ce0fe28ef6b64535bcddf552cf22748fe5e5f"} Nov 02 18:41:08.849535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.849194 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78" Nov 02 18:41:08.849535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.849229 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78" event={"ID":"998d4416-2115-40e0-b81f-6d14aaa8cbea","Type":"ContainerStarted","Data":"9fc1ac0ab9c3f6452d919036be30e029c706a36f8664fcc3039d450b5344a72b"} Nov 02 18:41:08.850695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.850669 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl" event={"ID":"797b3fdb-77db-4bd5-8371-5cfe85f7c1c6","Type":"ContainerStarted","Data":"4c4e7415d521c81cb5fe2d44b5879444c94a75c28e80f8a3cf2c970cbb9d760f"} Nov 02 18:41:08.850770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.850725 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl" Nov 02 18:41:08.850770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.850750 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl" event={"ID":"797b3fdb-77db-4bd5-8371-5cfe85f7c1c6","Type":"ContainerStarted","Data":"bf3a6cb785feea904ebb49a205927c68a562a4beaafe1511b293e7da97e14b23"} Nov 02 18:41:08.852284 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.852257 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" event={"ID":"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41","Type":"ContainerStarted","Data":"f2997703aaf83ae56d0b33091a3c99399aa67ccf1ce6b0888d2c00e06b713e6e"} Nov 02 18:41:08.852404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.852301 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" event={"ID":"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41","Type":"ContainerStarted","Data":"0e38e744f70f1c7b0dc4721d227d7864d2c15b3a414dc2ce8f538dbab603d58d"} Nov 02 18:41:08.852404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.852387 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" Nov 02 18:41:08.864823 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.864780 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78" podStartSLOduration=1.864769648 podStartE2EDuration="1.864769648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:41:08.8634795 +0000 UTC m=+4040.651985029" watchObservedRunningTime="2026-11-02 18:41:08.864769648 +0000 UTC m=+4040.653275175" Nov 02 18:41:08.881865 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.881826 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl" podStartSLOduration=1.8818159730000001 podStartE2EDuration="1.881815973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:41:08.87984727 +0000 UTC m=+4040.668352822" watchObservedRunningTime="2026-11-02 18:41:08.881815973 +0000 UTC m=+4040.670321500" Nov 02 18:41:08.895827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:08.895790 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" podStartSLOduration=1.435810733 podStartE2EDuration="1.895778384s" totalImagesPullingTime="459.967651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:41:08.894718109 +0000 UTC m=+4040.683223636" watchObservedRunningTime="2026-11-02 18:41:08.895778384 +0000 UTC m=+4040.684283909" Nov 02 18:41:09.230590 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:09.230556 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:09.669718 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:09.669635 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:41:09.862300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:09.862266 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:10.650894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:10.650853 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:10.781662 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:10.781625 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:11.720861 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:11.720826 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:11.944831 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:11.944789 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:12.257345 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:12.257305 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:12.527053 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:12.526942 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:41:13.033651 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:13.033609 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:13.895099 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:13.895066 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:14.643281 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:14.643245 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:14.863066 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:14.863038 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" Nov 02 18:41:15.257578 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:15.257534 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:15.859074 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:15.859030 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:16.513208 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:16.513175 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:16.650247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:16.650210 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:17.159097 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:17.159039 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:17.588267 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:41:17.588225 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:41:17.725186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:17.725139 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:17.857593 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:17.857491 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:18.551078 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:18.551032 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:19.254805 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:19.254762 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:19.850889 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:19.850852 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:19.861556 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:19.861527 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78" Nov 02 18:41:19.863806 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:19.863779 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl" Nov 02 18:41:20.453174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:20.453143 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:21.173862 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:21.173830 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:21.763406 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:21.763368 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:22.466771 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:22.466738 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:23.056424 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:23.056387 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:23.660113 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:23.660063 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:24.271929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:24.271888 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:24.938293 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:24.938261 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:25.664963 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:25.664913 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:25.807403 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:25.807365 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78"] Nov 02 18:41:25.810227 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:25.810195 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl"] Nov 02 18:41:25.919640 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:25.919563 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:26.052074 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:26.052017 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:26.332158 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:26.332101 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:26.851306 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:26.851262 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc"] Nov 02 18:41:26.851564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:26.851536 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" podUID="e6b90fc8-37f6-48fa-b3f0-0787b75f0f41" containerName="maas-api" containerID="cri-o://f2997703aaf83ae56d0b33091a3c99399aa67ccf1ce6b0888d2c00e06b713e6e" gracePeriod=30 Nov 02 18:41:26.948359 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:26.948329 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:27.108647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.108526 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:27.114167 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.114135 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" Nov 02 18:41:27.164824 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.164788 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-kube-api-access-qlxdh\" (UniqueName: \"kubernetes.io/projected/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-kube-api-access-qlxdh\") pod \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\" (UID: \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\") " Nov 02 18:41:27.164824 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.164826 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-openshift-service-ca\") pod \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\" (UID: \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\") " Nov 02 18:41:27.165137 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.164914 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-metrics-certs\") pod \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\" (UID: \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\") " Nov 02 18:41:27.165137 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.165122 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-maas-api-tls\") pod \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\" (UID: \"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41\") " Nov 02 18:41:27.165309 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.165286 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-openshift-service-ca" pod "e6b90fc8-37f6-48fa-b3f0-0787b75f0f41" (UID: "e6b90fc8-37f6-48fa-b3f0-0787b75f0f41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:41:27.165511 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.165488 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:27.167050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.167021 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-kube-api-access-qlxdh" pod "e6b90fc8-37f6-48fa-b3f0-0787b75f0f41" (UID: "e6b90fc8-37f6-48fa-b3f0-0787b75f0f41"). InnerVolumeSpecName "kube-api-access-qlxdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:41:27.167168 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.167065 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-metrics-certs" pod "e6b90fc8-37f6-48fa-b3f0-0787b75f0f41" (UID: "e6b90fc8-37f6-48fa-b3f0-0787b75f0f41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:41:27.167415 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.167395 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-maas-api-tls" pod "e6b90fc8-37f6-48fa-b3f0-0787b75f0f41" (UID: "e6b90fc8-37f6-48fa-b3f0-0787b75f0f41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:41:27.260607 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.260567 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:27.266925 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.266893 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:27.267004 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.266925 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:27.267004 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.266936 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlxdh\" (UniqueName: \"kubernetes.io/projected/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41-kube-api-access-qlxdh\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:27.439419 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.439308 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb"] Nov 02 18:41:27.439643 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.439565 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" podUID="59120275-fc6b-49f6-b250-5eedcf17fc5d" containerName="istio-proxy" containerID="cri-o://1aecf0f84ab6bf45b1b88b2ae8d1805f34e2a69298d4f1eedb5554c0e52b6e34" gracePeriod=30 Nov 02 18:41:27.520375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.520339 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:27.955222 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.955182 5549 generic.go:356] "Generic (PLEG): container finished" podID="e6b90fc8-37f6-48fa-b3f0-0787b75f0f41" containerID="f2997703aaf83ae56d0b33091a3c99399aa67ccf1ce6b0888d2c00e06b713e6e" exitCode=0 Nov 02 18:41:27.955429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.955224 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" Nov 02 18:41:27.955429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.955263 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" event={"ID":"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41","Type":"ContainerDied","Data":"f2997703aaf83ae56d0b33091a3c99399aa67ccf1ce6b0888d2c00e06b713e6e"} Nov 02 18:41:27.955429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.955321 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc" event={"ID":"e6b90fc8-37f6-48fa-b3f0-0787b75f0f41","Type":"ContainerDied","Data":"0e38e744f70f1c7b0dc4721d227d7864d2c15b3a414dc2ce8f538dbab603d58d"} Nov 02 18:41:27.955429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.955345 5549 scope.go:122] "RemoveContainer" containerID="f2997703aaf83ae56d0b33091a3c99399aa67ccf1ce6b0888d2c00e06b713e6e" Nov 02 18:41:27.964849 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.964831 5549 scope.go:122] "RemoveContainer" containerID="f2997703aaf83ae56d0b33091a3c99399aa67ccf1ce6b0888d2c00e06b713e6e" Nov 02 18:41:27.965153 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:41:27.965133 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2997703aaf83ae56d0b33091a3c99399aa67ccf1ce6b0888d2c00e06b713e6e\": container with ID starting with f2997703aaf83ae56d0b33091a3c99399aa67ccf1ce6b0888d2c00e06b713e6e not found: ID does not exist" containerID="f2997703aaf83ae56d0b33091a3c99399aa67ccf1ce6b0888d2c00e06b713e6e" Nov 02 18:41:27.965213 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.965162 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2997703aaf83ae56d0b33091a3c99399aa67ccf1ce6b0888d2c00e06b713e6e"} err="failed to get container status \"f2997703aaf83ae56d0b33091a3c99399aa67ccf1ce6b0888d2c00e06b713e6e\": rpc error: code = NotFound desc = could not find container \"f2997703aaf83ae56d0b33091a3c99399aa67ccf1ce6b0888d2c00e06b713e6e\": container with ID starting with f2997703aaf83ae56d0b33091a3c99399aa67ccf1ce6b0888d2c00e06b713e6e not found: ID does not exist" Nov 02 18:41:27.976630 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.976609 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc"] Nov 02 18:41:27.981655 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:27.981633 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3df6f-f47b5f77b-f55nc"] Nov 02 18:41:28.319262 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:28.319185 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6b90fc8-37f6-48fa-b3f0-0787b75f0f41" path="/var/lib/kubelet/pods/e6b90fc8-37f6-48fa-b3f0-0787b75f0f41/volumes" Nov 02 18:41:28.829998 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:28.829946 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:41:30.808381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:30.808284 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78" podUID="998d4416-2115-40e0-b81f-6d14aaa8cbea" containerName="payload-processing" containerID="cri-o://09552fef72ba8a319d677402fe0ce0fe28ef6b64535bcddf552cf22748fe5e5f" gracePeriod=25 Nov 02 18:41:30.810894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:30.810861 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl" podUID="797b3fdb-77db-4bd5-8371-5cfe85f7c1c6" containerName="payload-pre-processing" containerID="cri-o://4c4e7415d521c81cb5fe2d44b5879444c94a75c28e80f8a3cf2c970cbb9d760f" gracePeriod=25 Nov 02 18:41:30.972221 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:30.972184 5549 generic.go:356] "Generic (PLEG): container finished" podID="797b3fdb-77db-4bd5-8371-5cfe85f7c1c6" containerID="4c4e7415d521c81cb5fe2d44b5879444c94a75c28e80f8a3cf2c970cbb9d760f" exitCode=0 Nov 02 18:41:30.972415 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:30.972302 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl" event={"ID":"797b3fdb-77db-4bd5-8371-5cfe85f7c1c6","Type":"ContainerDied","Data":"4c4e7415d521c81cb5fe2d44b5879444c94a75c28e80f8a3cf2c970cbb9d760f"} Nov 02 18:41:30.973827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:30.973802 5549 generic.go:356] "Generic (PLEG): container finished" podID="998d4416-2115-40e0-b81f-6d14aaa8cbea" containerID="09552fef72ba8a319d677402fe0ce0fe28ef6b64535bcddf552cf22748fe5e5f" exitCode=0 Nov 02 18:41:30.973973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:30.973888 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78" event={"ID":"998d4416-2115-40e0-b81f-6d14aaa8cbea","Type":"ContainerDied","Data":"09552fef72ba8a319d677402fe0ce0fe28ef6b64535bcddf552cf22748fe5e5f"} Nov 02 18:41:31.095159 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.095127 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78" Nov 02 18:41:31.099122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.099094 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl" Nov 02 18:41:31.205916 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.205879 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998d4416-2115-40e0-b81f-6d14aaa8cbea-kube-api-access-jn656\" (UniqueName: \"kubernetes.io/projected/998d4416-2115-40e0-b81f-6d14aaa8cbea-kube-api-access-jn656\") pod \"998d4416-2115-40e0-b81f-6d14aaa8cbea\" (UID: \"998d4416-2115-40e0-b81f-6d14aaa8cbea\") " Nov 02 18:41:31.206102 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.205994 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/797b3fdb-77db-4bd5-8371-5cfe85f7c1c6-kube-api-access-9hcsf\" (UniqueName: \"kubernetes.io/projected/797b3fdb-77db-4bd5-8371-5cfe85f7c1c6-kube-api-access-9hcsf\") pod \"797b3fdb-77db-4bd5-8371-5cfe85f7c1c6\" (UID: \"797b3fdb-77db-4bd5-8371-5cfe85f7c1c6\") " Nov 02 18:41:31.206153 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.206098 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/797b3fdb-77db-4bd5-8371-5cfe85f7c1c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797b3fdb-77db-4bd5-8371-5cfe85f7c1c6-plugins-config-volume\") pod \"797b3fdb-77db-4bd5-8371-5cfe85f7c1c6\" (UID: \"797b3fdb-77db-4bd5-8371-5cfe85f7c1c6\") " Nov 02 18:41:31.206153 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.206136 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998d4416-2115-40e0-b81f-6d14aaa8cbea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998d4416-2115-40e0-b81f-6d14aaa8cbea-plugins-config-volume\") pod \"998d4416-2115-40e0-b81f-6d14aaa8cbea\" (UID: \"998d4416-2115-40e0-b81f-6d14aaa8cbea\") " Nov 02 18:41:31.206508 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.206476 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/797b3fdb-77db-4bd5-8371-5cfe85f7c1c6-plugins-config-volume" pod "797b3fdb-77db-4bd5-8371-5cfe85f7c1c6" (UID: "797b3fdb-77db-4bd5-8371-5cfe85f7c1c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:41:31.206647 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.206564 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998d4416-2115-40e0-b81f-6d14aaa8cbea-plugins-config-volume" pod "998d4416-2115-40e0-b81f-6d14aaa8cbea" (UID: "998d4416-2115-40e0-b81f-6d14aaa8cbea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:41:31.208154 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.208131 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797b3fdb-77db-4bd5-8371-5cfe85f7c1c6-kube-api-access-9hcsf" pod "797b3fdb-77db-4bd5-8371-5cfe85f7c1c6" (UID: "797b3fdb-77db-4bd5-8371-5cfe85f7c1c6"). InnerVolumeSpecName "kube-api-access-9hcsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:41:31.208221 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.208160 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998d4416-2115-40e0-b81f-6d14aaa8cbea-kube-api-access-jn656" pod "998d4416-2115-40e0-b81f-6d14aaa8cbea" (UID: "998d4416-2115-40e0-b81f-6d14aaa8cbea"). InnerVolumeSpecName "kube-api-access-jn656". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:41:31.241799 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.241767 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7"] Nov 02 18:41:31.242217 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.242202 5549 cpu_manager.go:414] "containerMap: removing container" podUID="998d4416-2115-40e0-b81f-6d14aaa8cbea" containerName="payload-processing" Nov 02 18:41:31.242265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.242219 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998d4416-2115-40e0-b81f-6d14aaa8cbea" containerName="payload-processing" Nov 02 18:41:31.242265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.242232 5549 cpu_manager.go:414] "containerMap: removing container" podUID="e6b90fc8-37f6-48fa-b3f0-0787b75f0f41" containerName="maas-api" Nov 02 18:41:31.242265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.242237 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6b90fc8-37f6-48fa-b3f0-0787b75f0f41" containerName="maas-api" Nov 02 18:41:31.242265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.242254 5549 cpu_manager.go:414] "containerMap: removing container" podUID="797b3fdb-77db-4bd5-8371-5cfe85f7c1c6" containerName="payload-pre-processing" Nov 02 18:41:31.242265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.242259 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="797b3fdb-77db-4bd5-8371-5cfe85f7c1c6" containerName="payload-pre-processing" Nov 02 18:41:31.242442 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.242332 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="998d4416-2115-40e0-b81f-6d14aaa8cbea" containerName="payload-processing" Nov 02 18:41:31.242442 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.242341 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6b90fc8-37f6-48fa-b3f0-0787b75f0f41" containerName="maas-api" Nov 02 18:41:31.242442 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.242349 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="797b3fdb-77db-4bd5-8371-5cfe85f7c1c6" containerName="payload-pre-processing" Nov 02 18:41:31.245862 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.245842 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.255784 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.255733 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c03c63e2-kserve-self-signed-certs\"" Nov 02 18:41:31.259358 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.259330 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7"] Nov 02 18:41:31.307941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.307904 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-tmp-dir\") pod \"e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.308144 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.307945 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-dshm\") pod \"e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.308144 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.308073 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5nbp\" (UniqueName: \"kubernetes.io/projected/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-kube-api-access-z5nbp\") pod \"e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.308144 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.308104 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-home\") pod \"e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.308271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.308206 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-tls-certs\") pod \"e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.308271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.308239 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-model-cache\") pod \"e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.308354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.308299 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn656\" (UniqueName: \"kubernetes.io/projected/998d4416-2115-40e0-b81f-6d14aaa8cbea-kube-api-access-jn656\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:31.308354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.308311 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hcsf\" (UniqueName: \"kubernetes.io/projected/797b3fdb-77db-4bd5-8371-5cfe85f7c1c6-kube-api-access-9hcsf\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:31.308354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.308322 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797b3fdb-77db-4bd5-8371-5cfe85f7c1c6-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:31.308354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.308331 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998d4416-2115-40e0-b81f-6d14aaa8cbea-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:31.343617 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.343541 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:31.409209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.409158 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-tls-certs\") pod \"e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.409407 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.409230 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-model-cache\") pod \"e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.409407 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.409346 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-tmp-dir\") pod \"e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.409407 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.409391 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-dshm\") pod \"e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.409653 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.409607 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5nbp\" (UniqueName: \"kubernetes.io/projected/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-kube-api-access-z5nbp\") pod \"e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.409757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.409655 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-model-cache\") pod \"e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.409757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.409656 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-home\") pod \"e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.409757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.409725 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-tmp-dir\") pod \"e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.409993 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.409946 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-home\") pod \"e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.411662 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.411642 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-dshm\") pod \"e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.411874 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.411859 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-tls-certs\") pod \"e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.417311 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.417287 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5nbp\" (UniqueName: \"kubernetes.io/projected/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-kube-api-access-z5nbp\") pod \"e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.564277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.564229 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:31.836169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.836135 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7"] Nov 02 18:41:31.839272 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:41:31.839237 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f9364f8_8f64_4f20_9391_b002fcf3f3c6.slice/crio-b4af4b2d4296b4980a55a24afeeadb1d7a17bdaadd554f76c4f666a96984429d WatchSource:0}: Error finding container b4af4b2d4296b4980a55a24afeeadb1d7a17bdaadd554f76c4f666a96984429d: Status 404 returned error can't find the container with id b4af4b2d4296b4980a55a24afeeadb1d7a17bdaadd554f76c4f666a96984429d Nov 02 18:41:31.980562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.980530 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl" Nov 02 18:41:31.980562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.980552 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl" event={"ID":"797b3fdb-77db-4bd5-8371-5cfe85f7c1c6","Type":"ContainerDied","Data":"bf3a6cb785feea904ebb49a205927c68a562a4beaafe1511b293e7da97e14b23"} Nov 02 18:41:31.980831 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.980612 5549 scope.go:122] "RemoveContainer" containerID="4c4e7415d521c81cb5fe2d44b5879444c94a75c28e80f8a3cf2c970cbb9d760f" Nov 02 18:41:31.981919 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.981899 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78" Nov 02 18:41:31.982064 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.981944 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78" event={"ID":"998d4416-2115-40e0-b81f-6d14aaa8cbea","Type":"ContainerDied","Data":"9fc1ac0ab9c3f6452d919036be30e029c706a36f8664fcc3039d450b5344a72b"} Nov 02 18:41:31.983571 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.983546 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" event={"ID":"7f9364f8-8f64-4f20-9391-b002fcf3f3c6","Type":"ContainerStarted","Data":"24af4c316c84d019fbdc1ad57a943d471513704ec945f17e929a27a1a9dd4650"} Nov 02 18:41:31.983709 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.983597 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" event={"ID":"7f9364f8-8f64-4f20-9391-b002fcf3f3c6","Type":"ContainerStarted","Data":"b4af4b2d4296b4980a55a24afeeadb1d7a17bdaadd554f76c4f666a96984429d"} Nov 02 18:41:31.990134 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:31.990093 5549 scope.go:122] "RemoveContainer" containerID="09552fef72ba8a319d677402fe0ce0fe28ef6b64535bcddf552cf22748fe5e5f" Nov 02 18:41:32.002230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.002186 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" podStartSLOduration=1.002173676 podStartE2EDuration="1.002173676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:41:32.001628812 +0000 UTC m=+4063.790134343" watchObservedRunningTime="2026-11-02 18:41:32.002173676 +0000 UTC m=+4063.790679236" Nov 02 18:41:32.017044 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.017015 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78"] Nov 02 18:41:32.019086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.019060 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c3df6f-69769776cgjn78"] Nov 02 18:41:32.029065 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.029036 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl"] Nov 02 18:41:32.036534 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.036509 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3df6f-85d65djbrl"] Nov 02 18:41:32.318276 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.318235 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="797b3fdb-77db-4bd5-8371-5cfe85f7c1c6" path="/var/lib/kubelet/pods/797b3fdb-77db-4bd5-8371-5cfe85f7c1c6/volumes" Nov 02 18:41:32.318652 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.318638 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998d4416-2115-40e0-b81f-6d14aaa8cbea" path="/var/lib/kubelet/pods/998d4416-2115-40e0-b81f-6d14aaa8cbea/volumes" Nov 02 18:41:32.564975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.564925 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:32.627680 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.627642 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:41:32.689855 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.689828 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:41:32.824447 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.824416 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-workload-socket\") pod \"59120275-fc6b-49f6-b250-5eedcf17fc5d\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " Nov 02 18:41:32.824657 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.824467 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-credential-socket\") pod \"59120275-fc6b-49f6-b250-5eedcf17fc5d\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " Nov 02 18:41:32.824657 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.824528 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59120275-fc6b-49f6-b250-5eedcf17fc5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59120275-fc6b-49f6-b250-5eedcf17fc5d-istiod-ca-cert\") pod \"59120275-fc6b-49f6-b250-5eedcf17fc5d\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " Nov 02 18:41:32.824657 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.824549 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-envoy\") pod \"59120275-fc6b-49f6-b250-5eedcf17fc5d\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " Nov 02 18:41:32.824657 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.824602 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59120275-fc6b-49f6-b250-5eedcf17fc5d-kube-api-access-bcm8w\" (UniqueName: \"kubernetes.io/projected/59120275-fc6b-49f6-b250-5eedcf17fc5d-kube-api-access-bcm8w\") pod \"59120275-fc6b-49f6-b250-5eedcf17fc5d\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " Nov 02 18:41:32.824657 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.824623 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-token\" (UniqueName: \"kubernetes.io/projected/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-token\") pod \"59120275-fc6b-49f6-b250-5eedcf17fc5d\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " Nov 02 18:41:32.824657 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.824656 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-data\") pod \"59120275-fc6b-49f6-b250-5eedcf17fc5d\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " Nov 02 18:41:32.825050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.824705 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-podinfo\") pod \"59120275-fc6b-49f6-b250-5eedcf17fc5d\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " Nov 02 18:41:32.825050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.824750 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-workload-socket" pod "59120275-fc6b-49f6-b250-5eedcf17fc5d" (UID: "59120275-fc6b-49f6-b250-5eedcf17fc5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:41:32.825050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.824773 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-workload-certs\") pod \"59120275-fc6b-49f6-b250-5eedcf17fc5d\" (UID: \"59120275-fc6b-49f6-b250-5eedcf17fc5d\") " Nov 02 18:41:32.825050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.824828 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-credential-socket" pod "59120275-fc6b-49f6-b250-5eedcf17fc5d" (UID: "59120275-fc6b-49f6-b250-5eedcf17fc5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:41:32.825050 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.825019 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-data" pod "59120275-fc6b-49f6-b250-5eedcf17fc5d" (UID: "59120275-fc6b-49f6-b250-5eedcf17fc5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:41:32.825354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.825123 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-workload-certs" pod "59120275-fc6b-49f6-b250-5eedcf17fc5d" (UID: "59120275-fc6b-49f6-b250-5eedcf17fc5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:41:32.825354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.825226 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:32.825354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.825244 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:32.825354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.825257 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:32.825354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.825270 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:32.825649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.825427 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59120275-fc6b-49f6-b250-5eedcf17fc5d-istiod-ca-cert" pod "59120275-fc6b-49f6-b250-5eedcf17fc5d" (UID: "59120275-fc6b-49f6-b250-5eedcf17fc5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:41:32.827079 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.827050 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-podinfo" pod "59120275-fc6b-49f6-b250-5eedcf17fc5d" (UID: "59120275-fc6b-49f6-b250-5eedcf17fc5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:41:32.827194 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.827090 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59120275-fc6b-49f6-b250-5eedcf17fc5d-kube-api-access-bcm8w" pod "59120275-fc6b-49f6-b250-5eedcf17fc5d" (UID: "59120275-fc6b-49f6-b250-5eedcf17fc5d"). InnerVolumeSpecName "kube-api-access-bcm8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:41:32.827292 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.827274 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-envoy" pod "59120275-fc6b-49f6-b250-5eedcf17fc5d" (UID: "59120275-fc6b-49f6-b250-5eedcf17fc5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:41:32.827383 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.827367 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-token" pod "59120275-fc6b-49f6-b250-5eedcf17fc5d" (UID: "59120275-fc6b-49f6-b250-5eedcf17fc5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:41:32.926376 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.926278 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59120275-fc6b-49f6-b250-5eedcf17fc5d-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:32.926376 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.926314 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:32.926376 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.926330 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcm8w\" (UniqueName: \"kubernetes.io/projected/59120275-fc6b-49f6-b250-5eedcf17fc5d-kube-api-access-bcm8w\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:32.926376 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.926344 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:32.926376 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.926353 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59120275-fc6b-49f6-b250-5eedcf17fc5d-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:41:32.992850 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.992808 5549 generic.go:356] "Generic (PLEG): container finished" podID="59120275-fc6b-49f6-b250-5eedcf17fc5d" containerID="1aecf0f84ab6bf45b1b88b2ae8d1805f34e2a69298d4f1eedb5554c0e52b6e34" exitCode=0 Nov 02 18:41:32.993083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.992871 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" Nov 02 18:41:32.993083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.992898 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" event={"ID":"59120275-fc6b-49f6-b250-5eedcf17fc5d","Type":"ContainerDied","Data":"1aecf0f84ab6bf45b1b88b2ae8d1805f34e2a69298d4f1eedb5554c0e52b6e34"} Nov 02 18:41:32.993083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.992976 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb" event={"ID":"59120275-fc6b-49f6-b250-5eedcf17fc5d","Type":"ContainerDied","Data":"7df3d756fe152fdc3659ee44445e1d424498726d80efcb4afa5a86859163d5dd"} Nov 02 18:41:32.993083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:32.993009 5549 scope.go:122] "RemoveContainer" containerID="1aecf0f84ab6bf45b1b88b2ae8d1805f34e2a69298d4f1eedb5554c0e52b6e34" Nov 02 18:41:33.002594 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:33.002574 5549 scope.go:122] "RemoveContainer" containerID="9573e694980df8b35246c8052d24dda12bc0f141321ff358e4c1e3625a261ce6" Nov 02 18:41:33.010444 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:33.010425 5549 scope.go:122] "RemoveContainer" containerID="1aecf0f84ab6bf45b1b88b2ae8d1805f34e2a69298d4f1eedb5554c0e52b6e34" Nov 02 18:41:33.010711 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:41:33.010695 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aecf0f84ab6bf45b1b88b2ae8d1805f34e2a69298d4f1eedb5554c0e52b6e34\": container with ID starting with 1aecf0f84ab6bf45b1b88b2ae8d1805f34e2a69298d4f1eedb5554c0e52b6e34 not found: ID does not exist" containerID="1aecf0f84ab6bf45b1b88b2ae8d1805f34e2a69298d4f1eedb5554c0e52b6e34" Nov 02 18:41:33.010757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:33.010720 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aecf0f84ab6bf45b1b88b2ae8d1805f34e2a69298d4f1eedb5554c0e52b6e34"} err="failed to get container status \"1aecf0f84ab6bf45b1b88b2ae8d1805f34e2a69298d4f1eedb5554c0e52b6e34\": rpc error: code = NotFound desc = could not find container \"1aecf0f84ab6bf45b1b88b2ae8d1805f34e2a69298d4f1eedb5554c0e52b6e34\": container with ID starting with 1aecf0f84ab6bf45b1b88b2ae8d1805f34e2a69298d4f1eedb5554c0e52b6e34 not found: ID does not exist" Nov 02 18:41:33.010757 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:33.010737 5549 scope.go:122] "RemoveContainer" containerID="9573e694980df8b35246c8052d24dda12bc0f141321ff358e4c1e3625a261ce6" Nov 02 18:41:33.010982 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:41:33.010944 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9573e694980df8b35246c8052d24dda12bc0f141321ff358e4c1e3625a261ce6\": container with ID starting with 9573e694980df8b35246c8052d24dda12bc0f141321ff358e4c1e3625a261ce6 not found: ID does not exist" containerID="9573e694980df8b35246c8052d24dda12bc0f141321ff358e4c1e3625a261ce6" Nov 02 18:41:33.011088 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:33.010985 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9573e694980df8b35246c8052d24dda12bc0f141321ff358e4c1e3625a261ce6"} err="failed to get container status \"9573e694980df8b35246c8052d24dda12bc0f141321ff358e4c1e3625a261ce6\": rpc error: code = NotFound desc = could not find container \"9573e694980df8b35246c8052d24dda12bc0f141321ff358e4c1e3625a261ce6\": container with ID starting with 9573e694980df8b35246c8052d24dda12bc0f141321ff358e4c1e3625a261ce6 not found: ID does not exist" Nov 02 18:41:33.014493 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:33.014472 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb"] Nov 02 18:41:33.022700 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:33.022676 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c3df6f-openshift-default-d595c96b9-ztstb"] Nov 02 18:41:34.369513 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:34.367780 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59120275-fc6b-49f6-b250-5eedcf17fc5d" path="/var/lib/kubelet/pods/59120275-fc6b-49f6-b250-5eedcf17fc5d/volumes" Nov 02 18:41:41.564784 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:41.564741 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:41.577518 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:41.577486 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:42.074215 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:42.074180 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:41:47.586570 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:41:47.586531 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:41:47.658865 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.658821 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq"] Nov 02 18:41:47.659361 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.659340 5549 cpu_manager.go:414] "containerMap: removing container" podUID="59120275-fc6b-49f6-b250-5eedcf17fc5d" containerName="istio-proxy" Nov 02 18:41:47.659468 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.659363 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59120275-fc6b-49f6-b250-5eedcf17fc5d" containerName="istio-proxy" Nov 02 18:41:47.659468 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.659376 5549 cpu_manager.go:414] "containerMap: removing container" podUID="59120275-fc6b-49f6-b250-5eedcf17fc5d" containerName="istio-proxy" Nov 02 18:41:47.659468 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.659383 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59120275-fc6b-49f6-b250-5eedcf17fc5d" containerName="istio-proxy" Nov 02 18:41:47.659648 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.659492 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="59120275-fc6b-49f6-b250-5eedcf17fc5d" containerName="istio-proxy" Nov 02 18:41:47.659648 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.659509 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="59120275-fc6b-49f6-b250-5eedcf17fc5d" containerName="istio-proxy" Nov 02 18:41:47.664443 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.664415 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.671128 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.671099 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq"] Nov 02 18:41:47.677210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.677180 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c03c63e2-kserve-self-signed-certs\"" Nov 02 18:41:47.773130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.773096 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:41:47.798586 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.798541 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-home\") pod \"e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.798808 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.798674 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-model-cache\") pod \"e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.798808 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.798771 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-dshm\") pod \"e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.798938 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.798819 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-tmp-dir\") pod \"e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.799040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.799016 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-tls-certs\") pod \"e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.799108 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.799054 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr7wq\" (UniqueName: \"kubernetes.io/projected/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-kube-api-access-kr7wq\") pod \"e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.899789 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.899694 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-model-cache\") pod \"e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.900002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.899843 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-dshm\") pod \"e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.900002 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.899893 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-tmp-dir\") pod \"e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.900148 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.900003 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-tls-certs\") pod \"e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.900148 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.900040 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr7wq\" (UniqueName: \"kubernetes.io/projected/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-kube-api-access-kr7wq\") pod \"e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.900148 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.900109 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-home\") pod \"e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.900148 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.900124 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-model-cache\") pod \"e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.900380 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.900355 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-tmp-dir\") pod \"e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.900472 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.900451 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-home\") pod \"e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.902186 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.902153 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-dshm\") pod \"e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.902545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.902525 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-tls-certs\") pod \"e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.908875 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.908849 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr7wq\" (UniqueName: \"kubernetes.io/projected/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-kube-api-access-kr7wq\") pod \"e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:47.982141 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:47.982105 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:48.253537 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:48.253509 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq"] Nov 02 18:41:48.256165 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:41:48.256132 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41eedf57_9ed7_43c8_abe3_7c5b8693d2df.slice/crio-6d7105719ec69a24f0f473b301093c356d021ee5a89bd0597cac458ea0e26271 WatchSource:0}: Error finding container 6d7105719ec69a24f0f473b301093c356d021ee5a89bd0597cac458ea0e26271: Status 404 returned error can't find the container with id 6d7105719ec69a24f0f473b301093c356d021ee5a89bd0597cac458ea0e26271 Nov 02 18:41:49.097429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:49.097391 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" event={"ID":"41eedf57-9ed7-43c8-abe3-7c5b8693d2df","Type":"ContainerStarted","Data":"c7da94f0c4cb97be5f89ab17ff0793e7eb15bc1cc6c8f28b3cbbdb4a3a96113e"} Nov 02 18:41:49.097854 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:49.097444 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" event={"ID":"41eedf57-9ed7-43c8-abe3-7c5b8693d2df","Type":"ContainerStarted","Data":"6d7105719ec69a24f0f473b301093c356d021ee5a89bd0597cac458ea0e26271"} Nov 02 18:41:49.117011 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:49.116926 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" podStartSLOduration=2.11691289 podStartE2EDuration="2.11691289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:41:49.115567622 +0000 UTC m=+4080.904073163" watchObservedRunningTime="2026-11-02 18:41:49.11691289 +0000 UTC m=+4080.905418412" Nov 02 18:41:49.160180 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:49.160146 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:41:49.982813 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:49.982753 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:57.983292 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:57.983250 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:57.995914 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:57.995886 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:41:58.202284 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:41:58.202252 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:42:03.391306 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:03.391261 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:42:05.553472 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:05.553418 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:42:08.937983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:08.931024 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-r2rbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:42:08.937983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:08.931088 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" podUID="6a7bea79-f2bb-489d-b861-28fa66d0ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:42:08.973977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:08.972021 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-kgx7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:42:08.973977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:08.972093 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" podUID="c208756b-40ff-4691-aad8-9008058e4291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:42:15.789857 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:15.789815 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7"] Nov 02 18:42:15.875596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:15.875560 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:42:17.007640 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:17.007599 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq"] Nov 02 18:42:17.098091 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:17.098049 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:42:17.268668 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:17.268575 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:42:17.587001 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:42:17.586894 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:42:22.311797 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:22.311756 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:42:29.426681 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:29.426644 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:42:30.817468 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:30.817426 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" podUID="7f9364f8-8f64-4f20-9391-b002fcf3f3c6" containerName="main" containerID="cri-o://24af4c316c84d019fbdc1ad57a943d471513704ec945f17e929a27a1a9dd4650" gracePeriod=45 Nov 02 18:42:31.071038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.067126 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" podUID="7f9364f8-8f64-4f20-9391-b002fcf3f3c6" containerName="main" probeResult="failure" output="Get \"https://10.133.1.82:8000/ready\": dial tcp 10.133.1.82:8000: connect: connection refused" Nov 02 18:42:31.609973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.609921 5549 generic.go:356] "Generic (PLEG): container finished" podID="7f9364f8-8f64-4f20-9391-b002fcf3f3c6" containerID="24af4c316c84d019fbdc1ad57a943d471513704ec945f17e929a27a1a9dd4650" exitCode=0 Nov 02 18:42:31.610332 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.610303 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" event={"ID":"7f9364f8-8f64-4f20-9391-b002fcf3f3c6","Type":"ContainerDied","Data":"24af4c316c84d019fbdc1ad57a943d471513704ec945f17e929a27a1a9dd4650"} Nov 02 18:42:31.707987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.707604 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:42:31.709413 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.708794 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-dshm\") pod \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " Nov 02 18:42:31.731111 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.731055 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-dshm" pod "7f9364f8-8f64-4f20-9391-b002fcf3f3c6" (UID: "7f9364f8-8f64-4f20-9391-b002fcf3f3c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:42:31.810234 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.810200 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-kube-api-access-z5nbp\" (UniqueName: \"kubernetes.io/projected/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-kube-api-access-z5nbp\") pod \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " Nov 02 18:42:31.810434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.810257 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-model-cache\") pod \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " Nov 02 18:42:31.810434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.810286 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-tmp-dir\") pod \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " Nov 02 18:42:31.810434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.810322 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-tls-certs\") pod \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " Nov 02 18:42:31.810434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.810431 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-home\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-home\") pod \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\" (UID: \"7f9364f8-8f64-4f20-9391-b002fcf3f3c6\") " Nov 02 18:42:31.810689 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.810609 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-tmp-dir" pod "7f9364f8-8f64-4f20-9391-b002fcf3f3c6" (UID: "7f9364f8-8f64-4f20-9391-b002fcf3f3c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:42:31.810689 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.810622 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-model-cache" pod "7f9364f8-8f64-4f20-9391-b002fcf3f3c6" (UID: "7f9364f8-8f64-4f20-9391-b002fcf3f3c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:42:31.810815 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.810781 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:42:31.810815 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.810803 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:42:31.810894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.810818 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:42:31.810894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.810820 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-home" pod "7f9364f8-8f64-4f20-9391-b002fcf3f3c6" (UID: "7f9364f8-8f64-4f20-9391-b002fcf3f3c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:42:31.812458 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.812433 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-kube-api-access-z5nbp" pod "7f9364f8-8f64-4f20-9391-b002fcf3f3c6" (UID: "7f9364f8-8f64-4f20-9391-b002fcf3f3c6"). InnerVolumeSpecName "kube-api-access-z5nbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:42:31.812985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.812937 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-tls-certs" pod "7f9364f8-8f64-4f20-9391-b002fcf3f3c6" (UID: "7f9364f8-8f64-4f20-9391-b002fcf3f3c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:42:31.912157 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.912127 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:42:31.912157 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.912154 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5nbp\" (UniqueName: \"kubernetes.io/projected/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-kube-api-access-z5nbp\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:42:31.912584 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:31.912165 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f9364f8-8f64-4f20-9391-b002fcf3f3c6-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:42:32.030413 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.030367 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" podUID="41eedf57-9ed7-43c8-abe3-7c5b8693d2df" containerName="main" containerID="cri-o://c7da94f0c4cb97be5f89ab17ff0793e7eb15bc1cc6c8f28b3cbbdb4a3a96113e" gracePeriod=45 Nov 02 18:42:32.192945 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.192620 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" podUID="41eedf57-9ed7-43c8-abe3-7c5b8693d2df" containerName="main" probeResult="failure" output="Get \"https://10.133.1.83:8000/ready\": dial tcp 10.133.1.83:8000: connect: connection refused" Nov 02 18:42:32.344331 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.344297 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:42:32.417440 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.417397 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-kube-api-access-kr7wq\" (UniqueName: \"kubernetes.io/projected/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-kube-api-access-kr7wq\") pod \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " Nov 02 18:42:32.417670 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.417496 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-model-cache\") pod \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " Nov 02 18:42:32.417670 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.417577 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-tls-certs\" (UniqueName: \"kubernetes.io/secret/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-tls-certs\") pod \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " Nov 02 18:42:32.417670 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.417611 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-dshm\") pod \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " Nov 02 18:42:32.417670 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.417665 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-home\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-home\") pod \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " Nov 02 18:42:32.418572 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.418539 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-home" pod "41eedf57-9ed7-43c8-abe3-7c5b8693d2df" (UID: "41eedf57-9ed7-43c8-abe3-7c5b8693d2df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:42:32.418802 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.418779 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-model-cache" pod "41eedf57-9ed7-43c8-abe3-7c5b8693d2df" (UID: "41eedf57-9ed7-43c8-abe3-7c5b8693d2df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:42:32.420966 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.420918 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-tls-certs" pod "41eedf57-9ed7-43c8-abe3-7c5b8693d2df" (UID: "41eedf57-9ed7-43c8-abe3-7c5b8693d2df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:42:32.422023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.421994 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-dshm" pod "41eedf57-9ed7-43c8-abe3-7c5b8693d2df" (UID: "41eedf57-9ed7-43c8-abe3-7c5b8693d2df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:42:32.422228 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.422204 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-kube-api-access-kr7wq" pod "41eedf57-9ed7-43c8-abe3-7c5b8693d2df" (UID: "41eedf57-9ed7-43c8-abe3-7c5b8693d2df"). InnerVolumeSpecName "kube-api-access-kr7wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:42:32.518976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.518918 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-tmp-dir\") pod \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\" (UID: \"41eedf57-9ed7-43c8-abe3-7c5b8693d2df\") " Nov 02 18:42:32.519320 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.519278 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-tmp-dir" pod "41eedf57-9ed7-43c8-abe3-7c5b8693d2df" (UID: "41eedf57-9ed7-43c8-abe3-7c5b8693d2df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:42:32.519415 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.519392 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:42:32.519490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.519416 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:42:32.519490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.519447 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:42:32.519490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.519462 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:42:32.519490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.519479 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:42:32.519728 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.519493 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr7wq\" (UniqueName: \"kubernetes.io/projected/41eedf57-9ed7-43c8-abe3-7c5b8693d2df-kube-api-access-kr7wq\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:42:32.618942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.618826 5549 generic.go:356] "Generic (PLEG): container finished" podID="41eedf57-9ed7-43c8-abe3-7c5b8693d2df" containerID="c7da94f0c4cb97be5f89ab17ff0793e7eb15bc1cc6c8f28b3cbbdb4a3a96113e" exitCode=0 Nov 02 18:42:32.618942 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.618891 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" Nov 02 18:42:32.619246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.619019 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" event={"ID":"41eedf57-9ed7-43c8-abe3-7c5b8693d2df","Type":"ContainerDied","Data":"c7da94f0c4cb97be5f89ab17ff0793e7eb15bc1cc6c8f28b3cbbdb4a3a96113e"} Nov 02 18:42:32.619246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.619058 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq" event={"ID":"41eedf57-9ed7-43c8-abe3-7c5b8693d2df","Type":"ContainerDied","Data":"6d7105719ec69a24f0f473b301093c356d021ee5a89bd0597cac458ea0e26271"} Nov 02 18:42:32.619246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.619082 5549 scope.go:122] "RemoveContainer" containerID="c7da94f0c4cb97be5f89ab17ff0793e7eb15bc1cc6c8f28b3cbbdb4a3a96113e" Nov 02 18:42:32.625174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.625133 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" event={"ID":"7f9364f8-8f64-4f20-9391-b002fcf3f3c6","Type":"ContainerDied","Data":"b4af4b2d4296b4980a55a24afeeadb1d7a17bdaadd554f76c4f666a96984429d"} Nov 02 18:42:32.642402 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.642363 5549 scope.go:122] "RemoveContainer" containerID="c7da94f0c4cb97be5f89ab17ff0793e7eb15bc1cc6c8f28b3cbbdb4a3a96113e" Nov 02 18:42:32.642814 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:42:32.642783 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7da94f0c4cb97be5f89ab17ff0793e7eb15bc1cc6c8f28b3cbbdb4a3a96113e\": container with ID starting with c7da94f0c4cb97be5f89ab17ff0793e7eb15bc1cc6c8f28b3cbbdb4a3a96113e not found: ID does not exist" containerID="c7da94f0c4cb97be5f89ab17ff0793e7eb15bc1cc6c8f28b3cbbdb4a3a96113e" Nov 02 18:42:32.642947 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.642828 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7da94f0c4cb97be5f89ab17ff0793e7eb15bc1cc6c8f28b3cbbdb4a3a96113e"} err="failed to get container status \"c7da94f0c4cb97be5f89ab17ff0793e7eb15bc1cc6c8f28b3cbbdb4a3a96113e\": rpc error: code = NotFound desc = could not find container \"c7da94f0c4cb97be5f89ab17ff0793e7eb15bc1cc6c8f28b3cbbdb4a3a96113e\": container with ID starting with c7da94f0c4cb97be5f89ab17ff0793e7eb15bc1cc6c8f28b3cbbdb4a3a96113e not found: ID does not exist" Nov 02 18:42:32.642947 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.642855 5549 scope.go:122] "RemoveContainer" containerID="24af4c316c84d019fbdc1ad57a943d471513704ec945f17e929a27a1a9dd4650" Nov 02 18:42:32.644294 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.644267 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7" Nov 02 18:42:32.668107 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.668071 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7"] Nov 02 18:42:32.672793 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.672767 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c03c63e2-kserve-7c45979757-gflr7"] Nov 02 18:42:32.684160 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.682850 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq"] Nov 02 18:42:32.685314 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:32.685286 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c03c63e2-kserve-6d5c95fc85-cgsmq"] Nov 02 18:42:33.722975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:33.722651 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:42:34.330720 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:34.330681 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41eedf57-9ed7-43c8-abe3-7c5b8693d2df" path="/var/lib/kubelet/pods/41eedf57-9ed7-43c8-abe3-7c5b8693d2df/volumes" Nov 02 18:42:34.331714 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:34.331688 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f9364f8-8f64-4f20-9391-b002fcf3f3c6" path="/var/lib/kubelet/pods/7f9364f8-8f64-4f20-9391-b002fcf3f3c6/volumes" Nov 02 18:42:46.579120 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:46.579083 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:42:47.585378 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:42:47.585338 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:42:48.915620 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:48.915585 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-99bb68447-w6bqc"] Nov 02 18:42:48.916188 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:48.916172 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7f9364f8-8f64-4f20-9391-b002fcf3f3c6" containerName="main" Nov 02 18:42:48.916250 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:48.916190 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f9364f8-8f64-4f20-9391-b002fcf3f3c6" containerName="main" Nov 02 18:42:48.916250 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:48.916214 5549 cpu_manager.go:414] "containerMap: removing container" podUID="41eedf57-9ed7-43c8-abe3-7c5b8693d2df" containerName="main" Nov 02 18:42:48.916250 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:48.916222 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41eedf57-9ed7-43c8-abe3-7c5b8693d2df" containerName="main" Nov 02 18:42:48.916372 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:48.916310 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f9364f8-8f64-4f20-9391-b002fcf3f3c6" containerName="main" Nov 02 18:42:48.916372 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:48.916325 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="41eedf57-9ed7-43c8-abe3-7c5b8693d2df" containerName="main" Nov 02 18:42:48.921091 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:48.921071 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-99bb68447-w6bqc" Nov 02 18:42:48.927560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:48.927535 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-99bb68447-w6bqc"] Nov 02 18:42:49.002221 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:49.002180 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p84n\" (UniqueName: \"kubernetes.io/projected/cc379d1f-5f1d-45c8-90f9-b3e3f88fc242-kube-api-access-6p84n\") pod \"payload-pre-processing-99bb68447-w6bqc\" (UID: \"cc379d1f-5f1d-45c8-90f9-b3e3f88fc242\") " pod="openshift-ingress/payload-pre-processing-99bb68447-w6bqc" Nov 02 18:42:49.002367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:49.002315 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc379d1f-5f1d-45c8-90f9-b3e3f88fc242-plugins-config-volume\") pod \"payload-pre-processing-99bb68447-w6bqc\" (UID: \"cc379d1f-5f1d-45c8-90f9-b3e3f88fc242\") " pod="openshift-ingress/payload-pre-processing-99bb68447-w6bqc" Nov 02 18:42:49.103726 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:49.103689 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p84n\" (UniqueName: \"kubernetes.io/projected/cc379d1f-5f1d-45c8-90f9-b3e3f88fc242-kube-api-access-6p84n\") pod \"payload-pre-processing-99bb68447-w6bqc\" (UID: \"cc379d1f-5f1d-45c8-90f9-b3e3f88fc242\") " pod="openshift-ingress/payload-pre-processing-99bb68447-w6bqc" Nov 02 18:42:49.103896 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:49.103802 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc379d1f-5f1d-45c8-90f9-b3e3f88fc242-plugins-config-volume\") pod \"payload-pre-processing-99bb68447-w6bqc\" (UID: \"cc379d1f-5f1d-45c8-90f9-b3e3f88fc242\") " pod="openshift-ingress/payload-pre-processing-99bb68447-w6bqc" Nov 02 18:42:49.104511 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:49.104493 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc379d1f-5f1d-45c8-90f9-b3e3f88fc242-plugins-config-volume\") pod \"payload-pre-processing-99bb68447-w6bqc\" (UID: \"cc379d1f-5f1d-45c8-90f9-b3e3f88fc242\") " pod="openshift-ingress/payload-pre-processing-99bb68447-w6bqc" Nov 02 18:42:49.113259 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:49.113225 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p84n\" (UniqueName: \"kubernetes.io/projected/cc379d1f-5f1d-45c8-90f9-b3e3f88fc242-kube-api-access-6p84n\") pod \"payload-pre-processing-99bb68447-w6bqc\" (UID: \"cc379d1f-5f1d-45c8-90f9-b3e3f88fc242\") " pod="openshift-ingress/payload-pre-processing-99bb68447-w6bqc" Nov 02 18:42:49.236428 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:49.236397 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-99bb68447-w6bqc" Nov 02 18:42:49.506065 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:49.506036 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-99bb68447-w6bqc"] Nov 02 18:42:49.508362 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:42:49.508318 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc379d1f_5f1d_45c8_90f9_b3e3f88fc242.slice/crio-68c46dbe048e5071b47e76628cdc402050ea3547332420a95266150f0e1aebc0 WatchSource:0}: Error finding container 68c46dbe048e5071b47e76628cdc402050ea3547332420a95266150f0e1aebc0: Status 404 returned error can't find the container with id 68c46dbe048e5071b47e76628cdc402050ea3547332420a95266150f0e1aebc0 Nov 02 18:42:49.730764 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:49.730722 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-99bb68447-w6bqc" event={"ID":"cc379d1f-5f1d-45c8-90f9-b3e3f88fc242","Type":"ContainerStarted","Data":"536cfb4de672c5783f9448c01918b98ab65b629bf54b9bd5124e780e5f1f82c3"} Nov 02 18:42:49.730965 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:49.730774 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-99bb68447-w6bqc" event={"ID":"cc379d1f-5f1d-45c8-90f9-b3e3f88fc242","Type":"ContainerStarted","Data":"68c46dbe048e5071b47e76628cdc402050ea3547332420a95266150f0e1aebc0"} Nov 02 18:42:49.730965 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:49.730827 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-99bb68447-w6bqc" Nov 02 18:42:49.748687 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:42:49.748638 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-99bb68447-w6bqc" podStartSLOduration=1.74862417 podStartE2EDuration="1.74862417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:42:49.747577933 +0000 UTC m=+4141.536083460" watchObservedRunningTime="2026-11-02 18:42:49.74862417 +0000 UTC m=+4141.537129700" Nov 02 18:43:00.738323 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:00.738229 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-99bb68447-w6bqc" Nov 02 18:43:00.783634 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:00.783590 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m"] Nov 02 18:43:02.504136 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:02.504097 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58b55f785c-rh8fz"] Nov 02 18:43:02.507606 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:02.507586 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58b55f785c-rh8fz" Nov 02 18:43:02.514473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:02.514449 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58b55f785c-rh8fz"] Nov 02 18:43:02.635209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:02.635165 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5pz6\" (UniqueName: \"kubernetes.io/projected/00fc32f0-057f-40d4-befe-156f8235e05f-kube-api-access-j5pz6\") pod \"payload-processing-58b55f785c-rh8fz\" (UID: \"00fc32f0-057f-40d4-befe-156f8235e05f\") " pod="openshift-ingress/payload-processing-58b55f785c-rh8fz" Nov 02 18:43:02.635395 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:02.635280 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fc32f0-057f-40d4-befe-156f8235e05f-plugins-config-volume\") pod \"payload-processing-58b55f785c-rh8fz\" (UID: \"00fc32f0-057f-40d4-befe-156f8235e05f\") " pod="openshift-ingress/payload-processing-58b55f785c-rh8fz" Nov 02 18:43:02.736687 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:02.736641 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5pz6\" (UniqueName: \"kubernetes.io/projected/00fc32f0-057f-40d4-befe-156f8235e05f-kube-api-access-j5pz6\") pod \"payload-processing-58b55f785c-rh8fz\" (UID: \"00fc32f0-057f-40d4-befe-156f8235e05f\") " pod="openshift-ingress/payload-processing-58b55f785c-rh8fz" Nov 02 18:43:02.736887 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:02.736707 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fc32f0-057f-40d4-befe-156f8235e05f-plugins-config-volume\") pod \"payload-processing-58b55f785c-rh8fz\" (UID: \"00fc32f0-057f-40d4-befe-156f8235e05f\") " pod="openshift-ingress/payload-processing-58b55f785c-rh8fz" Nov 02 18:43:02.737434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:02.737409 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fc32f0-057f-40d4-befe-156f8235e05f-plugins-config-volume\") pod \"payload-processing-58b55f785c-rh8fz\" (UID: \"00fc32f0-057f-40d4-befe-156f8235e05f\") " pod="openshift-ingress/payload-processing-58b55f785c-rh8fz" Nov 02 18:43:02.745194 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:02.745165 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5pz6\" (UniqueName: \"kubernetes.io/projected/00fc32f0-057f-40d4-befe-156f8235e05f-kube-api-access-j5pz6\") pod \"payload-processing-58b55f785c-rh8fz\" (UID: \"00fc32f0-057f-40d4-befe-156f8235e05f\") " pod="openshift-ingress/payload-processing-58b55f785c-rh8fz" Nov 02 18:43:02.822074 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:02.822005 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58b55f785c-rh8fz" Nov 02 18:43:03.094476 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:03.094441 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58b55f785c-rh8fz"] Nov 02 18:43:03.426527 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:03.426440 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:43:03.792872 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:03.792832 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58b55f785c-rh8fz" event={"ID":"00fc32f0-057f-40d4-befe-156f8235e05f","Type":"ContainerStarted","Data":"34dca6a15c51982af5554642ef2af36070be479af620699f9dbd29919286d3a4"} Nov 02 18:43:03.793299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:03.792888 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58b55f785c-rh8fz" event={"ID":"00fc32f0-057f-40d4-befe-156f8235e05f","Type":"ContainerStarted","Data":"8cbf67381d4be7d4cef425ba70764ff0d971ea572362628b815a9b9c05a4d341"} Nov 02 18:43:03.793299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:03.792989 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58b55f785c-rh8fz" Nov 02 18:43:03.809906 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:03.809856 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58b55f785c-rh8fz" podStartSLOduration=1.80984064 podStartE2EDuration="1.80984064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:43:03.80906589 +0000 UTC m=+4155.597571418" watchObservedRunningTime="2026-11-02 18:43:03.80984064 +0000 UTC m=+4155.598346170" Nov 02 18:43:05.784183 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:05.784131 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m" podUID="53e0ed62-81a5-4917-baef-0942cc4a7aee" containerName="payload-pre-processing" containerID="cri-o://75b1461b2e1d1bd992b437817f8a02e77740c8ab0c5aeec85e710c6eda10bfbf" gracePeriod=25 Nov 02 18:43:06.030018 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:06.029992 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m" Nov 02 18:43:06.172521 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:06.172369 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e0ed62-81a5-4917-baef-0942cc4a7aee-kube-api-access-d7kvv\" (UniqueName: \"kubernetes.io/projected/53e0ed62-81a5-4917-baef-0942cc4a7aee-kube-api-access-d7kvv\") pod \"53e0ed62-81a5-4917-baef-0942cc4a7aee\" (UID: \"53e0ed62-81a5-4917-baef-0942cc4a7aee\") " Nov 02 18:43:06.172717 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:06.172559 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e0ed62-81a5-4917-baef-0942cc4a7aee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e0ed62-81a5-4917-baef-0942cc4a7aee-plugins-config-volume\") pod \"53e0ed62-81a5-4917-baef-0942cc4a7aee\" (UID: \"53e0ed62-81a5-4917-baef-0942cc4a7aee\") " Nov 02 18:43:06.172907 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:06.172886 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e0ed62-81a5-4917-baef-0942cc4a7aee-plugins-config-volume" pod "53e0ed62-81a5-4917-baef-0942cc4a7aee" (UID: "53e0ed62-81a5-4917-baef-0942cc4a7aee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:43:06.174431 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:06.174408 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e0ed62-81a5-4917-baef-0942cc4a7aee-kube-api-access-d7kvv" pod "53e0ed62-81a5-4917-baef-0942cc4a7aee" (UID: "53e0ed62-81a5-4917-baef-0942cc4a7aee"). InnerVolumeSpecName "kube-api-access-d7kvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:43:06.273379 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:06.273325 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7kvv\" (UniqueName: \"kubernetes.io/projected/53e0ed62-81a5-4917-baef-0942cc4a7aee-kube-api-access-d7kvv\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:43:06.273379 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:06.273373 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e0ed62-81a5-4917-baef-0942cc4a7aee-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:43:06.807806 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:06.807768 5549 generic.go:356] "Generic (PLEG): container finished" podID="53e0ed62-81a5-4917-baef-0942cc4a7aee" containerID="75b1461b2e1d1bd992b437817f8a02e77740c8ab0c5aeec85e710c6eda10bfbf" exitCode=0 Nov 02 18:43:06.808286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:06.807814 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m" Nov 02 18:43:06.808286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:06.807849 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m" event={"ID":"53e0ed62-81a5-4917-baef-0942cc4a7aee","Type":"ContainerDied","Data":"75b1461b2e1d1bd992b437817f8a02e77740c8ab0c5aeec85e710c6eda10bfbf"} Nov 02 18:43:06.808286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:06.807902 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m" event={"ID":"53e0ed62-81a5-4917-baef-0942cc4a7aee","Type":"ContainerDied","Data":"1f62d99dda910f8d0162198ec4ff6dfed34b81ebc79da4acb5f16ba56c15e442"} Nov 02 18:43:06.808286 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:06.807923 5549 scope.go:122] "RemoveContainer" containerID="75b1461b2e1d1bd992b437817f8a02e77740c8ab0c5aeec85e710c6eda10bfbf" Nov 02 18:43:06.816927 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:06.816906 5549 scope.go:122] "RemoveContainer" containerID="75b1461b2e1d1bd992b437817f8a02e77740c8ab0c5aeec85e710c6eda10bfbf" Nov 02 18:43:06.817228 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:43:06.817208 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b1461b2e1d1bd992b437817f8a02e77740c8ab0c5aeec85e710c6eda10bfbf\": container with ID starting with 75b1461b2e1d1bd992b437817f8a02e77740c8ab0c5aeec85e710c6eda10bfbf not found: ID does not exist" containerID="75b1461b2e1d1bd992b437817f8a02e77740c8ab0c5aeec85e710c6eda10bfbf" Nov 02 18:43:06.817298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:06.817240 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b1461b2e1d1bd992b437817f8a02e77740c8ab0c5aeec85e710c6eda10bfbf"} err="failed to get container status \"75b1461b2e1d1bd992b437817f8a02e77740c8ab0c5aeec85e710c6eda10bfbf\": rpc error: code = NotFound desc = could not find container \"75b1461b2e1d1bd992b437817f8a02e77740c8ab0c5aeec85e710c6eda10bfbf\": container with ID starting with 75b1461b2e1d1bd992b437817f8a02e77740c8ab0c5aeec85e710c6eda10bfbf not found: ID does not exist" Nov 02 18:43:06.824526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:06.824498 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m"] Nov 02 18:43:06.830409 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:06.830382 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-nnt4m"] Nov 02 18:43:08.318217 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:08.318183 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53e0ed62-81a5-4917-baef-0942cc4a7aee" path="/var/lib/kubelet/pods/53e0ed62-81a5-4917-baef-0942cc4a7aee/volumes" Nov 02 18:43:14.800246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:14.800210 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58b55f785c-rh8fz" Nov 02 18:43:14.846335 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:14.846300 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-5hb2w"] Nov 02 18:43:16.672783 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:16.672739 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-fd758c546-br2db"] Nov 02 18:43:16.673257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:16.673224 5549 cpu_manager.go:414] "containerMap: removing container" podUID="53e0ed62-81a5-4917-baef-0942cc4a7aee" containerName="payload-pre-processing" Nov 02 18:43:16.673257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:16.673239 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e0ed62-81a5-4917-baef-0942cc4a7aee" containerName="payload-pre-processing" Nov 02 18:43:16.673336 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:16.673320 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e0ed62-81a5-4917-baef-0942cc4a7aee" containerName="payload-pre-processing" Nov 02 18:43:16.677904 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:16.677885 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-fd758c546-br2db" Nov 02 18:43:16.685972 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:16.685923 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-fd758c546-br2db"] Nov 02 18:43:16.842237 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:16.842193 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7d54786449-dvg2t"] Nov 02 18:43:16.845947 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:16.845930 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7d54786449-dvg2t" Nov 02 18:43:16.854646 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:16.854614 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7d54786449-dvg2t"] Nov 02 18:43:16.876669 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:16.876635 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/395c02e5-ad21-4b1b-a7ec-9bbd7dbfe204-plugins-config-volume\") pod \"payload-processing-fd758c546-br2db\" (UID: \"395c02e5-ad21-4b1b-a7ec-9bbd7dbfe204\") " pod="openshift-ingress/payload-processing-fd758c546-br2db" Nov 02 18:43:16.876823 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:16.876806 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r466j\" (UniqueName: \"kubernetes.io/projected/395c02e5-ad21-4b1b-a7ec-9bbd7dbfe204-kube-api-access-r466j\") pod \"payload-processing-fd758c546-br2db\" (UID: \"395c02e5-ad21-4b1b-a7ec-9bbd7dbfe204\") " pod="openshift-ingress/payload-processing-fd758c546-br2db" Nov 02 18:43:16.977746 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:16.977707 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/395c02e5-ad21-4b1b-a7ec-9bbd7dbfe204-plugins-config-volume\") pod \"payload-processing-fd758c546-br2db\" (UID: \"395c02e5-ad21-4b1b-a7ec-9bbd7dbfe204\") " pod="openshift-ingress/payload-processing-fd758c546-br2db" Nov 02 18:43:16.977946 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:16.977768 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1200f090-634e-446d-b876-b37375067c84-plugins-config-volume\") pod \"payload-pre-processing-7d54786449-dvg2t\" (UID: \"1200f090-634e-446d-b876-b37375067c84\") " pod="openshift-ingress/payload-pre-processing-7d54786449-dvg2t" Nov 02 18:43:16.977946 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:16.977829 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r9rh\" (UniqueName: \"kubernetes.io/projected/1200f090-634e-446d-b876-b37375067c84-kube-api-access-5r9rh\") pod \"payload-pre-processing-7d54786449-dvg2t\" (UID: \"1200f090-634e-446d-b876-b37375067c84\") " pod="openshift-ingress/payload-pre-processing-7d54786449-dvg2t" Nov 02 18:43:16.977946 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:16.977934 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r466j\" (UniqueName: \"kubernetes.io/projected/395c02e5-ad21-4b1b-a7ec-9bbd7dbfe204-kube-api-access-r466j\") pod \"payload-processing-fd758c546-br2db\" (UID: \"395c02e5-ad21-4b1b-a7ec-9bbd7dbfe204\") " pod="openshift-ingress/payload-processing-fd758c546-br2db" Nov 02 18:43:16.978449 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:16.978428 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/395c02e5-ad21-4b1b-a7ec-9bbd7dbfe204-plugins-config-volume\") pod \"payload-processing-fd758c546-br2db\" (UID: \"395c02e5-ad21-4b1b-a7ec-9bbd7dbfe204\") " pod="openshift-ingress/payload-processing-fd758c546-br2db" Nov 02 18:43:16.986856 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:16.986834 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r466j\" (UniqueName: \"kubernetes.io/projected/395c02e5-ad21-4b1b-a7ec-9bbd7dbfe204-kube-api-access-r466j\") pod \"payload-processing-fd758c546-br2db\" (UID: \"395c02e5-ad21-4b1b-a7ec-9bbd7dbfe204\") " pod="openshift-ingress/payload-processing-fd758c546-br2db" Nov 02 18:43:17.001680 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:17.001658 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-fd758c546-br2db" Nov 02 18:43:17.079995 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:17.079771 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1200f090-634e-446d-b876-b37375067c84-plugins-config-volume\") pod \"payload-pre-processing-7d54786449-dvg2t\" (UID: \"1200f090-634e-446d-b876-b37375067c84\") " pod="openshift-ingress/payload-pre-processing-7d54786449-dvg2t" Nov 02 18:43:17.079995 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:17.079829 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r9rh\" (UniqueName: \"kubernetes.io/projected/1200f090-634e-446d-b876-b37375067c84-kube-api-access-5r9rh\") pod \"payload-pre-processing-7d54786449-dvg2t\" (UID: \"1200f090-634e-446d-b876-b37375067c84\") " pod="openshift-ingress/payload-pre-processing-7d54786449-dvg2t" Nov 02 18:43:17.081358 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:17.081332 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1200f090-634e-446d-b876-b37375067c84-plugins-config-volume\") pod \"payload-pre-processing-7d54786449-dvg2t\" (UID: \"1200f090-634e-446d-b876-b37375067c84\") " pod="openshift-ingress/payload-pre-processing-7d54786449-dvg2t" Nov 02 18:43:17.090546 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:17.090520 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r9rh\" (UniqueName: \"kubernetes.io/projected/1200f090-634e-446d-b876-b37375067c84-kube-api-access-5r9rh\") pod \"payload-pre-processing-7d54786449-dvg2t\" (UID: \"1200f090-634e-446d-b876-b37375067c84\") " pod="openshift-ingress/payload-pre-processing-7d54786449-dvg2t" Nov 02 18:43:17.161388 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:17.161361 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7d54786449-dvg2t" Nov 02 18:43:17.277990 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:17.277946 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-fd758c546-br2db"] Nov 02 18:43:17.280035 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:43:17.280006 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395c02e5_ad21_4b1b_a7ec_9bbd7dbfe204.slice/crio-c36e6ad528b60e8de5d948005afee000ec42ad1e87f1a584f052c89df352b3a9 WatchSource:0}: Error finding container c36e6ad528b60e8de5d948005afee000ec42ad1e87f1a584f052c89df352b3a9: Status 404 returned error can't find the container with id c36e6ad528b60e8de5d948005afee000ec42ad1e87f1a584f052c89df352b3a9 Nov 02 18:43:17.431917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:17.431888 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7d54786449-dvg2t"] Nov 02 18:43:17.433554 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:43:17.433529 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1200f090_634e_446d_b876_b37375067c84.slice/crio-9a0fcc624082b837589b0bb7a43766bdcac3b9968404ff4b57d3de8ea5f8a5f2 WatchSource:0}: Error finding container 9a0fcc624082b837589b0bb7a43766bdcac3b9968404ff4b57d3de8ea5f8a5f2: Status 404 returned error can't find the container with id 9a0fcc624082b837589b0bb7a43766bdcac3b9968404ff4b57d3de8ea5f8a5f2 Nov 02 18:43:17.586156 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:43:17.586080 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:43:17.863108 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:17.863005 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-fd758c546-br2db" event={"ID":"395c02e5-ad21-4b1b-a7ec-9bbd7dbfe204","Type":"ContainerStarted","Data":"003a6be7a16240a5ee5b05ae9021fc8baf4dc4edd226750fad1d19b926ac0de6"} Nov 02 18:43:17.863108 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:17.863069 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-fd758c546-br2db" event={"ID":"395c02e5-ad21-4b1b-a7ec-9bbd7dbfe204","Type":"ContainerStarted","Data":"c36e6ad528b60e8de5d948005afee000ec42ad1e87f1a584f052c89df352b3a9"} Nov 02 18:43:17.863108 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:17.863102 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-fd758c546-br2db" Nov 02 18:43:17.864556 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:17.864527 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7d54786449-dvg2t" event={"ID":"1200f090-634e-446d-b876-b37375067c84","Type":"ContainerStarted","Data":"ef4f15dcc1681e90759f474de404e7eba36e3bf19cbca60033b0605c41b4c58d"} Nov 02 18:43:17.864684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:17.864581 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7d54786449-dvg2t" event={"ID":"1200f090-634e-446d-b876-b37375067c84","Type":"ContainerStarted","Data":"9a0fcc624082b837589b0bb7a43766bdcac3b9968404ff4b57d3de8ea5f8a5f2"} Nov 02 18:43:17.864684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:17.864613 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7d54786449-dvg2t" Nov 02 18:43:17.881099 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:17.881061 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-fd758c546-br2db" podStartSLOduration=1.881049042 podStartE2EDuration="1.881049042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:43:17.879419579 +0000 UTC m=+4169.667925104" watchObservedRunningTime="2026-11-02 18:43:17.881049042 +0000 UTC m=+4169.669554569" Nov 02 18:43:17.896908 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:17.896863 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7d54786449-dvg2t" podStartSLOduration=1.896849353 podStartE2EDuration="1.896849353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:43:17.895816277 +0000 UTC m=+4169.684321801" watchObservedRunningTime="2026-11-02 18:43:17.896849353 +0000 UTC m=+4169.685354880" Nov 02 18:43:19.847404 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:19.847359 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-5hb2w" podUID="b693ebb5-876a-426b-bf1d-6fdf00d3e96a" containerName="payload-processing" containerID="cri-o://795116a02f6d090dc09a63f9fff16400f64090d41eadd9531880582d94a3cbda" gracePeriod=25 Nov 02 18:43:20.096143 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:20.096119 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-5hb2w" Nov 02 18:43:20.103054 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:20.103000 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b693ebb5-876a-426b-bf1d-6fdf00d3e96a-kube-api-access-h9cvz\" (UniqueName: \"kubernetes.io/projected/b693ebb5-876a-426b-bf1d-6fdf00d3e96a-kube-api-access-h9cvz\") pod \"b693ebb5-876a-426b-bf1d-6fdf00d3e96a\" (UID: \"b693ebb5-876a-426b-bf1d-6fdf00d3e96a\") " Nov 02 18:43:20.103054 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:20.103046 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b693ebb5-876a-426b-bf1d-6fdf00d3e96a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b693ebb5-876a-426b-bf1d-6fdf00d3e96a-plugins-config-volume\") pod \"b693ebb5-876a-426b-bf1d-6fdf00d3e96a\" (UID: \"b693ebb5-876a-426b-bf1d-6fdf00d3e96a\") " Nov 02 18:43:20.103427 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:20.103399 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b693ebb5-876a-426b-bf1d-6fdf00d3e96a-plugins-config-volume" pod "b693ebb5-876a-426b-bf1d-6fdf00d3e96a" (UID: "b693ebb5-876a-426b-bf1d-6fdf00d3e96a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:43:20.104973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:20.104937 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b693ebb5-876a-426b-bf1d-6fdf00d3e96a-kube-api-access-h9cvz" pod "b693ebb5-876a-426b-bf1d-6fdf00d3e96a" (UID: "b693ebb5-876a-426b-bf1d-6fdf00d3e96a"). InnerVolumeSpecName "kube-api-access-h9cvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:43:20.203784 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:20.203746 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9cvz\" (UniqueName: \"kubernetes.io/projected/b693ebb5-876a-426b-bf1d-6fdf00d3e96a-kube-api-access-h9cvz\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:43:20.203784 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:20.203782 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b693ebb5-876a-426b-bf1d-6fdf00d3e96a-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:43:20.878230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:20.878194 5549 generic.go:356] "Generic (PLEG): container finished" podID="b693ebb5-876a-426b-bf1d-6fdf00d3e96a" containerID="795116a02f6d090dc09a63f9fff16400f64090d41eadd9531880582d94a3cbda" exitCode=0 Nov 02 18:43:20.878658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:20.878240 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-5hb2w" Nov 02 18:43:20.878658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:20.878284 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-5hb2w" event={"ID":"b693ebb5-876a-426b-bf1d-6fdf00d3e96a","Type":"ContainerDied","Data":"795116a02f6d090dc09a63f9fff16400f64090d41eadd9531880582d94a3cbda"} Nov 02 18:43:20.878658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:20.878337 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-5hb2w" event={"ID":"b693ebb5-876a-426b-bf1d-6fdf00d3e96a","Type":"ContainerDied","Data":"c6688d00cc84ad97ccf290e8d7e09f543167ccb05d758d1c7160c9f0094563dc"} Nov 02 18:43:20.878658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:20.878359 5549 scope.go:122] "RemoveContainer" containerID="795116a02f6d090dc09a63f9fff16400f64090d41eadd9531880582d94a3cbda" Nov 02 18:43:20.887522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:20.887500 5549 scope.go:122] "RemoveContainer" containerID="795116a02f6d090dc09a63f9fff16400f64090d41eadd9531880582d94a3cbda" Nov 02 18:43:20.887794 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:43:20.887774 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795116a02f6d090dc09a63f9fff16400f64090d41eadd9531880582d94a3cbda\": container with ID starting with 795116a02f6d090dc09a63f9fff16400f64090d41eadd9531880582d94a3cbda not found: ID does not exist" containerID="795116a02f6d090dc09a63f9fff16400f64090d41eadd9531880582d94a3cbda" Nov 02 18:43:20.887865 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:20.887805 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795116a02f6d090dc09a63f9fff16400f64090d41eadd9531880582d94a3cbda"} err="failed to get container status \"795116a02f6d090dc09a63f9fff16400f64090d41eadd9531880582d94a3cbda\": rpc error: code = NotFound desc = could not find container \"795116a02f6d090dc09a63f9fff16400f64090d41eadd9531880582d94a3cbda\": container with ID starting with 795116a02f6d090dc09a63f9fff16400f64090d41eadd9531880582d94a3cbda not found: ID does not exist" Nov 02 18:43:20.895829 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:20.895803 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-5hb2w"] Nov 02 18:43:20.900508 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:20.900478 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-5hb2w"] Nov 02 18:43:22.317828 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:22.317789 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b693ebb5-876a-426b-bf1d-6fdf00d3e96a" path="/var/lib/kubelet/pods/b693ebb5-876a-426b-bf1d-6fdf00d3e96a/volumes" Nov 02 18:43:28.871970 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:28.871922 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-fd758c546-br2db" Nov 02 18:43:28.872483 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:28.872098 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7d54786449-dvg2t" Nov 02 18:43:28.918381 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:28.918339 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-58b55f785c-rh8fz"] Nov 02 18:43:28.928732 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:28.928701 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-99bb68447-w6bqc"] Nov 02 18:43:31.194029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.193942 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:43:31.353117 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.353079 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:43:31.511247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.511208 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-smhvg"] Nov 02 18:43:31.511684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.511671 5549 cpu_manager.go:414] "containerMap: removing container" podUID="b693ebb5-876a-426b-bf1d-6fdf00d3e96a" containerName="payload-processing" Nov 02 18:43:31.511732 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.511685 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b693ebb5-876a-426b-bf1d-6fdf00d3e96a" containerName="payload-processing" Nov 02 18:43:31.511808 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.511799 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="b693ebb5-876a-426b-bf1d-6fdf00d3e96a" containerName="payload-processing" Nov 02 18:43:31.514926 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.514903 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" Nov 02 18:43:31.516108 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.516082 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:43:31.523419 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.523394 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 18:43:31.523551 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.523477 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 02 18:43:31.523660 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.523642 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 18:43:31.523742 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.523710 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-b85rj\"" Nov 02 18:43:31.523883 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.523862 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 18:43:31.524207 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.524185 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 18:43:31.524315 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.524226 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-smhvg"] Nov 02 18:43:31.599255 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.599216 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlzvv\" (UniqueName: \"kubernetes.io/projected/98efd017-eac7-4b29-babb-958ae3fa5230-kube-api-access-xlzvv\") pod \"payload-processing-external-model-5bcd7997-smhvg\" (UID: \"98efd017-eac7-4b29-babb-958ae3fa5230\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" Nov 02 18:43:31.599418 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.599280 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98efd017-eac7-4b29-babb-958ae3fa5230-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-smhvg\" (UID: \"98efd017-eac7-4b29-babb-958ae3fa5230\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" Nov 02 18:43:31.599418 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.599397 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98efd017-eac7-4b29-babb-958ae3fa5230-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-smhvg\" (UID: \"98efd017-eac7-4b29-babb-958ae3fa5230\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" Nov 02 18:43:31.599539 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.599522 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98efd017-eac7-4b29-babb-958ae3fa5230-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-smhvg\" (UID: \"98efd017-eac7-4b29-babb-958ae3fa5230\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" Nov 02 18:43:31.700795 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.700740 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlzvv\" (UniqueName: \"kubernetes.io/projected/98efd017-eac7-4b29-babb-958ae3fa5230-kube-api-access-xlzvv\") pod \"payload-processing-external-model-5bcd7997-smhvg\" (UID: \"98efd017-eac7-4b29-babb-958ae3fa5230\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" Nov 02 18:43:31.701036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.700829 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98efd017-eac7-4b29-babb-958ae3fa5230-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-smhvg\" (UID: \"98efd017-eac7-4b29-babb-958ae3fa5230\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" Nov 02 18:43:31.701036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.700865 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98efd017-eac7-4b29-babb-958ae3fa5230-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-smhvg\" (UID: \"98efd017-eac7-4b29-babb-958ae3fa5230\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" Nov 02 18:43:31.701036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.700984 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98efd017-eac7-4b29-babb-958ae3fa5230-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-smhvg\" (UID: \"98efd017-eac7-4b29-babb-958ae3fa5230\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" Nov 02 18:43:31.701921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.701893 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98efd017-eac7-4b29-babb-958ae3fa5230-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-smhvg\" (UID: \"98efd017-eac7-4b29-babb-958ae3fa5230\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" Nov 02 18:43:31.702083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.702012 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98efd017-eac7-4b29-babb-958ae3fa5230-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-smhvg\" (UID: \"98efd017-eac7-4b29-babb-958ae3fa5230\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" Nov 02 18:43:31.703786 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.703763 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98efd017-eac7-4b29-babb-958ae3fa5230-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-smhvg\" (UID: \"98efd017-eac7-4b29-babb-958ae3fa5230\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" Nov 02 18:43:31.709094 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.709047 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlzvv\" (UniqueName: \"kubernetes.io/projected/98efd017-eac7-4b29-babb-958ae3fa5230-kube-api-access-xlzvv\") pod \"payload-processing-external-model-5bcd7997-smhvg\" (UID: \"98efd017-eac7-4b29-babb-958ae3fa5230\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" Nov 02 18:43:31.831407 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:31.831318 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" Nov 02 18:43:32.103645 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:32.103621 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-smhvg"] Nov 02 18:43:32.106047 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:43:32.106005 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98efd017_eac7_4b29_babb_958ae3fa5230.slice/crio-29d22583206e916db3d9ba15f5376d7ae435b897fc68b15746e373e69becba6b WatchSource:0}: Error finding container 29d22583206e916db3d9ba15f5376d7ae435b897fc68b15746e373e69becba6b: Status 404 returned error can't find the container with id 29d22583206e916db3d9ba15f5376d7ae435b897fc68b15746e373e69becba6b Nov 02 18:43:32.739362 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:32.739324 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:43:32.944567 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:32.944530 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" event={"ID":"98efd017-eac7-4b29-babb-958ae3fa5230","Type":"ContainerStarted","Data":"0bf922ba190497fac49b97c8d12106bad911515cdb069e00f6c92325faffe97a"} Nov 02 18:43:32.944745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:32.944591 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" event={"ID":"98efd017-eac7-4b29-babb-958ae3fa5230","Type":"ContainerStarted","Data":"29d22583206e916db3d9ba15f5376d7ae435b897fc68b15746e373e69becba6b"} Nov 02 18:43:32.944745 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:32.944657 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" Nov 02 18:43:32.966634 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:32.966587 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" podStartSLOduration=1.966574235 podStartE2EDuration="1.966574235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:43:32.966172076 +0000 UTC m=+4184.754677604" watchObservedRunningTime="2026-11-02 18:43:32.966574235 +0000 UTC m=+4184.755079765" Nov 02 18:43:33.918715 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:33.918670 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-58b55f785c-rh8fz" podUID="00fc32f0-057f-40d4-befe-156f8235e05f" containerName="payload-processing" containerID="cri-o://34dca6a15c51982af5554642ef2af36070be479af620699f9dbd29919286d3a4" gracePeriod=25 Nov 02 18:43:33.930149 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:33.930116 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-99bb68447-w6bqc" podUID="cc379d1f-5f1d-45c8-90f9-b3e3f88fc242" containerName="payload-pre-processing" containerID="cri-o://536cfb4de672c5783f9448c01918b98ab65b629bf54b9bd5124e780e5f1f82c3" gracePeriod=25 Nov 02 18:43:34.708983 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:34.704533 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58b55f785c-rh8fz" Nov 02 18:43:34.772985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:34.765273 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00fc32f0-057f-40d4-befe-156f8235e05f-kube-api-access-j5pz6\" (UniqueName: \"kubernetes.io/projected/00fc32f0-057f-40d4-befe-156f8235e05f-kube-api-access-j5pz6\") pod \"00fc32f0-057f-40d4-befe-156f8235e05f\" (UID: \"00fc32f0-057f-40d4-befe-156f8235e05f\") " Nov 02 18:43:34.772985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:34.765372 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00fc32f0-057f-40d4-befe-156f8235e05f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fc32f0-057f-40d4-befe-156f8235e05f-plugins-config-volume\") pod \"00fc32f0-057f-40d4-befe-156f8235e05f\" (UID: \"00fc32f0-057f-40d4-befe-156f8235e05f\") " Nov 02 18:43:34.818021 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:34.812202 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00fc32f0-057f-40d4-befe-156f8235e05f-plugins-config-volume" pod "00fc32f0-057f-40d4-befe-156f8235e05f" (UID: "00fc32f0-057f-40d4-befe-156f8235e05f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:43:34.891562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:34.838133 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fc32f0-057f-40d4-befe-156f8235e05f-kube-api-access-j5pz6" pod "00fc32f0-057f-40d4-befe-156f8235e05f" (UID: "00fc32f0-057f-40d4-befe-156f8235e05f"). InnerVolumeSpecName "kube-api-access-j5pz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:43:34.943529 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:34.942453 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fc32f0-057f-40d4-befe-156f8235e05f-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:43:34.957913 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:34.956489 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-99bb68447-w6bqc" Nov 02 18:43:35.004389 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.002562 5549 generic.go:356] "Generic (PLEG): container finished" podID="00fc32f0-057f-40d4-befe-156f8235e05f" containerID="34dca6a15c51982af5554642ef2af36070be479af620699f9dbd29919286d3a4" exitCode=0 Nov 02 18:43:35.004389 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.002682 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58b55f785c-rh8fz" event={"ID":"00fc32f0-057f-40d4-befe-156f8235e05f","Type":"ContainerDied","Data":"34dca6a15c51982af5554642ef2af36070be479af620699f9dbd29919286d3a4"} Nov 02 18:43:35.004389 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.002732 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58b55f785c-rh8fz" event={"ID":"00fc32f0-057f-40d4-befe-156f8235e05f","Type":"ContainerDied","Data":"8cbf67381d4be7d4cef425ba70764ff0d971ea572362628b815a9b9c05a4d341"} Nov 02 18:43:35.004389 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.002761 5549 scope.go:122] "RemoveContainer" containerID="34dca6a15c51982af5554642ef2af36070be479af620699f9dbd29919286d3a4" Nov 02 18:43:35.004389 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.002911 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58b55f785c-rh8fz" Nov 02 18:43:35.062921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.062876 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc379d1f-5f1d-45c8-90f9-b3e3f88fc242-kube-api-access-6p84n\" (UniqueName: \"kubernetes.io/projected/cc379d1f-5f1d-45c8-90f9-b3e3f88fc242-kube-api-access-6p84n\") pod \"cc379d1f-5f1d-45c8-90f9-b3e3f88fc242\" (UID: \"cc379d1f-5f1d-45c8-90f9-b3e3f88fc242\") " Nov 02 18:43:35.068262 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.067810 5549 generic.go:356] "Generic (PLEG): container finished" podID="cc379d1f-5f1d-45c8-90f9-b3e3f88fc242" containerID="536cfb4de672c5783f9448c01918b98ab65b629bf54b9bd5124e780e5f1f82c3" exitCode=0 Nov 02 18:43:35.069003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.067854 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-99bb68447-w6bqc" event={"ID":"cc379d1f-5f1d-45c8-90f9-b3e3f88fc242","Type":"ContainerDied","Data":"536cfb4de672c5783f9448c01918b98ab65b629bf54b9bd5124e780e5f1f82c3"} Nov 02 18:43:35.069003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.068609 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-99bb68447-w6bqc" event={"ID":"cc379d1f-5f1d-45c8-90f9-b3e3f88fc242","Type":"ContainerDied","Data":"68c46dbe048e5071b47e76628cdc402050ea3547332420a95266150f0e1aebc0"} Nov 02 18:43:35.069003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.067934 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-99bb68447-w6bqc" Nov 02 18:43:35.069805 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.069781 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc379d1f-5f1d-45c8-90f9-b3e3f88fc242-plugins-config-volume" pod "cc379d1f-5f1d-45c8-90f9-b3e3f88fc242" (UID: "cc379d1f-5f1d-45c8-90f9-b3e3f88fc242"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:43:35.070102 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.070080 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc379d1f-5f1d-45c8-90f9-b3e3f88fc242-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc379d1f-5f1d-45c8-90f9-b3e3f88fc242-plugins-config-volume\") pod \"cc379d1f-5f1d-45c8-90f9-b3e3f88fc242\" (UID: \"cc379d1f-5f1d-45c8-90f9-b3e3f88fc242\") " Nov 02 18:43:35.071359 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.071338 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc379d1f-5f1d-45c8-90f9-b3e3f88fc242-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:43:35.071506 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.071488 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5pz6\" (UniqueName: \"kubernetes.io/projected/00fc32f0-057f-40d4-befe-156f8235e05f-kube-api-access-j5pz6\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:43:35.160646 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.103148 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-58b55f785c-rh8fz"] Nov 02 18:43:35.160980 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.160923 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-58b55f785c-rh8fz"] Nov 02 18:43:35.161176 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.107581 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc379d1f-5f1d-45c8-90f9-b3e3f88fc242-kube-api-access-6p84n" pod "cc379d1f-5f1d-45c8-90f9-b3e3f88fc242" (UID: "cc379d1f-5f1d-45c8-90f9-b3e3f88fc242"). InnerVolumeSpecName "kube-api-access-6p84n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:43:35.173995 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.173933 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p84n\" (UniqueName: \"kubernetes.io/projected/cc379d1f-5f1d-45c8-90f9-b3e3f88fc242-kube-api-access-6p84n\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:43:35.199675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.199559 5549 scope.go:122] "RemoveContainer" containerID="34dca6a15c51982af5554642ef2af36070be479af620699f9dbd29919286d3a4" Nov 02 18:43:35.200289 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:43:35.200247 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34dca6a15c51982af5554642ef2af36070be479af620699f9dbd29919286d3a4\": container with ID starting with 34dca6a15c51982af5554642ef2af36070be479af620699f9dbd29919286d3a4 not found: ID does not exist" containerID="34dca6a15c51982af5554642ef2af36070be479af620699f9dbd29919286d3a4" Nov 02 18:43:35.200502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.200451 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34dca6a15c51982af5554642ef2af36070be479af620699f9dbd29919286d3a4"} err="failed to get container status \"34dca6a15c51982af5554642ef2af36070be479af620699f9dbd29919286d3a4\": rpc error: code = NotFound desc = could not find container \"34dca6a15c51982af5554642ef2af36070be479af620699f9dbd29919286d3a4\": container with ID starting with 34dca6a15c51982af5554642ef2af36070be479af620699f9dbd29919286d3a4 not found: ID does not exist" Nov 02 18:43:35.200708 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.200483 5549 scope.go:122] "RemoveContainer" containerID="536cfb4de672c5783f9448c01918b98ab65b629bf54b9bd5124e780e5f1f82c3" Nov 02 18:43:35.255479 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.255203 5549 scope.go:122] "RemoveContainer" containerID="536cfb4de672c5783f9448c01918b98ab65b629bf54b9bd5124e780e5f1f82c3" Nov 02 18:43:35.256457 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:43:35.256419 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"536cfb4de672c5783f9448c01918b98ab65b629bf54b9bd5124e780e5f1f82c3\": container with ID starting with 536cfb4de672c5783f9448c01918b98ab65b629bf54b9bd5124e780e5f1f82c3 not found: ID does not exist" containerID="536cfb4de672c5783f9448c01918b98ab65b629bf54b9bd5124e780e5f1f82c3" Nov 02 18:43:35.256807 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.256776 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"536cfb4de672c5783f9448c01918b98ab65b629bf54b9bd5124e780e5f1f82c3"} err="failed to get container status \"536cfb4de672c5783f9448c01918b98ab65b629bf54b9bd5124e780e5f1f82c3\": rpc error: code = NotFound desc = could not find container \"536cfb4de672c5783f9448c01918b98ab65b629bf54b9bd5124e780e5f1f82c3\": container with ID starting with 536cfb4de672c5783f9448c01918b98ab65b629bf54b9bd5124e780e5f1f82c3 not found: ID does not exist" Nov 02 18:43:35.413989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.413934 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-99bb68447-w6bqc"] Nov 02 18:43:35.432975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:35.420869 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-99bb68447-w6bqc"] Nov 02 18:43:36.318096 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:36.318057 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00fc32f0-057f-40d4-befe-156f8235e05f" path="/var/lib/kubelet/pods/00fc32f0-057f-40d4-befe-156f8235e05f/volumes" Nov 02 18:43:36.318501 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:36.318458 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc379d1f-5f1d-45c8-90f9-b3e3f88fc242" path="/var/lib/kubelet/pods/cc379d1f-5f1d-45c8-90f9-b3e3f88fc242/volumes" Nov 02 18:43:43.955377 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:43.955345 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" Nov 02 18:43:47.586108 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:43:47.586071 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:43:50.096056 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:50.096028 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:43:50.313752 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:43:50.313722 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:44:03.497017 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:44:03.496981 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:44:17.586306 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:44:17.586264 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:44:47.587155 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:44:47.587113 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:45:03.523570 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:03.523530 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:45:17.586105 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:45:17.586062 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:45:22.872771 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:22.872732 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-smhvg"] Nov 02 18:45:22.915236 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:22.915190 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:45:23.049913 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:23.049856 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:45:23.238511 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:23.238472 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:45:23.269636 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:23.269596 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:45:27.878985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:27.876018 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" podUID="98efd017-eac7-4b29-babb-958ae3fa5230" containerName="payload-processing" containerID="cri-o://0bf922ba190497fac49b97c8d12106bad911515cdb069e00f6c92325faffe97a" gracePeriod=25 Nov 02 18:45:28.071028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.070997 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:45:28.533093 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.533034 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" Nov 02 18:45:28.565434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.565404 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98efd017-eac7-4b29-babb-958ae3fa5230-kube-api-access-xlzvv\" (UniqueName: \"kubernetes.io/projected/98efd017-eac7-4b29-babb-958ae3fa5230-kube-api-access-xlzvv\") pod \"98efd017-eac7-4b29-babb-958ae3fa5230\" (UID: \"98efd017-eac7-4b29-babb-958ae3fa5230\") " Nov 02 18:45:28.565684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.565665 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98efd017-eac7-4b29-babb-958ae3fa5230-provider-credentials\" (UniqueName: \"kubernetes.io/projected/98efd017-eac7-4b29-babb-958ae3fa5230-provider-credentials\") pod \"98efd017-eac7-4b29-babb-958ae3fa5230\" (UID: \"98efd017-eac7-4b29-babb-958ae3fa5230\") " Nov 02 18:45:28.565865 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.565845 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98efd017-eac7-4b29-babb-958ae3fa5230-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98efd017-eac7-4b29-babb-958ae3fa5230-routing-overlay-volume\") pod \"98efd017-eac7-4b29-babb-958ae3fa5230\" (UID: \"98efd017-eac7-4b29-babb-958ae3fa5230\") " Nov 02 18:45:28.566061 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.566044 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98efd017-eac7-4b29-babb-958ae3fa5230-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98efd017-eac7-4b29-babb-958ae3fa5230-plugins-config-volume\") pod \"98efd017-eac7-4b29-babb-958ae3fa5230\" (UID: \"98efd017-eac7-4b29-babb-958ae3fa5230\") " Nov 02 18:45:28.567162 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.567133 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98efd017-eac7-4b29-babb-958ae3fa5230-plugins-config-volume" pod "98efd017-eac7-4b29-babb-958ae3fa5230" (UID: "98efd017-eac7-4b29-babb-958ae3fa5230"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:45:28.567292 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.567178 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98efd017-eac7-4b29-babb-958ae3fa5230-routing-overlay-volume" pod "98efd017-eac7-4b29-babb-958ae3fa5230" (UID: "98efd017-eac7-4b29-babb-958ae3fa5230"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:45:28.574587 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.574560 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98efd017-eac7-4b29-babb-958ae3fa5230-provider-credentials" pod "98efd017-eac7-4b29-babb-958ae3fa5230" (UID: "98efd017-eac7-4b29-babb-958ae3fa5230"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:45:28.574922 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.574867 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98efd017-eac7-4b29-babb-958ae3fa5230-kube-api-access-xlzvv" pod "98efd017-eac7-4b29-babb-958ae3fa5230" (UID: "98efd017-eac7-4b29-babb-958ae3fa5230"). InnerVolumeSpecName "kube-api-access-xlzvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:45:28.668235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.668133 5549 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98efd017-eac7-4b29-babb-958ae3fa5230-routing-overlay-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:45:28.668235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.668174 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98efd017-eac7-4b29-babb-958ae3fa5230-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:45:28.668235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.668194 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlzvv\" (UniqueName: \"kubernetes.io/projected/98efd017-eac7-4b29-babb-958ae3fa5230-kube-api-access-xlzvv\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:45:28.668235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.668209 5549 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98efd017-eac7-4b29-babb-958ae3fa5230-provider-credentials\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:45:28.677029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.676282 5549 generic.go:356] "Generic (PLEG): container finished" podID="98efd017-eac7-4b29-babb-958ae3fa5230" containerID="0bf922ba190497fac49b97c8d12106bad911515cdb069e00f6c92325faffe97a" exitCode=0 Nov 02 18:45:28.677029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.676340 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" event={"ID":"98efd017-eac7-4b29-babb-958ae3fa5230","Type":"ContainerDied","Data":"0bf922ba190497fac49b97c8d12106bad911515cdb069e00f6c92325faffe97a"} Nov 02 18:45:28.677029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.676428 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" event={"ID":"98efd017-eac7-4b29-babb-958ae3fa5230","Type":"ContainerDied","Data":"29d22583206e916db3d9ba15f5376d7ae435b897fc68b15746e373e69becba6b"} Nov 02 18:45:28.677029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.676460 5549 scope.go:122] "RemoveContainer" containerID="0bf922ba190497fac49b97c8d12106bad911515cdb069e00f6c92325faffe97a" Nov 02 18:45:28.677029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.676618 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-smhvg" Nov 02 18:45:28.692699 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.692658 5549 scope.go:122] "RemoveContainer" containerID="0bf922ba190497fac49b97c8d12106bad911515cdb069e00f6c92325faffe97a" Nov 02 18:45:28.693192 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:45:28.693158 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf922ba190497fac49b97c8d12106bad911515cdb069e00f6c92325faffe97a\": container with ID starting with 0bf922ba190497fac49b97c8d12106bad911515cdb069e00f6c92325faffe97a not found: ID does not exist" containerID="0bf922ba190497fac49b97c8d12106bad911515cdb069e00f6c92325faffe97a" Nov 02 18:45:28.693322 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.693203 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf922ba190497fac49b97c8d12106bad911515cdb069e00f6c92325faffe97a"} err="failed to get container status \"0bf922ba190497fac49b97c8d12106bad911515cdb069e00f6c92325faffe97a\": rpc error: code = NotFound desc = could not find container \"0bf922ba190497fac49b97c8d12106bad911515cdb069e00f6c92325faffe97a\": container with ID starting with 0bf922ba190497fac49b97c8d12106bad911515cdb069e00f6c92325faffe97a not found: ID does not exist" Nov 02 18:45:28.705893 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.705814 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-smhvg"] Nov 02 18:45:28.707933 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:28.707905 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-smhvg"] Nov 02 18:45:30.329151 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:30.327619 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98efd017-eac7-4b29-babb-958ae3fa5230" path="/var/lib/kubelet/pods/98efd017-eac7-4b29-babb-958ae3fa5230/volumes" Nov 02 18:45:43.746826 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:43.746783 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:45:47.381976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:47.381320 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:45:47.736048 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:45:47.735993 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:45:48.015184 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:48.015073 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-fl8ch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:45:48.015517 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:48.015484 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" podUID="b9b1010f-8c24-44d2-8495-f11e545f2187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:45:59.933414 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:45:59.933379 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:46:03.849993 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:46:03.846526 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:46:12.736519 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:46:12.736483 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:46:17.585778 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:46:17.585737 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:46:33.966793 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:46:33.966754 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:46:36.271250 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:46:36.271205 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xsg9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:46:36.271999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:46:36.271938 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" podUID="4740514e-6d04-468b-930f-8be299ef0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:46:38.277281 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:46:38.277241 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:46:38.411664 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:46:38.411624 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:46:42.610123 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:46:42.610086 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:46:44.161671 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:46:44.161634 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:46:47.683354 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:46:47.683313 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:46:49.278649 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:46:49.278602 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:47:03.843130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:47:03.843095 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:47:04.936320 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:47:04.936283 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:47:08.501815 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:47:08.501772 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:47:08.944038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:47:08.939061 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-r2rbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:47:08.944038 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:47:08.939131 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" podUID="6a7bea79-f2bb-489d-b861-28fa66d0ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:47:08.979054 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:47:08.972039 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-kgx7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:47:08.979054 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:47:08.972113 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" podUID="c208756b-40ff-4691-aad8-9008058e4291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:47:11.863945 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:47:11.863905 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:47:17.586308 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:47:17.586263 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:47:22.843838 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:47:22.843795 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:47:36.473412 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:47:36.473314 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:47:40.044463 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:47:40.044405 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:47:43.474910 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:47:43.474866 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:47:47.585790 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:47:47.585746 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:48:01.761267 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:48:01.761225 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:48:03.994018 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:48:03.993979 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:48:06.113855 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:48:06.113805 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:48:17.586869 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:48:17.586824 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:48:18.445961 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:48:18.445906 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:48:21.873776 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:48:21.873740 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:48:25.851040 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:48:25.851002 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:48:40.871808 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:48:40.871765 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:48:44.331299 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:48:44.331258 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:48:47.586055 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:48:47.586014 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:48:50.126354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:48:50.126323 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:48:50.341330 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:48:50.341303 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:48:51.038687 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:48:51.038645 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:49:04.116230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:04.116154 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:04.249659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:04.249624 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:06.448841 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:06.448802 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:49:10.373121 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:10.373080 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:49:10.641170 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:10.641139 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:10.985100 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:10.985064 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd"] Nov 02 18:49:10.985681 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:10.985653 5549 cpu_manager.go:414] "containerMap: removing container" podUID="98efd017-eac7-4b29-babb-958ae3fa5230" containerName="payload-processing" Nov 02 18:49:10.985681 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:10.985678 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98efd017-eac7-4b29-babb-958ae3fa5230" containerName="payload-processing" Nov 02 18:49:10.985901 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:10.985699 5549 cpu_manager.go:414] "containerMap: removing container" podUID="00fc32f0-057f-40d4-befe-156f8235e05f" containerName="payload-processing" Nov 02 18:49:10.985901 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:10.985708 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00fc32f0-057f-40d4-befe-156f8235e05f" containerName="payload-processing" Nov 02 18:49:10.985901 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:10.985741 5549 cpu_manager.go:414] "containerMap: removing container" podUID="cc379d1f-5f1d-45c8-90f9-b3e3f88fc242" containerName="payload-pre-processing" Nov 02 18:49:10.985901 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:10.985750 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc379d1f-5f1d-45c8-90f9-b3e3f88fc242" containerName="payload-pre-processing" Nov 02 18:49:10.985901 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:10.985844 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="98efd017-eac7-4b29-babb-958ae3fa5230" containerName="payload-processing" Nov 02 18:49:10.985901 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:10.985857 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="00fc32f0-057f-40d4-befe-156f8235e05f" containerName="payload-processing" Nov 02 18:49:10.985901 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:10.985867 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc379d1f-5f1d-45c8-90f9-b3e3f88fc242" containerName="payload-pre-processing" Nov 02 18:49:10.989039 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:10.989016 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:10.997844 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:10.997817 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-3bd024-openshift-default-dockercfg-6j27d\"" Nov 02 18:49:10.999917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:10.999893 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd"] Nov 02 18:49:11.021503 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.021476 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zj7n\" (UniqueName: \"kubernetes.io/projected/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-kube-api-access-2zj7n\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.021665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.021522 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-workload-certs\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.021665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.021540 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-data\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.021813 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.021715 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-podinfo\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.021813 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.021755 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-token\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.021938 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.021825 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-credential-socket\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.021938 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.021870 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-envoy\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.021938 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.021890 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-workload-socket\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.022158 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.022040 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istiod-ca-cert\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.082296 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.082252 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:11.122682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.122643 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-podinfo\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.122682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.122684 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-token\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.122947 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.122708 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-credential-socket\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.122947 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.122763 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-envoy\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.122947 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.122791 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-workload-socket\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.122947 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.122847 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istiod-ca-cert\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.122947 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.122908 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zj7n\" (UniqueName: \"kubernetes.io/projected/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-kube-api-access-2zj7n\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.123231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.122976 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-workload-certs\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.123231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.123001 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-data\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.123565 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.123531 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-workload-socket\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.123662 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.123608 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-credential-socket\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.123912 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.123884 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-workload-certs\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.124047 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.123892 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-data\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.124104 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.124072 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istiod-ca-cert\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.125407 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.125388 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-envoy\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.125494 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.125477 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-podinfo\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.130294 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.130268 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-token\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.130392 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.130349 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zj7n\" (UniqueName: \"kubernetes.io/projected/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-kube-api-access-2zj7n\") pod \"e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.223213 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.223180 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:11.309210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.309117 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:11.585817 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.585789 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd"] Nov 02 18:49:11.588842 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:49:11.588811 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbbd9f3_ec69_414d_9f23_5c38134c8bfd.slice/crio-ac002d26674d53b9bf54fc9baf2ca6964e9391556f43eeacd76cb4e426fe54a9 WatchSource:0}: Error finding container ac002d26674d53b9bf54fc9baf2ca6964e9391556f43eeacd76cb4e426fe54a9: Status 404 returned error can't find the container with id ac002d26674d53b9bf54fc9baf2ca6964e9391556f43eeacd76cb4e426fe54a9 Nov 02 18:49:11.597800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.592799 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:49:11.597800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.593008 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:49:11.597800 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:11.593105 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:49:12.104028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:12.103995 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" event={"ID":"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd","Type":"ContainerStarted","Data":"1dcbdd6dfa96b76f68b81901cbd209855b0c93ce4693503cb1784ee594667e0e"} Nov 02 18:49:12.104189 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:12.104082 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" event={"ID":"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd","Type":"ContainerStarted","Data":"ac002d26674d53b9bf54fc9baf2ca6964e9391556f43eeacd76cb4e426fe54a9"} Nov 02 18:49:12.127623 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:12.127565 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" podStartSLOduration=2.127551806 podStartE2EDuration="2.127551806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:49:12.125319948 +0000 UTC m=+4523.913825482" watchObservedRunningTime="2026-11-02 18:49:12.127551806 +0000 UTC m=+4523.916057333" Nov 02 18:49:12.309435 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:12.309395 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:12.317881 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:12.317854 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:13.108696 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:13.108665 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:13.109790 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:13.109766 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:14.329533 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:14.329502 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:14.463877 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:14.463829 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:15.478668 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.478634 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv"] Nov 02 18:49:15.482615 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.482584 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" Nov 02 18:49:15.491174 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.491062 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-3bd024-serving-cert\"" Nov 02 18:49:15.504422 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.504389 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv"] Nov 02 18:49:15.531245 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.531181 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:15.553624 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.553591 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s"] Nov 02 18:49:15.557308 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.557286 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" Nov 02 18:49:15.564871 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.564846 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-3bd024\"" Nov 02 18:49:15.568173 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.568144 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gmds\" (UniqueName: \"kubernetes.io/projected/7f1ad6b7-09da-431d-8a78-90ffe66346d5-kube-api-access-7gmds\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv\" (UID: \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" Nov 02 18:49:15.568341 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.568319 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f1ad6b7-09da-431d-8a78-90ffe66346d5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv\" (UID: \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" Nov 02 18:49:15.568445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.568427 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f1ad6b7-09da-431d-8a78-90ffe66346d5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv\" (UID: \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" Nov 02 18:49:15.568512 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.568473 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f1ad6b7-09da-431d-8a78-90ffe66346d5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv\" (UID: \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" Nov 02 18:49:15.568775 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.568756 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s"] Nov 02 18:49:15.588724 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.588696 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-c47aa932-dockercfg-g57xt\"" Nov 02 18:49:15.613068 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.613040 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf"] Nov 02 18:49:15.616662 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.616645 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" Nov 02 18:49:15.628468 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.628440 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf"] Nov 02 18:49:15.669414 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.669360 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6488b4c6-7b8b-4903-a1ba-89b687c503a8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf\" (UID: \"6488b4c6-7b8b-4903-a1ba-89b687c503a8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" Nov 02 18:49:15.669738 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.669711 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z84lf\" (UniqueName: \"kubernetes.io/projected/6488b4c6-7b8b-4903-a1ba-89b687c503a8-kube-api-access-z84lf\") pod \"payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf\" (UID: \"6488b4c6-7b8b-4903-a1ba-89b687c503a8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" Nov 02 18:49:15.669986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.669941 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f1ad6b7-09da-431d-8a78-90ffe66346d5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv\" (UID: \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" Nov 02 18:49:15.670209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.670173 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f1ad6b7-09da-431d-8a78-90ffe66346d5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv\" (UID: \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" Nov 02 18:49:15.671194 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.671172 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gmds\" (UniqueName: \"kubernetes.io/projected/7f1ad6b7-09da-431d-8a78-90ffe66346d5-kube-api-access-7gmds\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv\" (UID: \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" Nov 02 18:49:15.671383 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.671362 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333b6b05-370e-43a8-b8b9-3378565c68ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s\" (UID: \"333b6b05-370e-43a8-b8b9-3378565c68ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" Nov 02 18:49:15.697036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.684063 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sqm6\" (UniqueName: \"kubernetes.io/projected/333b6b05-370e-43a8-b8b9-3378565c68ed-kube-api-access-7sqm6\") pod \"payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s\" (UID: \"333b6b05-370e-43a8-b8b9-3378565c68ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" Nov 02 18:49:15.697036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.673802 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:15.697036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.684147 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f1ad6b7-09da-431d-8a78-90ffe66346d5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv\" (UID: \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" Nov 02 18:49:15.697036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.681438 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gmds\" (UniqueName: \"kubernetes.io/projected/7f1ad6b7-09da-431d-8a78-90ffe66346d5-kube-api-access-7gmds\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv\" (UID: \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" Nov 02 18:49:15.697036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.678420 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f1ad6b7-09da-431d-8a78-90ffe66346d5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv\" (UID: \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" Nov 02 18:49:15.697036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.685703 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f1ad6b7-09da-431d-8a78-90ffe66346d5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv\" (UID: \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" Nov 02 18:49:15.697036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.688406 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f1ad6b7-09da-431d-8a78-90ffe66346d5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv\" (UID: \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" Nov 02 18:49:15.785051 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.784945 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6488b4c6-7b8b-4903-a1ba-89b687c503a8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf\" (UID: \"6488b4c6-7b8b-4903-a1ba-89b687c503a8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" Nov 02 18:49:15.785051 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.785027 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z84lf\" (UniqueName: \"kubernetes.io/projected/6488b4c6-7b8b-4903-a1ba-89b687c503a8-kube-api-access-z84lf\") pod \"payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf\" (UID: \"6488b4c6-7b8b-4903-a1ba-89b687c503a8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" Nov 02 18:49:15.785277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.785125 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333b6b05-370e-43a8-b8b9-3378565c68ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s\" (UID: \"333b6b05-370e-43a8-b8b9-3378565c68ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" Nov 02 18:49:15.785277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.785151 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sqm6\" (UniqueName: \"kubernetes.io/projected/333b6b05-370e-43a8-b8b9-3378565c68ed-kube-api-access-7sqm6\") pod \"payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s\" (UID: \"333b6b05-370e-43a8-b8b9-3378565c68ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" Nov 02 18:49:15.785673 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.785653 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6488b4c6-7b8b-4903-a1ba-89b687c503a8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf\" (UID: \"6488b4c6-7b8b-4903-a1ba-89b687c503a8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" Nov 02 18:49:15.785854 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.785832 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333b6b05-370e-43a8-b8b9-3378565c68ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s\" (UID: \"333b6b05-370e-43a8-b8b9-3378565c68ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" Nov 02 18:49:15.795504 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.795473 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sqm6\" (UniqueName: \"kubernetes.io/projected/333b6b05-370e-43a8-b8b9-3378565c68ed-kube-api-access-7sqm6\") pod \"payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s\" (UID: \"333b6b05-370e-43a8-b8b9-3378565c68ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" Nov 02 18:49:15.795667 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.795646 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z84lf\" (UniqueName: \"kubernetes.io/projected/6488b4c6-7b8b-4903-a1ba-89b687c503a8-kube-api-access-z84lf\") pod \"payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf\" (UID: \"6488b4c6-7b8b-4903-a1ba-89b687c503a8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" Nov 02 18:49:15.797354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.797334 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" Nov 02 18:49:15.813853 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.813828 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:15.873123 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.872757 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" Nov 02 18:49:15.933613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:15.932739 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" Nov 02 18:49:16.091328 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:16.091300 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv"] Nov 02 18:49:16.094020 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:49:16.093987 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1ad6b7_09da_431d_8a78_90ffe66346d5.slice/crio-dc553dc0df6a90c3ea430ad1096ff9fe34a4ac64a92fe981854436876898847f WatchSource:0}: Error finding container dc553dc0df6a90c3ea430ad1096ff9fe34a4ac64a92fe981854436876898847f: Status 404 returned error can't find the container with id dc553dc0df6a90c3ea430ad1096ff9fe34a4ac64a92fe981854436876898847f Nov 02 18:49:16.095562 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:16.095540 5549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:49:16.124032 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:16.123995 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" event={"ID":"7f1ad6b7-09da-431d-8a78-90ffe66346d5","Type":"ContainerStarted","Data":"dc553dc0df6a90c3ea430ad1096ff9fe34a4ac64a92fe981854436876898847f"} Nov 02 18:49:16.151868 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:16.151842 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s"] Nov 02 18:49:16.154459 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:49:16.154428 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333b6b05_370e_43a8_b8b9_3378565c68ed.slice/crio-ee831133f2ae718fc42580d074be5986d19b015764efd98aac17eaf9c991ae69 WatchSource:0}: Error finding container ee831133f2ae718fc42580d074be5986d19b015764efd98aac17eaf9c991ae69: Status 404 returned error can't find the container with id ee831133f2ae718fc42580d074be5986d19b015764efd98aac17eaf9c991ae69 Nov 02 18:49:16.230511 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:16.230483 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf"] Nov 02 18:49:16.232111 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:49:16.232087 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6488b4c6_7b8b_4903_a1ba_89b687c503a8.slice/crio-37ed3f2229d32926389964681e3613055b1a89dd323f2694b68a73d79fb1fc94 WatchSource:0}: Error finding container 37ed3f2229d32926389964681e3613055b1a89dd323f2694b68a73d79fb1fc94: Status 404 returned error can't find the container with id 37ed3f2229d32926389964681e3613055b1a89dd323f2694b68a73d79fb1fc94 Nov 02 18:49:16.383787 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:16.383692 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:17.131123 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:17.131082 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" event={"ID":"7f1ad6b7-09da-431d-8a78-90ffe66346d5","Type":"ContainerStarted","Data":"949996e77c4a9b8cc55f58206bc092712072b3d80302fed0d6858a471646a28f"} Nov 02 18:49:17.131632 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:17.131274 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" Nov 02 18:49:17.133028 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:17.132999 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" event={"ID":"6488b4c6-7b8b-4903-a1ba-89b687c503a8","Type":"ContainerStarted","Data":"2c5f4a6751ec9f72740cca1d7223e514809c03201dfd9bf247ed886e746aeafb"} Nov 02 18:49:17.133150 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:17.133040 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" event={"ID":"6488b4c6-7b8b-4903-a1ba-89b687c503a8","Type":"ContainerStarted","Data":"37ed3f2229d32926389964681e3613055b1a89dd323f2694b68a73d79fb1fc94"} Nov 02 18:49:17.133150 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:17.133097 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" Nov 02 18:49:17.134633 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:17.134607 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" event={"ID":"333b6b05-370e-43a8-b8b9-3378565c68ed","Type":"ContainerStarted","Data":"8507be70721581124632ccccc9d6fa683fd079213f476b6c24a3f6ee69b715da"} Nov 02 18:49:17.134744 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:17.134641 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" event={"ID":"333b6b05-370e-43a8-b8b9-3378565c68ed","Type":"ContainerStarted","Data":"ee831133f2ae718fc42580d074be5986d19b015764efd98aac17eaf9c991ae69"} Nov 02 18:49:17.134827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:17.134750 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" Nov 02 18:49:17.149900 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:17.149859 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:17.154651 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:17.152848 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" podStartSLOduration=1.667972626 podStartE2EDuration="2.152834216s" totalImagesPullingTime="484.86159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:49:17.152339459 +0000 UTC m=+4528.940844988" watchObservedRunningTime="2026-11-02 18:49:17.152834216 +0000 UTC m=+4528.941339740" Nov 02 18:49:17.168230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:17.168186 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" podStartSLOduration=2.16817373 podStartE2EDuration="2.16817373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:49:17.167622165 +0000 UTC m=+4528.956127694" watchObservedRunningTime="2026-11-02 18:49:17.16817373 +0000 UTC m=+4528.956679257" Nov 02 18:49:17.586285 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:49:17.586245 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:49:17.745224 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:17.745190 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:17.868542 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:17.868436 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" podStartSLOduration=2.868421157 podStartE2EDuration="2.868421157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:49:17.184557943 +0000 UTC m=+4528.973063471" watchObservedRunningTime="2026-11-02 18:49:17.868421157 +0000 UTC m=+4529.656926713" Nov 02 18:49:17.869025 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:17.869009 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:49:18.463490 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:18.463449 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:19.165431 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:19.165374 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:19.866654 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:19.866605 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:20.574473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:20.574421 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:21.373065 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:21.373024 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:22.112496 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:22.112449 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:22.765355 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:22.765315 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:23.144298 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:23.144226 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" Nov 02 18:49:23.472498 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:23.472451 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:24.160210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:24.160174 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:24.912355 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:24.912307 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:25.588155 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:25.588119 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:26.245196 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:26.245159 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:26.966451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:26.966360 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:27.649313 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:27.649274 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:28.110248 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:28.110216 5549 patch_prober.go:29] interesting pod/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Nov 02 18:49:28.110644 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:28.110269 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" podUID="7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Nov 02 18:49:28.141803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:28.141776 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" Nov 02 18:49:28.141985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:28.141848 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" Nov 02 18:49:28.190846 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:28.190810 5549 generic.go:356] "Generic (PLEG): container finished" podID="7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" containerID="1dcbdd6dfa96b76f68b81901cbd209855b0c93ce4693503cb1784ee594667e0e" exitCode=0 Nov 02 18:49:28.191070 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:28.190885 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" event={"ID":"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd","Type":"ContainerDied","Data":"1dcbdd6dfa96b76f68b81901cbd209855b0c93ce4693503cb1784ee594667e0e"} Nov 02 18:49:28.191329 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:28.191309 5549 scope.go:122] "RemoveContainer" containerID="1dcbdd6dfa96b76f68b81901cbd209855b0c93ce4693503cb1784ee594667e0e" Nov 02 18:49:28.192836 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:28.192805 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:49:28.192948 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:28.192885 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:49:28.192948 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:28.192936 5549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:49:28.259219 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:28.259195 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:28.973326 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:28.973285 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:29.211906 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:29.211869 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" event={"ID":"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd","Type":"ContainerStarted","Data":"53a7fcad8968274fa076785fa303cdd9b29c3a4f6c3da8f2a037bce7c11191f5"} Nov 02 18:49:29.309373 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:29.309285 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:29.309582 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:29.309554 5549 patch_prober.go:29] interesting pod/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Nov 02 18:49:29.309655 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:29.309626 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" podUID="7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Nov 02 18:49:29.318843 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:29.318801 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:29.676454 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:29.676352 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:29.938815 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:29.938727 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:30.075135 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:30.075096 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:30.205804 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:30.205765 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:30.310428 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:30.310383 5549 patch_prober.go:29] interesting pod/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Nov 02 18:49:30.310836 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:30.310468 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" podUID="7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Nov 02 18:49:30.543561 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:30.543463 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:31.254303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.254264 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:31.310493 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.310448 5549 patch_prober.go:29] interesting pod/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Nov 02 18:49:31.310929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.310522 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" podUID="7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Nov 02 18:49:31.732457 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.732423 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg"] Nov 02 18:49:31.736277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.736254 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:31.743296 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.743265 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3bd024\"/\"kube-root-ca.crt\"" Nov 02 18:49:31.745742 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.745714 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3bd024\"/\"default-dockercfg-8h5wc\"" Nov 02 18:49:31.745912 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.745754 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3bd024\"/\"simulator-single-main-3bd024-kserve-self-signed-certs\"" Nov 02 18:49:31.746060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.745808 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3bd024\"/\"openshift-service-ca.crt\"" Nov 02 18:49:31.748318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.748291 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg"] Nov 02 18:49:31.828395 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.828359 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:31.851683 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.851652 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-home\") pod \"simulator-single-main-3bd024-kserve-ffcc66749-txxsg\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:31.851846 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.851697 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-model-cache\") pod \"simulator-single-main-3bd024-kserve-ffcc66749-txxsg\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:31.851846 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.851816 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c9524e2-3937-45cc-8643-327d0b112e34-tls-certs\") pod \"simulator-single-main-3bd024-kserve-ffcc66749-txxsg\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:31.851934 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.851897 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-dshm\") pod \"simulator-single-main-3bd024-kserve-ffcc66749-txxsg\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:31.852023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.851945 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcfcd\" (UniqueName: \"kubernetes.io/projected/0c9524e2-3937-45cc-8643-327d0b112e34-kube-api-access-zcfcd\") pod \"simulator-single-main-3bd024-kserve-ffcc66749-txxsg\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:31.852023 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.852017 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-tmp-dir\") pod \"simulator-single-main-3bd024-kserve-ffcc66749-txxsg\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:31.953409 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.953372 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcfcd\" (UniqueName: \"kubernetes.io/projected/0c9524e2-3937-45cc-8643-327d0b112e34-kube-api-access-zcfcd\") pod \"simulator-single-main-3bd024-kserve-ffcc66749-txxsg\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:31.953601 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.953436 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-tmp-dir\") pod \"simulator-single-main-3bd024-kserve-ffcc66749-txxsg\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:31.953601 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.953483 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-home\") pod \"simulator-single-main-3bd024-kserve-ffcc66749-txxsg\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:31.953601 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.953580 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-model-cache\") pod \"simulator-single-main-3bd024-kserve-ffcc66749-txxsg\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:31.953824 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.953653 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c9524e2-3937-45cc-8643-327d0b112e34-tls-certs\") pod \"simulator-single-main-3bd024-kserve-ffcc66749-txxsg\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:31.953824 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.953775 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-dshm\") pod \"simulator-single-main-3bd024-kserve-ffcc66749-txxsg\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:31.953932 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.953911 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-tmp-dir\") pod \"simulator-single-main-3bd024-kserve-ffcc66749-txxsg\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:31.953999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.953944 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-home\") pod \"simulator-single-main-3bd024-kserve-ffcc66749-txxsg\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:31.954073 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.954018 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-model-cache\") pod \"simulator-single-main-3bd024-kserve-ffcc66749-txxsg\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:31.956022 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.955999 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-dshm\") pod \"simulator-single-main-3bd024-kserve-ffcc66749-txxsg\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:31.956185 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.956168 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c9524e2-3937-45cc-8643-327d0b112e34-tls-certs\") pod \"simulator-single-main-3bd024-kserve-ffcc66749-txxsg\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:31.961239 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:31.961217 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcfcd\" (UniqueName: \"kubernetes.io/projected/0c9524e2-3937-45cc-8643-327d0b112e34-kube-api-access-zcfcd\") pod \"simulator-single-main-3bd024-kserve-ffcc66749-txxsg\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:32.009642 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:32.009538 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:32.051334 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:32.051295 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:32.309840 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:32.309753 5549 patch_prober.go:29] interesting pod/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Nov 02 18:49:32.309840 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:32.309811 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" podUID="7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Nov 02 18:49:32.349509 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:32.349479 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg"] Nov 02 18:49:32.351930 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:49:32.351905 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c9524e2_3937_45cc_8643_327d0b112e34.slice/crio-11606f03cdea868112f0fc7ff682f84d84aa867c18d78b1ab3985cb5c82baf35 WatchSource:0}: Error finding container 11606f03cdea868112f0fc7ff682f84d84aa867c18d78b1ab3985cb5c82baf35: Status 404 returned error can't find the container with id 11606f03cdea868112f0fc7ff682f84d84aa867c18d78b1ab3985cb5c82baf35 Nov 02 18:49:32.764277 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:32.763904 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:33.233552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:33.233505 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" event={"ID":"0c9524e2-3937-45cc-8643-327d0b112e34","Type":"ContainerStarted","Data":"775b1be06b6c6eb5ffdc5093560d94ab3feef94da52f33cc9b5bf9d0d04a8d0b"} Nov 02 18:49:33.233552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:33.233553 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" event={"ID":"0c9524e2-3937-45cc-8643-327d0b112e34","Type":"ContainerStarted","Data":"11606f03cdea868112f0fc7ff682f84d84aa867c18d78b1ab3985cb5c82baf35"} Nov 02 18:49:33.250374 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:33.250323 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" podStartSLOduration=2.250310865 podStartE2EDuration="2.250310865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:49:33.248749629 +0000 UTC m=+4545.037255156" watchObservedRunningTime="2026-11-02 18:49:33.250310865 +0000 UTC m=+4545.038816384" Nov 02 18:49:33.310564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:33.310524 5549 patch_prober.go:29] interesting pod/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Nov 02 18:49:33.310746 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:33.310592 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" podUID="7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Nov 02 18:49:33.357503 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:33.357465 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:49:33.471209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:33.471171 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:34.051576 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:34.051541 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:34.310471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:34.310386 5549 patch_prober.go:29] interesting pod/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Nov 02 18:49:34.310471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:34.310448 5549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" podUID="7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Nov 02 18:49:34.344256 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:34.344213 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:35.099509 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:35.099471 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:35.314380 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:35.314346 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:35.314734 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:35.314712 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:35.315414 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:35.315393 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:49:35.745721 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:35.745683 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:36.353913 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:36.353867 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:36.961514 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:36.961476 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:37.560872 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:37.560839 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:38.153700 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:38.153159 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:38.777631 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:38.777588 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:39.352519 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:39.352479 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:40.062629 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:40.062589 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:40.651321 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:40.651287 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:41.273116 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:41.273077 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:41.955770 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:41.955737 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:42.052261 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:42.052225 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:42.064705 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:42.064676 5549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:42.294353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:42.294276 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:49:42.625060 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:42.624919 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:43.249284 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:43.249249 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:43.853626 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:43.853592 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:44.545422 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:44.545383 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:45.148149 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:45.148109 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:45.754143 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:45.754107 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:46.358108 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:46.358068 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:46.952573 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:46.952535 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:47.586130 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:49:47.586096 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:49:47.662343 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:47.662307 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:48.346881 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:48.346841 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:48.636499 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:48.636419 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:49:49.023862 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:49.023821 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:49.666456 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:49.666415 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:50.259703 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:50.259663 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:50.974531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:50.974494 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:51.559855 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:51.559811 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:52.162240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:52.162203 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:52.756235 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:52.756161 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:53.348502 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:53.348417 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:53.964354 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:53.964273 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:54.564595 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:54.564560 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:55.163181 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:55.163143 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:55.848240 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:55.847248 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:57.452837 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:57.452801 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:57.575470 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:57.575438 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:57.701011 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:57.700978 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:58.246227 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:58.246191 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:58.851535 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:58.851498 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:49:59.551855 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:49:59.551800 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:00.148523 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:00.148470 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:00.757330 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:00.757293 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:01.432590 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:01.432548 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:02.250394 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:02.250347 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:02.851545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:02.851508 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:03.464695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:03.464654 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:04.065350 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:04.065312 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:04.195549 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:04.195514 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:04.660338 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:04.660300 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:05.257341 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:05.257303 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:05.855728 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:05.855677 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:08.226567 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:08.226527 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:08.385661 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:08.385625 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:08.972716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:08.972682 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:09.564161 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:09.564119 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:10.246350 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:10.246279 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:10.868728 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:10.868695 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:11.662635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:11.662593 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:12.060099 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:12.060033 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:12.262748 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:12.262705 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:12.538451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:12.538402 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:12.750727 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:12.750695 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:50:12.940043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:12.939966 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:13.652215 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:13.652174 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:14.363876 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:14.363833 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:15.012862 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:15.012826 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:15.656366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:15.656331 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:16.351979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:16.351928 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:16.960248 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:16.960212 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg"] Nov 02 18:50:16.974887 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:16.974767 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:17.102246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:17.102216 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:17.588758 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:50:17.588719 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:50:17.606703 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:17.606591 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:18.059399 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:18.059357 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf"] Nov 02 18:50:18.072823 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:18.072794 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s"] Nov 02 18:50:18.182376 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:18.182344 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:18.308658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:18.308621 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:18.602318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:18.602285 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:18.897841 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:18.897756 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:19.025244 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.025202 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv"] Nov 02 18:50:19.025465 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.025432 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" podUID="7f1ad6b7-09da-431d-8a78-90ffe66346d5" containerName="maas-api" containerID="cri-o://949996e77c4a9b8cc55f58206bc092712072b3d80302fed0d6858a471646a28f" gracePeriod=30 Nov 02 18:50:19.122665 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.122634 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:19.459751 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.459719 5549 generic.go:356] "Generic (PLEG): container finished" podID="7f1ad6b7-09da-431d-8a78-90ffe66346d5" containerID="949996e77c4a9b8cc55f58206bc092712072b3d80302fed0d6858a471646a28f" exitCode=0 Nov 02 18:50:19.459902 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.459772 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" event={"ID":"7f1ad6b7-09da-431d-8a78-90ffe66346d5","Type":"ContainerDied","Data":"949996e77c4a9b8cc55f58206bc092712072b3d80302fed0d6858a471646a28f"} Nov 02 18:50:19.459902 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.459838 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" event={"ID":"7f1ad6b7-09da-431d-8a78-90ffe66346d5","Type":"ContainerDied","Data":"dc553dc0df6a90c3ea430ad1096ff9fe34a4ac64a92fe981854436876898847f"} Nov 02 18:50:19.459902 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.459863 5549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc553dc0df6a90c3ea430ad1096ff9fe34a4ac64a92fe981854436876898847f" Nov 02 18:50:19.472266 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.472242 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" Nov 02 18:50:19.587106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.587022 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f1ad6b7-09da-431d-8a78-90ffe66346d5-kube-api-access-7gmds\" (UniqueName: \"kubernetes.io/projected/7f1ad6b7-09da-431d-8a78-90ffe66346d5-kube-api-access-7gmds\") pod \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\" (UID: \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\") " Nov 02 18:50:19.587106 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.587072 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f1ad6b7-09da-431d-8a78-90ffe66346d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f1ad6b7-09da-431d-8a78-90ffe66346d5-metrics-certs\") pod \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\" (UID: \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\") " Nov 02 18:50:19.587327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.587193 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f1ad6b7-09da-431d-8a78-90ffe66346d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f1ad6b7-09da-431d-8a78-90ffe66346d5-maas-api-tls\") pod \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\" (UID: \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\") " Nov 02 18:50:19.587327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.587240 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f1ad6b7-09da-431d-8a78-90ffe66346d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f1ad6b7-09da-431d-8a78-90ffe66346d5-openshift-service-ca\") pod \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\" (UID: \"7f1ad6b7-09da-431d-8a78-90ffe66346d5\") " Nov 02 18:50:19.587720 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.587689 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f1ad6b7-09da-431d-8a78-90ffe66346d5-openshift-service-ca" pod "7f1ad6b7-09da-431d-8a78-90ffe66346d5" (UID: "7f1ad6b7-09da-431d-8a78-90ffe66346d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:50:19.589281 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.589259 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f1ad6b7-09da-431d-8a78-90ffe66346d5-kube-api-access-7gmds" pod "7f1ad6b7-09da-431d-8a78-90ffe66346d5" (UID: "7f1ad6b7-09da-431d-8a78-90ffe66346d5"). InnerVolumeSpecName "kube-api-access-7gmds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:50:19.589385 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.589353 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f1ad6b7-09da-431d-8a78-90ffe66346d5-metrics-certs" pod "7f1ad6b7-09da-431d-8a78-90ffe66346d5" (UID: "7f1ad6b7-09da-431d-8a78-90ffe66346d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:50:19.589454 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.589441 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f1ad6b7-09da-431d-8a78-90ffe66346d5-maas-api-tls" pod "7f1ad6b7-09da-431d-8a78-90ffe66346d5" (UID: "7f1ad6b7-09da-431d-8a78-90ffe66346d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:50:19.688271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.688233 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gmds\" (UniqueName: \"kubernetes.io/projected/7f1ad6b7-09da-431d-8a78-90ffe66346d5-kube-api-access-7gmds\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:50:19.688271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.688264 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f1ad6b7-09da-431d-8a78-90ffe66346d5-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:50:19.688271 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.688275 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f1ad6b7-09da-431d-8a78-90ffe66346d5-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:50:19.688827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:19.688284 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f1ad6b7-09da-431d-8a78-90ffe66346d5-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:50:20.465031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:20.464993 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv" Nov 02 18:50:20.481267 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:20.481232 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv"] Nov 02 18:50:20.484851 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:20.484822 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-7pvlv"] Nov 02 18:50:20.642284 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:20.642201 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:50:22.318439 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:22.318399 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f1ad6b7-09da-431d-8a78-90ffe66346d5" path="/var/lib/kubelet/pods/7f1ad6b7-09da-431d-8a78-90ffe66346d5/volumes" Nov 02 18:50:23.060639 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:23.060595 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" podUID="6488b4c6-7b8b-4903-a1ba-89b687c503a8" containerName="payload-processing" containerID="cri-o://2c5f4a6751ec9f72740cca1d7223e514809c03201dfd9bf247ed886e746aeafb" gracePeriod=25 Nov 02 18:50:23.073116 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:23.073083 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" podUID="333b6b05-370e-43a8-b8b9-3378565c68ed" containerName="payload-pre-processing" containerID="cri-o://8507be70721581124632ccccc9d6fa683fd079213f476b6c24a3f6ee69b715da" gracePeriod=25 Nov 02 18:50:24.520466 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:24.520433 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:24.659314 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:24.659278 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:28.140170 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:28.140122 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" podUID="6488b4c6-7b8b-4903-a1ba-89b687c503a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:50:28.140564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:28.140125 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" podUID="333b6b05-370e-43a8-b8b9-3378565c68ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:50:31.987403 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:31.987301 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" podUID="0c9524e2-3937-45cc-8643-327d0b112e34" containerName="main" containerID="cri-o://775b1be06b6c6eb5ffdc5093560d94ab3feef94da52f33cc9b5bf9d0d04a8d0b" gracePeriod=45 Nov 02 18:50:32.249093 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.249028 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:50:32.313779 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.313745 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-home\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-home\") pod \"0c9524e2-3937-45cc-8643-327d0b112e34\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " Nov 02 18:50:32.313779 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.313787 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9524e2-3937-45cc-8643-327d0b112e34-kube-api-access-zcfcd\" (UniqueName: \"kubernetes.io/projected/0c9524e2-3937-45cc-8643-327d0b112e34-kube-api-access-zcfcd\") pod \"0c9524e2-3937-45cc-8643-327d0b112e34\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " Nov 02 18:50:32.314044 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.313877 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-tmp-dir\") pod \"0c9524e2-3937-45cc-8643-327d0b112e34\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " Nov 02 18:50:32.314044 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.313916 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-dshm\") pod \"0c9524e2-3937-45cc-8643-327d0b112e34\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " Nov 02 18:50:32.314044 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.313946 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9524e2-3937-45cc-8643-327d0b112e34-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c9524e2-3937-45cc-8643-327d0b112e34-tls-certs\") pod \"0c9524e2-3937-45cc-8643-327d0b112e34\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " Nov 02 18:50:32.314044 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.313994 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-model-cache\") pod \"0c9524e2-3937-45cc-8643-327d0b112e34\" (UID: \"0c9524e2-3937-45cc-8643-327d0b112e34\") " Nov 02 18:50:32.314284 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.314079 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-home" pod "0c9524e2-3937-45cc-8643-327d0b112e34" (UID: "0c9524e2-3937-45cc-8643-327d0b112e34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:32.314284 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.314210 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-tmp-dir" pod "0c9524e2-3937-45cc-8643-327d0b112e34" (UID: "0c9524e2-3937-45cc-8643-327d0b112e34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:32.314429 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.314410 5549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-tmp-dir\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:50:32.314514 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.314435 5549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-home\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:50:32.314716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.314673 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-model-cache" pod "0c9524e2-3937-45cc-8643-327d0b112e34" (UID: "0c9524e2-3937-45cc-8643-327d0b112e34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:32.316383 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.316354 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-dshm" pod "0c9524e2-3937-45cc-8643-327d0b112e34" (UID: "0c9524e2-3937-45cc-8643-327d0b112e34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:32.317297 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.317267 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9524e2-3937-45cc-8643-327d0b112e34-tls-certs" pod "0c9524e2-3937-45cc-8643-327d0b112e34" (UID: "0c9524e2-3937-45cc-8643-327d0b112e34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:50:32.317434 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.317303 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9524e2-3937-45cc-8643-327d0b112e34-kube-api-access-zcfcd" pod "0c9524e2-3937-45cc-8643-327d0b112e34" (UID: "0c9524e2-3937-45cc-8643-327d0b112e34"). InnerVolumeSpecName "kube-api-access-zcfcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:50:32.415352 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.415309 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcfcd\" (UniqueName: \"kubernetes.io/projected/0c9524e2-3937-45cc-8643-327d0b112e34-kube-api-access-zcfcd\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:50:32.415352 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.415346 5549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-dshm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:50:32.415638 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.415363 5549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c9524e2-3937-45cc-8643-327d0b112e34-tls-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:50:32.415638 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.415376 5549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c9524e2-3937-45cc-8643-327d0b112e34-model-cache\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:50:32.518087 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.517989 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:32.521265 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.521233 5549 generic.go:356] "Generic (PLEG): container finished" podID="0c9524e2-3937-45cc-8643-327d0b112e34" containerID="775b1be06b6c6eb5ffdc5093560d94ab3feef94da52f33cc9b5bf9d0d04a8d0b" exitCode=0 Nov 02 18:50:32.521375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.521279 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" Nov 02 18:50:32.521375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.521327 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" event={"ID":"0c9524e2-3937-45cc-8643-327d0b112e34","Type":"ContainerDied","Data":"775b1be06b6c6eb5ffdc5093560d94ab3feef94da52f33cc9b5bf9d0d04a8d0b"} Nov 02 18:50:32.521375 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.521364 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg" event={"ID":"0c9524e2-3937-45cc-8643-327d0b112e34","Type":"ContainerDied","Data":"11606f03cdea868112f0fc7ff682f84d84aa867c18d78b1ab3985cb5c82baf35"} Nov 02 18:50:32.521527 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.521383 5549 scope.go:122] "RemoveContainer" containerID="775b1be06b6c6eb5ffdc5093560d94ab3feef94da52f33cc9b5bf9d0d04a8d0b" Nov 02 18:50:32.530230 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.530209 5549 scope.go:122] "RemoveContainer" containerID="775b1be06b6c6eb5ffdc5093560d94ab3feef94da52f33cc9b5bf9d0d04a8d0b" Nov 02 18:50:32.530491 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:50:32.530468 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775b1be06b6c6eb5ffdc5093560d94ab3feef94da52f33cc9b5bf9d0d04a8d0b\": container with ID starting with 775b1be06b6c6eb5ffdc5093560d94ab3feef94da52f33cc9b5bf9d0d04a8d0b not found: ID does not exist" containerID="775b1be06b6c6eb5ffdc5093560d94ab3feef94da52f33cc9b5bf9d0d04a8d0b" Nov 02 18:50:32.530541 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.530500 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775b1be06b6c6eb5ffdc5093560d94ab3feef94da52f33cc9b5bf9d0d04a8d0b"} err="failed to get container status \"775b1be06b6c6eb5ffdc5093560d94ab3feef94da52f33cc9b5bf9d0d04a8d0b\": rpc error: code = NotFound desc = could not find container \"775b1be06b6c6eb5ffdc5093560d94ab3feef94da52f33cc9b5bf9d0d04a8d0b\": container with ID starting with 775b1be06b6c6eb5ffdc5093560d94ab3feef94da52f33cc9b5bf9d0d04a8d0b not found: ID does not exist" Nov 02 18:50:32.537716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.537692 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg"] Nov 02 18:50:32.541263 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.541240 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-3bd024/simulator-single-main-3bd024-kserve-ffcc66749-txxsg"] Nov 02 18:50:32.642083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:32.642050 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:34.318716 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:34.318679 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c9524e2-3937-45cc-8643-327d0b112e34" path="/var/lib/kubelet/pods/0c9524e2-3937-45cc-8643-327d0b112e34/volumes" Nov 02 18:50:38.140779 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:38.140735 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" podUID="333b6b05-370e-43a8-b8b9-3378565c68ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:50:38.140779 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:38.140765 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" podUID="6488b4c6-7b8b-4903-a1ba-89b687c503a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:50:43.324209 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.324143 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" Nov 02 18:50:43.350835 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.350808 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" Nov 02 18:50:43.428636 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.428603 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/333b6b05-370e-43a8-b8b9-3378565c68ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333b6b05-370e-43a8-b8b9-3378565c68ed-plugins-config-volume\") pod \"333b6b05-370e-43a8-b8b9-3378565c68ed\" (UID: \"333b6b05-370e-43a8-b8b9-3378565c68ed\") " Nov 02 18:50:43.428834 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.428813 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6488b4c6-7b8b-4903-a1ba-89b687c503a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6488b4c6-7b8b-4903-a1ba-89b687c503a8-plugins-config-volume\") pod \"6488b4c6-7b8b-4903-a1ba-89b687c503a8\" (UID: \"6488b4c6-7b8b-4903-a1ba-89b687c503a8\") " Nov 02 18:50:43.428879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.428862 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6488b4c6-7b8b-4903-a1ba-89b687c503a8-kube-api-access-z84lf\" (UniqueName: \"kubernetes.io/projected/6488b4c6-7b8b-4903-a1ba-89b687c503a8-kube-api-access-z84lf\") pod \"6488b4c6-7b8b-4903-a1ba-89b687c503a8\" (UID: \"6488b4c6-7b8b-4903-a1ba-89b687c503a8\") " Nov 02 18:50:43.428919 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.428889 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/333b6b05-370e-43a8-b8b9-3378565c68ed-kube-api-access-7sqm6\" (UniqueName: \"kubernetes.io/projected/333b6b05-370e-43a8-b8b9-3378565c68ed-kube-api-access-7sqm6\") pod \"333b6b05-370e-43a8-b8b9-3378565c68ed\" (UID: \"333b6b05-370e-43a8-b8b9-3378565c68ed\") " Nov 02 18:50:43.429246 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.429215 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333b6b05-370e-43a8-b8b9-3378565c68ed-plugins-config-volume" pod "333b6b05-370e-43a8-b8b9-3378565c68ed" (UID: "333b6b05-370e-43a8-b8b9-3378565c68ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:50:43.429393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.429264 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6488b4c6-7b8b-4903-a1ba-89b687c503a8-plugins-config-volume" pod "6488b4c6-7b8b-4903-a1ba-89b687c503a8" (UID: "6488b4c6-7b8b-4903-a1ba-89b687c503a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:50:43.429393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.429344 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333b6b05-370e-43a8-b8b9-3378565c68ed-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:50:43.429393 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.429363 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6488b4c6-7b8b-4903-a1ba-89b687c503a8-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:50:43.431140 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.431116 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6488b4c6-7b8b-4903-a1ba-89b687c503a8-kube-api-access-z84lf" pod "6488b4c6-7b8b-4903-a1ba-89b687c503a8" (UID: "6488b4c6-7b8b-4903-a1ba-89b687c503a8"). InnerVolumeSpecName "kube-api-access-z84lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:50:43.431228 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.431155 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333b6b05-370e-43a8-b8b9-3378565c68ed-kube-api-access-7sqm6" pod "333b6b05-370e-43a8-b8b9-3378565c68ed" (UID: "333b6b05-370e-43a8-b8b9-3378565c68ed"). InnerVolumeSpecName "kube-api-access-7sqm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:50:43.530043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.529999 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z84lf\" (UniqueName: \"kubernetes.io/projected/6488b4c6-7b8b-4903-a1ba-89b687c503a8-kube-api-access-z84lf\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:50:43.530043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.530035 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sqm6\" (UniqueName: \"kubernetes.io/projected/333b6b05-370e-43a8-b8b9-3378565c68ed-kube-api-access-7sqm6\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:50:43.574264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.574176 5549 generic.go:356] "Generic (PLEG): container finished" podID="6488b4c6-7b8b-4903-a1ba-89b687c503a8" containerID="2c5f4a6751ec9f72740cca1d7223e514809c03201dfd9bf247ed886e746aeafb" exitCode=0 Nov 02 18:50:43.574264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.574225 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" Nov 02 18:50:43.574264 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.574256 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" event={"ID":"6488b4c6-7b8b-4903-a1ba-89b687c503a8","Type":"ContainerDied","Data":"2c5f4a6751ec9f72740cca1d7223e514809c03201dfd9bf247ed886e746aeafb"} Nov 02 18:50:43.574547 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.574304 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf" event={"ID":"6488b4c6-7b8b-4903-a1ba-89b687c503a8","Type":"ContainerDied","Data":"37ed3f2229d32926389964681e3613055b1a89dd323f2694b68a73d79fb1fc94"} Nov 02 18:50:43.574547 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.574331 5549 scope.go:122] "RemoveContainer" containerID="2c5f4a6751ec9f72740cca1d7223e514809c03201dfd9bf247ed886e746aeafb" Nov 02 18:50:43.575754 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.575729 5549 generic.go:356] "Generic (PLEG): container finished" podID="333b6b05-370e-43a8-b8b9-3378565c68ed" containerID="8507be70721581124632ccccc9d6fa683fd079213f476b6c24a3f6ee69b715da" exitCode=0 Nov 02 18:50:43.575917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.575779 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" Nov 02 18:50:43.575917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.575829 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" event={"ID":"333b6b05-370e-43a8-b8b9-3378565c68ed","Type":"ContainerDied","Data":"8507be70721581124632ccccc9d6fa683fd079213f476b6c24a3f6ee69b715da"} Nov 02 18:50:43.575917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.575880 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s" event={"ID":"333b6b05-370e-43a8-b8b9-3378565c68ed","Type":"ContainerDied","Data":"ee831133f2ae718fc42580d074be5986d19b015764efd98aac17eaf9c991ae69"} Nov 02 18:50:43.591822 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.587113 5549 scope.go:122] "RemoveContainer" containerID="2c5f4a6751ec9f72740cca1d7223e514809c03201dfd9bf247ed886e746aeafb" Nov 02 18:50:43.591822 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:50:43.590539 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5f4a6751ec9f72740cca1d7223e514809c03201dfd9bf247ed886e746aeafb\": container with ID starting with 2c5f4a6751ec9f72740cca1d7223e514809c03201dfd9bf247ed886e746aeafb not found: ID does not exist" containerID="2c5f4a6751ec9f72740cca1d7223e514809c03201dfd9bf247ed886e746aeafb" Nov 02 18:50:43.591822 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.590590 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5f4a6751ec9f72740cca1d7223e514809c03201dfd9bf247ed886e746aeafb"} err="failed to get container status \"2c5f4a6751ec9f72740cca1d7223e514809c03201dfd9bf247ed886e746aeafb\": rpc error: code = NotFound desc = could not find container \"2c5f4a6751ec9f72740cca1d7223e514809c03201dfd9bf247ed886e746aeafb\": container with ID starting with 2c5f4a6751ec9f72740cca1d7223e514809c03201dfd9bf247ed886e746aeafb not found: ID does not exist" Nov 02 18:50:43.591822 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.590618 5549 scope.go:122] "RemoveContainer" containerID="8507be70721581124632ccccc9d6fa683fd079213f476b6c24a3f6ee69b715da" Nov 02 18:50:43.600408 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.600375 5549 scope.go:122] "RemoveContainer" containerID="8507be70721581124632ccccc9d6fa683fd079213f476b6c24a3f6ee69b715da" Nov 02 18:50:43.600682 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:50:43.600659 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8507be70721581124632ccccc9d6fa683fd079213f476b6c24a3f6ee69b715da\": container with ID starting with 8507be70721581124632ccccc9d6fa683fd079213f476b6c24a3f6ee69b715da not found: ID does not exist" containerID="8507be70721581124632ccccc9d6fa683fd079213f476b6c24a3f6ee69b715da" Nov 02 18:50:43.600730 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.600690 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8507be70721581124632ccccc9d6fa683fd079213f476b6c24a3f6ee69b715da"} err="failed to get container status \"8507be70721581124632ccccc9d6fa683fd079213f476b6c24a3f6ee69b715da\": rpc error: code = NotFound desc = could not find container \"8507be70721581124632ccccc9d6fa683fd079213f476b6c24a3f6ee69b715da\": container with ID starting with 8507be70721581124632ccccc9d6fa683fd079213f476b6c24a3f6ee69b715da not found: ID does not exist" Nov 02 18:50:43.608287 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.608264 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s"] Nov 02 18:50:43.614654 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.614630 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885d7sx2s"] Nov 02 18:50:43.624300 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.624276 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf"] Nov 02 18:50:43.626621 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:43.626598 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccczvxf"] Nov 02 18:50:44.320966 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:44.320919 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="333b6b05-370e-43a8-b8b9-3378565c68ed" path="/var/lib/kubelet/pods/333b6b05-370e-43a8-b8b9-3378565c68ed/volumes" Nov 02 18:50:44.321366 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:44.321351 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6488b4c6-7b8b-4903-a1ba-89b687c503a8" path="/var/lib/kubelet/pods/6488b4c6-7b8b-4903-a1ba-89b687c503a8/volumes" Nov 02 18:50:47.585819 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:50:47.585780 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:50:49.288237 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:49.288200 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:49.420065 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:49.420031 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:50.360437 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.360400 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp"] Nov 02 18:50:50.360859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.360808 5549 cpu_manager.go:414] "containerMap: removing container" podUID="6488b4c6-7b8b-4903-a1ba-89b687c503a8" containerName="payload-processing" Nov 02 18:50:50.360859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.360819 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6488b4c6-7b8b-4903-a1ba-89b687c503a8" containerName="payload-processing" Nov 02 18:50:50.360859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.360837 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7f1ad6b7-09da-431d-8a78-90ffe66346d5" containerName="maas-api" Nov 02 18:50:50.360859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.360844 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f1ad6b7-09da-431d-8a78-90ffe66346d5" containerName="maas-api" Nov 02 18:50:50.360859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.360851 5549 cpu_manager.go:414] "containerMap: removing container" podUID="0c9524e2-3937-45cc-8643-327d0b112e34" containerName="main" Nov 02 18:50:50.360859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.360856 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9524e2-3937-45cc-8643-327d0b112e34" containerName="main" Nov 02 18:50:50.361122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.360874 5549 cpu_manager.go:414] "containerMap: removing container" podUID="333b6b05-370e-43a8-b8b9-3378565c68ed" containerName="payload-pre-processing" Nov 02 18:50:50.361122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.360880 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="333b6b05-370e-43a8-b8b9-3378565c68ed" containerName="payload-pre-processing" Nov 02 18:50:50.361122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.360943 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6488b4c6-7b8b-4903-a1ba-89b687c503a8" containerName="payload-processing" Nov 02 18:50:50.361122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.360971 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f1ad6b7-09da-431d-8a78-90ffe66346d5" containerName="maas-api" Nov 02 18:50:50.361122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.360977 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9524e2-3937-45cc-8643-327d0b112e34" containerName="main" Nov 02 18:50:50.361122 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.360987 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="333b6b05-370e-43a8-b8b9-3378565c68ed" containerName="payload-pre-processing" Nov 02 18:50:50.365687 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.365662 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" Nov 02 18:50:50.375444 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.375411 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp"] Nov 02 18:50:50.375792 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.375767 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-3bd024-serving-cert\"" Nov 02 18:50:50.463798 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.463749 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:50.484925 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.484889 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h"] Nov 02 18:50:50.488565 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.488541 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb"] Nov 02 18:50:50.488705 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.488654 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h" Nov 02 18:50:50.491835 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.491818 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb" Nov 02 18:50:50.497138 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.497116 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-3bd024\"" Nov 02 18:50:50.503078 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.503041 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h"] Nov 02 18:50:50.503275 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.503254 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-c47aa932-dockercfg-rwb2t\"" Nov 02 18:50:50.503450 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.503424 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb"] Nov 02 18:50:50.504019 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.503985 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576b906e-d10a-4f68-8f2d-8034c076210c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp\" (UID: \"576b906e-d10a-4f68-8f2d-8034c076210c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" Nov 02 18:50:50.504244 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.504223 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/576b906e-d10a-4f68-8f2d-8034c076210c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp\" (UID: \"576b906e-d10a-4f68-8f2d-8034c076210c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" Nov 02 18:50:50.504324 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.504258 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/576b906e-d10a-4f68-8f2d-8034c076210c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp\" (UID: \"576b906e-d10a-4f68-8f2d-8034c076210c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" Nov 02 18:50:50.504388 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.504326 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kxgm\" (UniqueName: \"kubernetes.io/projected/576b906e-d10a-4f68-8f2d-8034c076210c-kube-api-access-5kxgm\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp\" (UID: \"576b906e-d10a-4f68-8f2d-8034c076210c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" Nov 02 18:50:50.605911 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.605872 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkpb5\" (UniqueName: \"kubernetes.io/projected/2f486ee5-9aad-4e65-8d32-d0e6997b3ec6-kube-api-access-nkpb5\") pod \"payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h\" (UID: \"2f486ee5-9aad-4e65-8d32-d0e6997b3ec6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h" Nov 02 18:50:50.606112 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.605942 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf55j\" (UniqueName: \"kubernetes.io/projected/31d32da2-d6a3-4fb5-b7e6-7e3773eabe66-kube-api-access-cf55j\") pod \"payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb\" (UID: \"31d32da2-d6a3-4fb5-b7e6-7e3773eabe66\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb" Nov 02 18:50:50.606112 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.606065 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d32da2-d6a3-4fb5-b7e6-7e3773eabe66-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb\" (UID: \"31d32da2-d6a3-4fb5-b7e6-7e3773eabe66\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb" Nov 02 18:50:50.606112 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.606093 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/576b906e-d10a-4f68-8f2d-8034c076210c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp\" (UID: \"576b906e-d10a-4f68-8f2d-8034c076210c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" Nov 02 18:50:50.606231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.606113 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/576b906e-d10a-4f68-8f2d-8034c076210c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp\" (UID: \"576b906e-d10a-4f68-8f2d-8034c076210c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" Nov 02 18:50:50.606231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.606186 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kxgm\" (UniqueName: \"kubernetes.io/projected/576b906e-d10a-4f68-8f2d-8034c076210c-kube-api-access-5kxgm\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp\" (UID: \"576b906e-d10a-4f68-8f2d-8034c076210c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" Nov 02 18:50:50.606316 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.606235 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f486ee5-9aad-4e65-8d32-d0e6997b3ec6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h\" (UID: \"2f486ee5-9aad-4e65-8d32-d0e6997b3ec6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h" Nov 02 18:50:50.606316 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.606279 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576b906e-d10a-4f68-8f2d-8034c076210c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp\" (UID: \"576b906e-d10a-4f68-8f2d-8034c076210c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" Nov 02 18:50:50.607129 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.607105 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/576b906e-d10a-4f68-8f2d-8034c076210c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp\" (UID: \"576b906e-d10a-4f68-8f2d-8034c076210c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" Nov 02 18:50:50.608999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.608971 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:50.610568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.610509 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576b906e-d10a-4f68-8f2d-8034c076210c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp\" (UID: \"576b906e-d10a-4f68-8f2d-8034c076210c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" Nov 02 18:50:50.610568 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.610509 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/576b906e-d10a-4f68-8f2d-8034c076210c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp\" (UID: \"576b906e-d10a-4f68-8f2d-8034c076210c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" Nov 02 18:50:50.615257 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.615227 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kxgm\" (UniqueName: \"kubernetes.io/projected/576b906e-d10a-4f68-8f2d-8034c076210c-kube-api-access-5kxgm\") pod \"maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp\" (UID: \"576b906e-d10a-4f68-8f2d-8034c076210c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" Nov 02 18:50:50.684545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.684506 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" Nov 02 18:50:50.707327 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.707284 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f486ee5-9aad-4e65-8d32-d0e6997b3ec6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h\" (UID: \"2f486ee5-9aad-4e65-8d32-d0e6997b3ec6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h" Nov 02 18:50:50.707466 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.707433 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkpb5\" (UniqueName: \"kubernetes.io/projected/2f486ee5-9aad-4e65-8d32-d0e6997b3ec6-kube-api-access-nkpb5\") pod \"payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h\" (UID: \"2f486ee5-9aad-4e65-8d32-d0e6997b3ec6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h" Nov 02 18:50:50.707551 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.707530 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf55j\" (UniqueName: \"kubernetes.io/projected/31d32da2-d6a3-4fb5-b7e6-7e3773eabe66-kube-api-access-cf55j\") pod \"payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb\" (UID: \"31d32da2-d6a3-4fb5-b7e6-7e3773eabe66\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb" Nov 02 18:50:50.707615 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.707569 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d32da2-d6a3-4fb5-b7e6-7e3773eabe66-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb\" (UID: \"31d32da2-d6a3-4fb5-b7e6-7e3773eabe66\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb" Nov 02 18:50:50.708331 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.708304 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f486ee5-9aad-4e65-8d32-d0e6997b3ec6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h\" (UID: \"2f486ee5-9aad-4e65-8d32-d0e6997b3ec6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h" Nov 02 18:50:50.708436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.708392 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d32da2-d6a3-4fb5-b7e6-7e3773eabe66-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb\" (UID: \"31d32da2-d6a3-4fb5-b7e6-7e3773eabe66\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb" Nov 02 18:50:50.715471 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.715447 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf55j\" (UniqueName: \"kubernetes.io/projected/31d32da2-d6a3-4fb5-b7e6-7e3773eabe66-kube-api-access-cf55j\") pod \"payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb\" (UID: \"31d32da2-d6a3-4fb5-b7e6-7e3773eabe66\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb" Nov 02 18:50:50.715593 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.715565 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkpb5\" (UniqueName: \"kubernetes.io/projected/2f486ee5-9aad-4e65-8d32-d0e6997b3ec6-kube-api-access-nkpb5\") pod \"payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h\" (UID: \"2f486ee5-9aad-4e65-8d32-d0e6997b3ec6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h" Nov 02 18:50:50.803977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.803923 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h" Nov 02 18:50:50.814307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.814279 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb" Nov 02 18:50:50.961929 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:50.961817 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp"] Nov 02 18:50:50.964830 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:50:50.964802 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576b906e_d10a_4f68_8f2d_8034c076210c.slice/crio-c18017c6364950b86cfd8da5dda2bb0c206c1564ec6cfb6008acd44a75552a21 WatchSource:0}: Error finding container c18017c6364950b86cfd8da5dda2bb0c206c1564ec6cfb6008acd44a75552a21: Status 404 returned error can't find the container with id c18017c6364950b86cfd8da5dda2bb0c206c1564ec6cfb6008acd44a75552a21 Nov 02 18:50:51.080055 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:51.080027 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h"] Nov 02 18:50:51.082783 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:50:51.082752 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f486ee5_9aad_4e65_8d32_d0e6997b3ec6.slice/crio-1316b87bd59b2da5cbdbbae3e166c300c41e311982e3d97fdec639f4d868715d WatchSource:0}: Error finding container 1316b87bd59b2da5cbdbbae3e166c300c41e311982e3d97fdec639f4d868715d: Status 404 returned error can't find the container with id 1316b87bd59b2da5cbdbbae3e166c300c41e311982e3d97fdec639f4d868715d Nov 02 18:50:51.099884 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:51.099848 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb"] Nov 02 18:50:51.270452 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:51.270413 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:51.619868 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:51.619815 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" event={"ID":"576b906e-d10a-4f68-8f2d-8034c076210c","Type":"ContainerStarted","Data":"6581642ab18939ac53427fab7d8f41bac6e37e8f2616c27ef72a43ee47bf860a"} Nov 02 18:50:51.620367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:51.619887 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" event={"ID":"576b906e-d10a-4f68-8f2d-8034c076210c","Type":"ContainerStarted","Data":"c18017c6364950b86cfd8da5dda2bb0c206c1564ec6cfb6008acd44a75552a21"} Nov 02 18:50:51.620367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:51.620052 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" Nov 02 18:50:51.621460 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:51.621437 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h" event={"ID":"2f486ee5-9aad-4e65-8d32-d0e6997b3ec6","Type":"ContainerStarted","Data":"f99193cca26d49cc69bb5ac970bf9e2a3817722946d4892c7dc3780f47fff362"} Nov 02 18:50:51.621552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:51.621475 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h" event={"ID":"2f486ee5-9aad-4e65-8d32-d0e6997b3ec6","Type":"ContainerStarted","Data":"1316b87bd59b2da5cbdbbae3e166c300c41e311982e3d97fdec639f4d868715d"} Nov 02 18:50:51.621552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:51.621505 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h" Nov 02 18:50:51.623086 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:51.623061 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb" event={"ID":"31d32da2-d6a3-4fb5-b7e6-7e3773eabe66","Type":"ContainerStarted","Data":"b78fc77971d9eb74ddcdf5181544c22062fa1b2073581ba852c804cf08766758"} Nov 02 18:50:51.623200 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:51.623103 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb" event={"ID":"31d32da2-d6a3-4fb5-b7e6-7e3773eabe66","Type":"ContainerStarted","Data":"1d1a293800b4d63de692e85be45da1e5eab6f63e39b1f548162ba168fbf6402a"} Nov 02 18:50:51.623200 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:51.623165 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb" Nov 02 18:50:51.636675 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:51.636624 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" podStartSLOduration=1.107057949 podStartE2EDuration="1.636609683s" totalImagesPullingTime="529.551734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:50:51.636072435 +0000 UTC m=+4623.424577963" watchObservedRunningTime="2026-11-02 18:50:51.636609683 +0000 UTC m=+4623.425115211" Nov 02 18:50:51.656717 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:51.656664 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h" podStartSLOduration=1.656649379 podStartE2EDuration="1.656649379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:50:51.655321519 +0000 UTC m=+4623.443827047" watchObservedRunningTime="2026-11-02 18:50:51.656649379 +0000 UTC m=+4623.445154906" Nov 02 18:50:51.672863 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:51.672810 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb" podStartSLOduration=1.672799819 podStartE2EDuration="1.672799819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:50:51.67141396 +0000 UTC m=+4623.459919489" watchObservedRunningTime="2026-11-02 18:50:51.672799819 +0000 UTC m=+4623.461305369" Nov 02 18:50:51.931995 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:51.931871 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:52.475120 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:52.475084 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:50:52.629755 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:52.629714 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:53.356162 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:53.356118 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:53.559064 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:53.559036 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:54.163407 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:54.163312 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:54.486767 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:54.486729 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:54.908999 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:54.908904 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:55.270368 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:55.270330 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:50:55.853681 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:55.853642 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:56.657297 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:56.657262 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:57.255134 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:57.255089 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:57.637225 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:57.637144 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" Nov 02 18:50:57.873453 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:57.873418 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:58.564072 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:58.564032 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:59.170635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:59.170599 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:50:59.765613 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:50:59.765572 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:00.452292 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:00.452250 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:00.682110 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:00.682071 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:00.835859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:00.835765 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:01.083033 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:01.082347 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:01.748830 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:01.748795 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:02.361093 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:02.361054 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:02.635229 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:02.635142 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h" Nov 02 18:51:02.635229 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:02.635211 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb" Nov 02 18:51:03.034857 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:03.034761 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:03.647142 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:03.647104 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:04.139288 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:04.139254 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:04.263161 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:04.263129 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:04.857498 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:04.857435 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:05.560484 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:05.560453 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:06.192233 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:06.192195 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:06.763452 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:06.763367 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:07.353120 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:07.353083 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:08.041762 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:08.041725 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:08.421198 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:08.421088 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb"] Nov 02 18:51:08.433603 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:08.433568 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h"] Nov 02 18:51:08.531932 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:08.531897 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:08.661208 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:08.661171 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:08.794560 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:08.794524 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:09.107280 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.107180 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp"] Nov 02 18:51:09.107768 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.107448 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" podUID="576b906e-d10a-4f68-8f2d-8034c076210c" containerName="maas-api" containerID="cri-o://6581642ab18939ac53427fab7d8f41bac6e37e8f2616c27ef72a43ee47bf860a" gracePeriod=30 Nov 02 18:51:09.193454 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.193401 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:09.326772 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.326731 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:09.375091 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.374993 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" Nov 02 18:51:09.455864 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.455833 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:09.520779 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.520739 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/576b906e-d10a-4f68-8f2d-8034c076210c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/576b906e-d10a-4f68-8f2d-8034c076210c-metrics-certs\") pod \"576b906e-d10a-4f68-8f2d-8034c076210c\" (UID: \"576b906e-d10a-4f68-8f2d-8034c076210c\") " Nov 02 18:51:09.521017 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.520804 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/576b906e-d10a-4f68-8f2d-8034c076210c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/576b906e-d10a-4f68-8f2d-8034c076210c-openshift-service-ca\") pod \"576b906e-d10a-4f68-8f2d-8034c076210c\" (UID: \"576b906e-d10a-4f68-8f2d-8034c076210c\") " Nov 02 18:51:09.521017 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.520899 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/576b906e-d10a-4f68-8f2d-8034c076210c-kube-api-access-5kxgm\" (UniqueName: \"kubernetes.io/projected/576b906e-d10a-4f68-8f2d-8034c076210c-kube-api-access-5kxgm\") pod \"576b906e-d10a-4f68-8f2d-8034c076210c\" (UID: \"576b906e-d10a-4f68-8f2d-8034c076210c\") " Nov 02 18:51:09.521017 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.520985 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/576b906e-d10a-4f68-8f2d-8034c076210c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/576b906e-d10a-4f68-8f2d-8034c076210c-maas-api-tls\") pod \"576b906e-d10a-4f68-8f2d-8034c076210c\" (UID: \"576b906e-d10a-4f68-8f2d-8034c076210c\") " Nov 02 18:51:09.521241 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.521223 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/576b906e-d10a-4f68-8f2d-8034c076210c-openshift-service-ca" pod "576b906e-d10a-4f68-8f2d-8034c076210c" (UID: "576b906e-d10a-4f68-8f2d-8034c076210c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:51:09.521416 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.521396 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/576b906e-d10a-4f68-8f2d-8034c076210c-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:51:09.522989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.522929 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/576b906e-d10a-4f68-8f2d-8034c076210c-metrics-certs" pod "576b906e-d10a-4f68-8f2d-8034c076210c" (UID: "576b906e-d10a-4f68-8f2d-8034c076210c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:51:09.523109 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.523037 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/576b906e-d10a-4f68-8f2d-8034c076210c-maas-api-tls" pod "576b906e-d10a-4f68-8f2d-8034c076210c" (UID: "576b906e-d10a-4f68-8f2d-8034c076210c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:51:09.523184 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.523160 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576b906e-d10a-4f68-8f2d-8034c076210c-kube-api-access-5kxgm" pod "576b906e-d10a-4f68-8f2d-8034c076210c" (UID: "576b906e-d10a-4f68-8f2d-8034c076210c"). InnerVolumeSpecName "kube-api-access-5kxgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:51:09.622007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.621935 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576b906e-d10a-4f68-8f2d-8034c076210c-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:51:09.622007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.622003 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kxgm\" (UniqueName: \"kubernetes.io/projected/576b906e-d10a-4f68-8f2d-8034c076210c-kube-api-access-5kxgm\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:51:09.622007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.622016 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/576b906e-d10a-4f68-8f2d-8034c076210c-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:51:09.664592 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.664497 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd"] Nov 02 18:51:09.664759 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.664694 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" podUID="7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" containerName="istio-proxy" containerID="cri-o://53a7fcad8968274fa076785fa303cdd9b29c3a4f6c3da8f2a037bce7c11191f5" gracePeriod=30 Nov 02 18:51:09.720094 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.720056 5549 generic.go:356] "Generic (PLEG): container finished" podID="576b906e-d10a-4f68-8f2d-8034c076210c" containerID="6581642ab18939ac53427fab7d8f41bac6e37e8f2616c27ef72a43ee47bf860a" exitCode=0 Nov 02 18:51:09.720307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.720106 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" Nov 02 18:51:09.720307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.720121 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" event={"ID":"576b906e-d10a-4f68-8f2d-8034c076210c","Type":"ContainerDied","Data":"6581642ab18939ac53427fab7d8f41bac6e37e8f2616c27ef72a43ee47bf860a"} Nov 02 18:51:09.720307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.720190 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp" event={"ID":"576b906e-d10a-4f68-8f2d-8034c076210c","Type":"ContainerDied","Data":"c18017c6364950b86cfd8da5dda2bb0c206c1564ec6cfb6008acd44a75552a21"} Nov 02 18:51:09.720307 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.720220 5549 scope.go:122] "RemoveContainer" containerID="6581642ab18939ac53427fab7d8f41bac6e37e8f2616c27ef72a43ee47bf860a" Nov 02 18:51:09.730439 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.730409 5549 scope.go:122] "RemoveContainer" containerID="6581642ab18939ac53427fab7d8f41bac6e37e8f2616c27ef72a43ee47bf860a" Nov 02 18:51:09.730749 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:51:09.730724 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6581642ab18939ac53427fab7d8f41bac6e37e8f2616c27ef72a43ee47bf860a\": container with ID starting with 6581642ab18939ac53427fab7d8f41bac6e37e8f2616c27ef72a43ee47bf860a not found: ID does not exist" containerID="6581642ab18939ac53427fab7d8f41bac6e37e8f2616c27ef72a43ee47bf860a" Nov 02 18:51:09.730805 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.730759 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6581642ab18939ac53427fab7d8f41bac6e37e8f2616c27ef72a43ee47bf860a"} err="failed to get container status \"6581642ab18939ac53427fab7d8f41bac6e37e8f2616c27ef72a43ee47bf860a\": rpc error: code = NotFound desc = could not find container \"6581642ab18939ac53427fab7d8f41bac6e37e8f2616c27ef72a43ee47bf860a\": container with ID starting with 6581642ab18939ac53427fab7d8f41bac6e37e8f2616c27ef72a43ee47bf860a not found: ID does not exist" Nov 02 18:51:09.741465 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.741420 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:51:09.741987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.741935 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp"] Nov 02 18:51:09.744425 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:09.744401 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bd024-88798d9f9-fxjcp"] Nov 02 18:51:10.318680 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:10.318647 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="576b906e-d10a-4f68-8f2d-8034c076210c" path="/var/lib/kubelet/pods/576b906e-d10a-4f68-8f2d-8034c076210c/volumes" Nov 02 18:51:10.562938 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:10.562897 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:51:13.422108 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.422056 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb" podUID="31d32da2-d6a3-4fb5-b7e6-7e3773eabe66" containerName="payload-processing" containerID="cri-o://b78fc77971d9eb74ddcdf5181544c22062fa1b2073581ba852c804cf08766758" gracePeriod=25 Nov 02 18:51:13.434107 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.434069 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h" podUID="2f486ee5-9aad-4e65-8d32-d0e6997b3ec6" containerName="payload-pre-processing" containerID="cri-o://f99193cca26d49cc69bb5ac970bf9e2a3817722946d4892c7dc3780f47fff362" gracePeriod=25 Nov 02 18:51:13.675287 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.675222 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb" Nov 02 18:51:13.727359 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.727333 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h" Nov 02 18:51:13.740740 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.740711 5549 generic.go:356] "Generic (PLEG): container finished" podID="2f486ee5-9aad-4e65-8d32-d0e6997b3ec6" containerID="f99193cca26d49cc69bb5ac970bf9e2a3817722946d4892c7dc3780f47fff362" exitCode=0 Nov 02 18:51:13.740879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.740751 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h" Nov 02 18:51:13.740879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.740794 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h" event={"ID":"2f486ee5-9aad-4e65-8d32-d0e6997b3ec6","Type":"ContainerDied","Data":"f99193cca26d49cc69bb5ac970bf9e2a3817722946d4892c7dc3780f47fff362"} Nov 02 18:51:13.740879 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.740862 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h" event={"ID":"2f486ee5-9aad-4e65-8d32-d0e6997b3ec6","Type":"ContainerDied","Data":"1316b87bd59b2da5cbdbbae3e166c300c41e311982e3d97fdec639f4d868715d"} Nov 02 18:51:13.741072 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.740884 5549 scope.go:122] "RemoveContainer" containerID="f99193cca26d49cc69bb5ac970bf9e2a3817722946d4892c7dc3780f47fff362" Nov 02 18:51:13.742289 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.742221 5549 generic.go:356] "Generic (PLEG): container finished" podID="31d32da2-d6a3-4fb5-b7e6-7e3773eabe66" containerID="b78fc77971d9eb74ddcdf5181544c22062fa1b2073581ba852c804cf08766758" exitCode=0 Nov 02 18:51:13.742400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.742326 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb" Nov 02 18:51:13.742400 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.742355 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb" event={"ID":"31d32da2-d6a3-4fb5-b7e6-7e3773eabe66","Type":"ContainerDied","Data":"b78fc77971d9eb74ddcdf5181544c22062fa1b2073581ba852c804cf08766758"} Nov 02 18:51:13.742532 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.742432 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb" event={"ID":"31d32da2-d6a3-4fb5-b7e6-7e3773eabe66","Type":"ContainerDied","Data":"1d1a293800b4d63de692e85be45da1e5eab6f63e39b1f548162ba168fbf6402a"} Nov 02 18:51:13.751334 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.751314 5549 scope.go:122] "RemoveContainer" containerID="f99193cca26d49cc69bb5ac970bf9e2a3817722946d4892c7dc3780f47fff362" Nov 02 18:51:13.751628 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:51:13.751609 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f99193cca26d49cc69bb5ac970bf9e2a3817722946d4892c7dc3780f47fff362\": container with ID starting with f99193cca26d49cc69bb5ac970bf9e2a3817722946d4892c7dc3780f47fff362 not found: ID does not exist" containerID="f99193cca26d49cc69bb5ac970bf9e2a3817722946d4892c7dc3780f47fff362" Nov 02 18:51:13.751736 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.751635 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f99193cca26d49cc69bb5ac970bf9e2a3817722946d4892c7dc3780f47fff362"} err="failed to get container status \"f99193cca26d49cc69bb5ac970bf9e2a3817722946d4892c7dc3780f47fff362\": rpc error: code = NotFound desc = could not find container \"f99193cca26d49cc69bb5ac970bf9e2a3817722946d4892c7dc3780f47fff362\": container with ID starting with f99193cca26d49cc69bb5ac970bf9e2a3817722946d4892c7dc3780f47fff362 not found: ID does not exist" Nov 02 18:51:13.751736 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.751651 5549 scope.go:122] "RemoveContainer" containerID="b78fc77971d9eb74ddcdf5181544c22062fa1b2073581ba852c804cf08766758" Nov 02 18:51:13.760926 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.760902 5549 scope.go:122] "RemoveContainer" containerID="b78fc77971d9eb74ddcdf5181544c22062fa1b2073581ba852c804cf08766758" Nov 02 18:51:13.761273 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:51:13.761248 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78fc77971d9eb74ddcdf5181544c22062fa1b2073581ba852c804cf08766758\": container with ID starting with b78fc77971d9eb74ddcdf5181544c22062fa1b2073581ba852c804cf08766758 not found: ID does not exist" containerID="b78fc77971d9eb74ddcdf5181544c22062fa1b2073581ba852c804cf08766758" Nov 02 18:51:13.761348 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.761283 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78fc77971d9eb74ddcdf5181544c22062fa1b2073581ba852c804cf08766758"} err="failed to get container status \"b78fc77971d9eb74ddcdf5181544c22062fa1b2073581ba852c804cf08766758\": rpc error: code = NotFound desc = could not find container \"b78fc77971d9eb74ddcdf5181544c22062fa1b2073581ba852c804cf08766758\": container with ID starting with b78fc77971d9eb74ddcdf5181544c22062fa1b2073581ba852c804cf08766758 not found: ID does not exist" Nov 02 18:51:13.763692 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.763665 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d32da2-d6a3-4fb5-b7e6-7e3773eabe66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d32da2-d6a3-4fb5-b7e6-7e3773eabe66-plugins-config-volume\") pod \"31d32da2-d6a3-4fb5-b7e6-7e3773eabe66\" (UID: \"31d32da2-d6a3-4fb5-b7e6-7e3773eabe66\") " Nov 02 18:51:13.763853 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.763838 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d32da2-d6a3-4fb5-b7e6-7e3773eabe66-kube-api-access-cf55j\" (UniqueName: \"kubernetes.io/projected/31d32da2-d6a3-4fb5-b7e6-7e3773eabe66-kube-api-access-cf55j\") pod \"31d32da2-d6a3-4fb5-b7e6-7e3773eabe66\" (UID: \"31d32da2-d6a3-4fb5-b7e6-7e3773eabe66\") " Nov 02 18:51:13.764112 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.764086 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d32da2-d6a3-4fb5-b7e6-7e3773eabe66-plugins-config-volume" pod "31d32da2-d6a3-4fb5-b7e6-7e3773eabe66" (UID: "31d32da2-d6a3-4fb5-b7e6-7e3773eabe66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:51:13.765930 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.765908 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d32da2-d6a3-4fb5-b7e6-7e3773eabe66-kube-api-access-cf55j" pod "31d32da2-d6a3-4fb5-b7e6-7e3773eabe66" (UID: "31d32da2-d6a3-4fb5-b7e6-7e3773eabe66"). InnerVolumeSpecName "kube-api-access-cf55j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:51:13.864755 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.864716 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f486ee5-9aad-4e65-8d32-d0e6997b3ec6-kube-api-access-nkpb5\" (UniqueName: \"kubernetes.io/projected/2f486ee5-9aad-4e65-8d32-d0e6997b3ec6-kube-api-access-nkpb5\") pod \"2f486ee5-9aad-4e65-8d32-d0e6997b3ec6\" (UID: \"2f486ee5-9aad-4e65-8d32-d0e6997b3ec6\") " Nov 02 18:51:13.864926 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.864772 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f486ee5-9aad-4e65-8d32-d0e6997b3ec6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f486ee5-9aad-4e65-8d32-d0e6997b3ec6-plugins-config-volume\") pod \"2f486ee5-9aad-4e65-8d32-d0e6997b3ec6\" (UID: \"2f486ee5-9aad-4e65-8d32-d0e6997b3ec6\") " Nov 02 18:51:13.865037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.865022 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d32da2-d6a3-4fb5-b7e6-7e3773eabe66-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:51:13.865083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.865042 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf55j\" (UniqueName: \"kubernetes.io/projected/31d32da2-d6a3-4fb5-b7e6-7e3773eabe66-kube-api-access-cf55j\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:51:13.865193 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.865170 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f486ee5-9aad-4e65-8d32-d0e6997b3ec6-plugins-config-volume" pod "2f486ee5-9aad-4e65-8d32-d0e6997b3ec6" (UID: "2f486ee5-9aad-4e65-8d32-d0e6997b3ec6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:51:13.866849 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.866830 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f486ee5-9aad-4e65-8d32-d0e6997b3ec6-kube-api-access-nkpb5" pod "2f486ee5-9aad-4e65-8d32-d0e6997b3ec6" (UID: "2f486ee5-9aad-4e65-8d32-d0e6997b3ec6"). InnerVolumeSpecName "kube-api-access-nkpb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:51:13.966152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.966119 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkpb5\" (UniqueName: \"kubernetes.io/projected/2f486ee5-9aad-4e65-8d32-d0e6997b3ec6-kube-api-access-nkpb5\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:51:13.966152 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:13.966150 5549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f486ee5-9aad-4e65-8d32-d0e6997b3ec6-plugins-config-volume\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:51:14.067137 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:14.067104 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb"] Nov 02 18:51:14.071316 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:14.071285 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3bd024-779dd77ccrhgpb"] Nov 02 18:51:14.081552 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:14.081524 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h"] Nov 02 18:51:14.085481 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:14.085458 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bd024-7885dfkt9h"] Nov 02 18:51:14.318107 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:14.318016 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f486ee5-9aad-4e65-8d32-d0e6997b3ec6" path="/var/lib/kubelet/pods/2f486ee5-9aad-4e65-8d32-d0e6997b3ec6/volumes" Nov 02 18:51:14.318522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:14.318503 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31d32da2-d6a3-4fb5-b7e6-7e3773eabe66" path="/var/lib/kubelet/pods/31d32da2-d6a3-4fb5-b7e6-7e3773eabe66/volumes" Nov 02 18:51:14.923368 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:14.923340 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:51:15.079267 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.079236 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-credential-socket\") pod \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " Nov 02 18:51:15.079267 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.079274 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-kube-api-access-2zj7n\" (UniqueName: \"kubernetes.io/projected/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-kube-api-access-2zj7n\") pod \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " Nov 02 18:51:15.079555 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.079310 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istiod-ca-cert\") pod \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " Nov 02 18:51:15.079555 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.079337 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-podinfo\") pod \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " Nov 02 18:51:15.079555 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.079384 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-workload-certs\") pod \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " Nov 02 18:51:15.079555 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.079419 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-workload-socket\") pod \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " Nov 02 18:51:15.079555 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.079466 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-data\") pod \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " Nov 02 18:51:15.079555 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.079502 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-envoy\") pod \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " Nov 02 18:51:15.079555 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.079549 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-token\" (UniqueName: \"kubernetes.io/projected/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-token\") pod \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\" (UID: \"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd\") " Nov 02 18:51:15.079941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.079599 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-credential-socket" pod "7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" (UID: "7bbbd9f3-ec69-414d-9f23-5c38134c8bfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:51:15.079941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.079776 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-workload-certs" pod "7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" (UID: "7bbbd9f3-ec69-414d-9f23-5c38134c8bfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:51:15.079941 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.079814 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-data" pod "7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" (UID: "7bbbd9f3-ec69-414d-9f23-5c38134c8bfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:51:15.080166 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.080015 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istiod-ca-cert" pod "7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" (UID: "7bbbd9f3-ec69-414d-9f23-5c38134c8bfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:51:15.080166 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.080065 5549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-data\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:51:15.080166 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.080090 5549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-credential-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:51:15.080166 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.080106 5549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-workload-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:51:15.080398 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.080327 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-workload-socket" pod "7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" (UID: "7bbbd9f3-ec69-414d-9f23-5c38134c8bfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:51:15.081714 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.081690 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-kube-api-access-2zj7n" pod "7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" (UID: "7bbbd9f3-ec69-414d-9f23-5c38134c8bfd"). InnerVolumeSpecName "kube-api-access-2zj7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:51:15.081841 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.081790 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-envoy" pod "7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" (UID: "7bbbd9f3-ec69-414d-9f23-5c38134c8bfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:51:15.082043 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.082021 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-podinfo" pod "7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" (UID: "7bbbd9f3-ec69-414d-9f23-5c38134c8bfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:51:15.082303 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.082287 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-token" pod "7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" (UID: "7bbbd9f3-ec69-414d-9f23-5c38134c8bfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:51:15.181011 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.180977 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zj7n\" (UniqueName: \"kubernetes.io/projected/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-kube-api-access-2zj7n\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:51:15.181011 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.181007 5549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istiod-ca-cert\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:51:15.181011 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.181018 5549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-podinfo\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:51:15.181247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.181028 5549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-workload-socket\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:51:15.181247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.181037 5549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-envoy\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:51:15.181247 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.181045 5549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd-istio-token\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:51:15.754746 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.754711 5549 generic.go:356] "Generic (PLEG): container finished" podID="7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" containerID="53a7fcad8968274fa076785fa303cdd9b29c3a4f6c3da8f2a037bce7c11191f5" exitCode=0 Nov 02 18:51:15.754987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.754764 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" event={"ID":"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd","Type":"ContainerDied","Data":"53a7fcad8968274fa076785fa303cdd9b29c3a4f6c3da8f2a037bce7c11191f5"} Nov 02 18:51:15.754987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.754805 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" event={"ID":"7bbbd9f3-ec69-414d-9f23-5c38134c8bfd","Type":"ContainerDied","Data":"ac002d26674d53b9bf54fc9baf2ca6964e9391556f43eeacd76cb4e426fe54a9"} Nov 02 18:51:15.754987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.754825 5549 scope.go:122] "RemoveContainer" containerID="53a7fcad8968274fa076785fa303cdd9b29c3a4f6c3da8f2a037bce7c11191f5" Nov 02 18:51:15.754987 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.754764 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd" Nov 02 18:51:15.763979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.763942 5549 scope.go:122] "RemoveContainer" containerID="1dcbdd6dfa96b76f68b81901cbd209855b0c93ce4693503cb1784ee594667e0e" Nov 02 18:51:15.771897 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.771877 5549 scope.go:122] "RemoveContainer" containerID="53a7fcad8968274fa076785fa303cdd9b29c3a4f6c3da8f2a037bce7c11191f5" Nov 02 18:51:15.772190 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:51:15.772169 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53a7fcad8968274fa076785fa303cdd9b29c3a4f6c3da8f2a037bce7c11191f5\": container with ID starting with 53a7fcad8968274fa076785fa303cdd9b29c3a4f6c3da8f2a037bce7c11191f5 not found: ID does not exist" containerID="53a7fcad8968274fa076785fa303cdd9b29c3a4f6c3da8f2a037bce7c11191f5" Nov 02 18:51:15.772243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.772201 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a7fcad8968274fa076785fa303cdd9b29c3a4f6c3da8f2a037bce7c11191f5"} err="failed to get container status \"53a7fcad8968274fa076785fa303cdd9b29c3a4f6c3da8f2a037bce7c11191f5\": rpc error: code = NotFound desc = could not find container \"53a7fcad8968274fa076785fa303cdd9b29c3a4f6c3da8f2a037bce7c11191f5\": container with ID starting with 53a7fcad8968274fa076785fa303cdd9b29c3a4f6c3da8f2a037bce7c11191f5 not found: ID does not exist" Nov 02 18:51:15.772243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.772218 5549 scope.go:122] "RemoveContainer" containerID="1dcbdd6dfa96b76f68b81901cbd209855b0c93ce4693503cb1784ee594667e0e" Nov 02 18:51:15.772475 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:51:15.772456 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dcbdd6dfa96b76f68b81901cbd209855b0c93ce4693503cb1784ee594667e0e\": container with ID starting with 1dcbdd6dfa96b76f68b81901cbd209855b0c93ce4693503cb1784ee594667e0e not found: ID does not exist" containerID="1dcbdd6dfa96b76f68b81901cbd209855b0c93ce4693503cb1784ee594667e0e" Nov 02 18:51:15.772534 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.772488 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dcbdd6dfa96b76f68b81901cbd209855b0c93ce4693503cb1784ee594667e0e"} err="failed to get container status \"1dcbdd6dfa96b76f68b81901cbd209855b0c93ce4693503cb1784ee594667e0e\": rpc error: code = NotFound desc = could not find container \"1dcbdd6dfa96b76f68b81901cbd209855b0c93ce4693503cb1784ee594667e0e\": container with ID starting with 1dcbdd6dfa96b76f68b81901cbd209855b0c93ce4693503cb1784ee594667e0e not found: ID does not exist" Nov 02 18:51:15.776805 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.776779 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd"] Nov 02 18:51:15.780335 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:15.780315 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3bd024-openshift-default-d85b56ffc-tlfbd"] Nov 02 18:51:16.318010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:16.317947 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" path="/var/lib/kubelet/pods/7bbbd9f3-ec69-414d-9f23-5c38134c8bfd/volumes" Nov 02 18:51:17.585606 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:51:17.585571 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:51:21.766864 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:21.766821 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:51:23.930986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:23.929096 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-r2rbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:51:23.930986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:23.929168 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" podUID="6a7bea79-f2bb-489d-b861-28fa66d0ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:51:23.995787 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:23.995049 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-kgx7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:51:23.995787 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:23.995118 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" podUID="c208756b-40ff-4691-aad8-9008058e4291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:51:25.747731 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:51:25.747692 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:51:47.585326 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:51:47.585287 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:52:04.199669 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:52:04.199543 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:52:17.588886 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:52:17.588841 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:52:47.586147 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:52:47.586109 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:52:50.462353 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:52:50.462303 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:52:51.275977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:52:51.275147 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xsg9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:52:51.275977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:52:51.275232 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" podUID="4740514e-6d04-468b-930f-8be299ef0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:53:03.559260 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:03.559217 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:53:04.241000 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:04.240896 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:53:07.848979 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:07.848785 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:53:10.961130 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:10.961087 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:53:17.586087 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:53:17.586042 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:53:19.660127 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:19.660094 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:53:21.056147 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.056110 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff"] Nov 02 18:53:21.056704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.056664 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" containerName="istio-proxy" Nov 02 18:53:21.056704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.056686 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" containerName="istio-proxy" Nov 02 18:53:21.056864 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.056722 5549 cpu_manager.go:414] "containerMap: removing container" podUID="31d32da2-d6a3-4fb5-b7e6-7e3773eabe66" containerName="payload-processing" Nov 02 18:53:21.056864 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.056733 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d32da2-d6a3-4fb5-b7e6-7e3773eabe66" containerName="payload-processing" Nov 02 18:53:21.056864 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.056762 5549 cpu_manager.go:414] "containerMap: removing container" podUID="7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" containerName="istio-proxy" Nov 02 18:53:21.056864 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.056769 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" containerName="istio-proxy" Nov 02 18:53:21.056864 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.056779 5549 cpu_manager.go:414] "containerMap: removing container" podUID="576b906e-d10a-4f68-8f2d-8034c076210c" containerName="maas-api" Nov 02 18:53:21.056864 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.056786 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="576b906e-d10a-4f68-8f2d-8034c076210c" containerName="maas-api" Nov 02 18:53:21.056864 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.056799 5549 cpu_manager.go:414] "containerMap: removing container" podUID="2f486ee5-9aad-4e65-8d32-d0e6997b3ec6" containerName="payload-pre-processing" Nov 02 18:53:21.056864 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.056805 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f486ee5-9aad-4e65-8d32-d0e6997b3ec6" containerName="payload-pre-processing" Nov 02 18:53:21.057243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.056886 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d32da2-d6a3-4fb5-b7e6-7e3773eabe66" containerName="payload-processing" Nov 02 18:53:21.057243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.056904 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" containerName="istio-proxy" Nov 02 18:53:21.057243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.056913 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="576b906e-d10a-4f68-8f2d-8034c076210c" containerName="maas-api" Nov 02 18:53:21.057243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.056924 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f486ee5-9aad-4e65-8d32-d0e6997b3ec6" containerName="payload-pre-processing" Nov 02 18:53:21.057243 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.056933 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bbbd9f3-ec69-414d-9f23-5c38134c8bfd" containerName="istio-proxy" Nov 02 18:53:21.060351 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.060330 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" Nov 02 18:53:21.071684 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.071649 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff"] Nov 02 18:53:21.132844 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.132802 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppjbs\" (UniqueName: \"kubernetes.io/projected/a7979c38-f2f9-4405-bb70-ac34a3c57098-kube-api-access-ppjbs\") pod \"maas-api-5fbc95fb78-qfvff\" (UID: \"a7979c38-f2f9-4405-bb70-ac34a3c57098\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" Nov 02 18:53:21.133058 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.132851 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7979c38-f2f9-4405-bb70-ac34a3c57098-maas-api-tls\") pod \"maas-api-5fbc95fb78-qfvff\" (UID: \"a7979c38-f2f9-4405-bb70-ac34a3c57098\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" Nov 02 18:53:21.133058 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.132947 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7979c38-f2f9-4405-bb70-ac34a3c57098-openshift-service-ca\") pod \"maas-api-5fbc95fb78-qfvff\" (UID: \"a7979c38-f2f9-4405-bb70-ac34a3c57098\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" Nov 02 18:53:21.133058 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.133003 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7979c38-f2f9-4405-bb70-ac34a3c57098-metrics-certs\") pod \"maas-api-5fbc95fb78-qfvff\" (UID: \"a7979c38-f2f9-4405-bb70-ac34a3c57098\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" Nov 02 18:53:21.234263 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.234222 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppjbs\" (UniqueName: \"kubernetes.io/projected/a7979c38-f2f9-4405-bb70-ac34a3c57098-kube-api-access-ppjbs\") pod \"maas-api-5fbc95fb78-qfvff\" (UID: \"a7979c38-f2f9-4405-bb70-ac34a3c57098\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" Nov 02 18:53:21.234263 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.234266 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7979c38-f2f9-4405-bb70-ac34a3c57098-maas-api-tls\") pod \"maas-api-5fbc95fb78-qfvff\" (UID: \"a7979c38-f2f9-4405-bb70-ac34a3c57098\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" Nov 02 18:53:21.234542 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.234299 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7979c38-f2f9-4405-bb70-ac34a3c57098-openshift-service-ca\") pod \"maas-api-5fbc95fb78-qfvff\" (UID: \"a7979c38-f2f9-4405-bb70-ac34a3c57098\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" Nov 02 18:53:21.234542 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.234316 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7979c38-f2f9-4405-bb70-ac34a3c57098-metrics-certs\") pod \"maas-api-5fbc95fb78-qfvff\" (UID: \"a7979c38-f2f9-4405-bb70-ac34a3c57098\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" Nov 02 18:53:21.235111 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.235086 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7979c38-f2f9-4405-bb70-ac34a3c57098-openshift-service-ca\") pod \"maas-api-5fbc95fb78-qfvff\" (UID: \"a7979c38-f2f9-4405-bb70-ac34a3c57098\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" Nov 02 18:53:21.236971 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.236930 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7979c38-f2f9-4405-bb70-ac34a3c57098-maas-api-tls\") pod \"maas-api-5fbc95fb78-qfvff\" (UID: \"a7979c38-f2f9-4405-bb70-ac34a3c57098\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" Nov 02 18:53:21.237091 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.236930 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7979c38-f2f9-4405-bb70-ac34a3c57098-metrics-certs\") pod \"maas-api-5fbc95fb78-qfvff\" (UID: \"a7979c38-f2f9-4405-bb70-ac34a3c57098\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" Nov 02 18:53:21.241882 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.241855 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppjbs\" (UniqueName: \"kubernetes.io/projected/a7979c38-f2f9-4405-bb70-ac34a3c57098-kube-api-access-ppjbs\") pod \"maas-api-5fbc95fb78-qfvff\" (UID: \"a7979c38-f2f9-4405-bb70-ac34a3c57098\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" Nov 02 18:53:21.376126 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.376037 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" Nov 02 18:53:21.652914 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:21.652883 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff"] Nov 02 18:53:21.655761 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:53:21.655732 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7979c38_f2f9_4405_bb70_ac34a3c57098.slice/crio-cbbdf7d39a67bccb17cb1d57d88d0e36e55e715ede02b789e2f8cb61a2ed7e60 WatchSource:0}: Error finding container cbbdf7d39a67bccb17cb1d57d88d0e36e55e715ede02b789e2f8cb61a2ed7e60: Status 404 returned error can't find the container with id cbbdf7d39a67bccb17cb1d57d88d0e36e55e715ede02b789e2f8cb61a2ed7e60 Nov 02 18:53:22.444031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:22.443920 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" event={"ID":"a7979c38-f2f9-4405-bb70-ac34a3c57098","Type":"ContainerStarted","Data":"7d00aa39bc769136be88ded014e3800f1862640b0c3416a4e4fb097cabaa1939"} Nov 02 18:53:22.444031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:22.443995 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" event={"ID":"a7979c38-f2f9-4405-bb70-ac34a3c57098","Type":"ContainerStarted","Data":"cbbdf7d39a67bccb17cb1d57d88d0e36e55e715ede02b789e2f8cb61a2ed7e60"} Nov 02 18:53:22.444031 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:22.444016 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" Nov 02 18:53:22.462063 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:22.462000 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" podStartSLOduration=0.978894092 podStartE2EDuration="1.461986946s" totalImagesPullingTime="483.092854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:53:22.460022891 +0000 UTC m=+4774.248528420" watchObservedRunningTime="2026-11-02 18:53:22.461986946 +0000 UTC m=+4774.250492473" Nov 02 18:53:28.453511 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:28.453480 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-qfvff" Nov 02 18:53:28.504859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:28.500284 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc"] Nov 02 18:53:28.504859 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:28.500614 5549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" podUID="1ae546ff-a649-4895-9aaf-53395a39e5a5" containerName="maas-api" containerID="cri-o://3239ec55f4865c697c101b822965bb97deaec0315473f3c73a9609fbf0673e1d" gracePeriod=30 Nov 02 18:53:29.748822 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:29.748789 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:53:29.762029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:29.762002 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" Nov 02 18:53:29.822319 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:29.822281 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ae546ff-a649-4895-9aaf-53395a39e5a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae546ff-a649-4895-9aaf-53395a39e5a5-metrics-certs\") pod \"1ae546ff-a649-4895-9aaf-53395a39e5a5\" (UID: \"1ae546ff-a649-4895-9aaf-53395a39e5a5\") " Nov 02 18:53:29.822527 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:29.822377 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae546ff-a649-4895-9aaf-53395a39e5a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae546ff-a649-4895-9aaf-53395a39e5a5-openshift-service-ca\") pod \"1ae546ff-a649-4895-9aaf-53395a39e5a5\" (UID: \"1ae546ff-a649-4895-9aaf-53395a39e5a5\") " Nov 02 18:53:29.822605 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:29.822533 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae546ff-a649-4895-9aaf-53395a39e5a5-kube-api-access-cmb4z\" (UniqueName: \"kubernetes.io/projected/1ae546ff-a649-4895-9aaf-53395a39e5a5-kube-api-access-cmb4z\") pod \"1ae546ff-a649-4895-9aaf-53395a39e5a5\" (UID: \"1ae546ff-a649-4895-9aaf-53395a39e5a5\") " Nov 02 18:53:29.822605 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:29.822569 5549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ae546ff-a649-4895-9aaf-53395a39e5a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae546ff-a649-4895-9aaf-53395a39e5a5-maas-api-tls\") pod \"1ae546ff-a649-4895-9aaf-53395a39e5a5\" (UID: \"1ae546ff-a649-4895-9aaf-53395a39e5a5\") " Nov 02 18:53:29.822760 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:29.822736 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae546ff-a649-4895-9aaf-53395a39e5a5-openshift-service-ca" pod "1ae546ff-a649-4895-9aaf-53395a39e5a5" (UID: "1ae546ff-a649-4895-9aaf-53395a39e5a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:53:29.822993 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:29.822945 5549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae546ff-a649-4895-9aaf-53395a39e5a5-openshift-service-ca\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:53:29.824624 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:29.824602 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae546ff-a649-4895-9aaf-53395a39e5a5-metrics-certs" pod "1ae546ff-a649-4895-9aaf-53395a39e5a5" (UID: "1ae546ff-a649-4895-9aaf-53395a39e5a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:53:29.824747 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:29.824727 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae546ff-a649-4895-9aaf-53395a39e5a5-maas-api-tls" pod "1ae546ff-a649-4895-9aaf-53395a39e5a5" (UID: "1ae546ff-a649-4895-9aaf-53395a39e5a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:53:29.825003 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:29.824981 5549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae546ff-a649-4895-9aaf-53395a39e5a5-kube-api-access-cmb4z" pod "1ae546ff-a649-4895-9aaf-53395a39e5a5" (UID: "1ae546ff-a649-4895-9aaf-53395a39e5a5"). InnerVolumeSpecName "kube-api-access-cmb4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:53:29.924627 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:29.924584 5549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae546ff-a649-4895-9aaf-53395a39e5a5-maas-api-tls\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:53:29.924627 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:29.924624 5549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae546ff-a649-4895-9aaf-53395a39e5a5-metrics-certs\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:53:29.924870 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:29.924641 5549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmb4z\" (UniqueName: \"kubernetes.io/projected/1ae546ff-a649-4895-9aaf-53395a39e5a5-kube-api-access-cmb4z\") on node \"ip-10-0-142-133.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:53:30.487316 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:30.487270 5549 generic.go:356] "Generic (PLEG): container finished" podID="1ae546ff-a649-4895-9aaf-53395a39e5a5" containerID="3239ec55f4865c697c101b822965bb97deaec0315473f3c73a9609fbf0673e1d" exitCode=0 Nov 02 18:53:30.487545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:30.487346 5549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" Nov 02 18:53:30.487712 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:30.487677 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" event={"ID":"1ae546ff-a649-4895-9aaf-53395a39e5a5","Type":"ContainerDied","Data":"3239ec55f4865c697c101b822965bb97deaec0315473f3c73a9609fbf0673e1d"} Nov 02 18:53:30.487868 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:30.487849 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc" event={"ID":"1ae546ff-a649-4895-9aaf-53395a39e5a5","Type":"ContainerDied","Data":"9805ff650041d9e8328156dcbacd6a9c54a3ac449e2fe43410fed149587ef30c"} Nov 02 18:53:30.488029 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:30.487943 5549 scope.go:122] "RemoveContainer" containerID="3239ec55f4865c697c101b822965bb97deaec0315473f3c73a9609fbf0673e1d" Nov 02 18:53:30.507571 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:30.507527 5549 scope.go:122] "RemoveContainer" containerID="3239ec55f4865c697c101b822965bb97deaec0315473f3c73a9609fbf0673e1d" Nov 02 18:53:30.508133 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:30.507849 5549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc"] Nov 02 18:53:30.508531 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:53:30.508453 5549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3239ec55f4865c697c101b822965bb97deaec0315473f3c73a9609fbf0673e1d\": container with ID starting with 3239ec55f4865c697c101b822965bb97deaec0315473f3c73a9609fbf0673e1d not found: ID does not exist" containerID="3239ec55f4865c697c101b822965bb97deaec0315473f3c73a9609fbf0673e1d" Nov 02 18:53:30.508531 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:30.508494 5549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3239ec55f4865c697c101b822965bb97deaec0315473f3c73a9609fbf0673e1d"} err="failed to get container status \"3239ec55f4865c697c101b822965bb97deaec0315473f3c73a9609fbf0673e1d\": rpc error: code = NotFound desc = could not find container \"3239ec55f4865c697c101b822965bb97deaec0315473f3c73a9609fbf0673e1d\": container with ID starting with 3239ec55f4865c697c101b822965bb97deaec0315473f3c73a9609fbf0673e1d not found: ID does not exist" Nov 02 18:53:30.510263 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:30.510233 5549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f9c6994b-xjlfc"] Nov 02 18:53:32.337451 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:32.337406 5549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae546ff-a649-4895-9aaf-53395a39e5a5" path="/var/lib/kubelet/pods/1ae546ff-a649-4895-9aaf-53395a39e5a5/volumes" Nov 02 18:53:36.266137 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:36.266089 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xsg9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:53:36.266827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:36.266184 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" podUID="4740514e-6d04-468b-930f-8be299ef0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:53:47.587448 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:53:47.587409 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:53:50.157337 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:50.157301 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:53:50.368659 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:50.368632 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:53:59.232318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:53:59.232277 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:54:02.271385 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:54:02.271348 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:54:04.372971 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:54:04.372775 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:54:09.171679 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:54:09.171641 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:54:09.299012 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:54:09.298978 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:54:17.586497 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:54:17.586455 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:54:19.131424 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:54:19.131379 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:54:23.936526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:54:23.936037 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-r2rbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:54:23.936526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:54:23.936114 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" podUID="6a7bea79-f2bb-489d-b861-28fa66d0ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:54:23.979169 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:54:23.979129 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-kgx7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:54:23.990119 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:54:23.990077 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" podUID="c208756b-40ff-4691-aad8-9008058e4291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:54:24.727771 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:54:24.727734 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:54:29.950680 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:54:29.950645 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:54:46.668783 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:54:46.668744 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:54:47.585562 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:54:47.585513 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:54:48.038526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:54:48.038485 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:54:48.173827 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:54:48.173789 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:54:48.593525 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:54:48.593484 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:54:56.779413 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:54:56.779377 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:55:02.132894 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:55:02.132810 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:55:04.499975 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:55:04.499008 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:55:07.349313 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:55:07.349271 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:55:17.586298 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:55:17.586254 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:55:23.561083 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:55:23.561044 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:55:33.736158 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:55:33.736120 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:55:38.111456 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:55:38.111400 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:55:47.736482 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:55:47.736428 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:55:48.018225 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:55:48.018124 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-fl8ch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:55:48.018526 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:55:48.018490 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" podUID="b9b1010f-8c24-44d2-8495-f11e545f2187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:55:48.664764 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:55:48.664732 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:55:51.262590 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:55:51.262070 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xsg9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:55:51.262590 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:55:51.262160 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xsg9n" podUID="4740514e-6d04-468b-930f-8be299ef0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:55:53.877358 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:55:53.877321 5549 scope.go:122] "RemoveContainer" containerID="949996e77c4a9b8cc55f58206bc092712072b3d80302fed0d6858a471646a28f" Nov 02 18:56:04.541231 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:56:04.541193 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:56:04.658472 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:56:04.658434 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:56:17.586421 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:56:17.586382 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:56:26.735411 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:56:26.735377 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:56:32.414034 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:56:32.413929 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:56:47.722724 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:56:47.722678 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:56:48.009907 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:56:48.009808 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-fl8ch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:56:48.010301 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:56:48.010189 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" podUID="b9b1010f-8c24-44d2-8495-f11e545f2187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:57:04.621460 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:57:04.621423 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:57:17.676116 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:57:17.676066 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:57:18.014192 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:57:18.014133 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-fl8ch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:57:18.014596 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:57:18.014544 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" podUID="b9b1010f-8c24-44d2-8495-f11e545f2187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:57:38.930977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:57:38.930145 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-r2rbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:57:38.930977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:57:38.930250 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-r2rbg" podUID="6a7bea79-f2bb-489d-b861-28fa66d0ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:57:38.972545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:57:38.972094 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-kgx7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:57:38.972545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:57:38.972176 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-kgx7c" podUID="c208756b-40ff-4691-aad8-9008058e4291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:57:47.676631 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:57:47.676584 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:57:48.025977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:57:48.015044 5549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-fl8ch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:57:48.025977 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:57:48.015110 5549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-fl8ch" podUID="b9b1010f-8c24-44d2-8495-f11e545f2187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:57:56.570227 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:57:56.570188 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-99bp7"] Nov 02 18:58:06.760704 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:06.760617 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:58:17.584786 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:58:17.584747 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:58:20.328982 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:20.328935 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-56bcd69c95-g7ngq_b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54/manager/0.log" Nov 02 18:58:21.228145 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:21.228110 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56bc685b8f-tzvbj_248b5aef-8f56-4496-9bc2-521bc36176ba/manager/0.log" Nov 02 18:58:21.463606 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:21.463569 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4q597_585a619b-ddf1-497a-9fa0-40d105881005/server/0.log" Nov 02 18:58:21.653995 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:21.653902 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2wms8_13885bf3-7cdc-40aa-8e58-01eced7a6249/manager/0.log" Nov 02 18:58:21.897857 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:21.897822 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-xswfx_aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4/proxy/0.log" Nov 02 18:58:23.233059 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:23.233024 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fbc95fb78-qfvff_a7979c38-f2f9-4405-bb70-ac34a3c57098/maas-api/0.log" Nov 02 18:58:25.088759 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:25.088726 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:58:29.479896 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:29.479864 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-f74t5_db3845da-3e81-491c-9e2d-09f220ea1b9d/kuadrant-console-plugin/0.log" Nov 02 18:58:29.917049 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:29.916936 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-99bp7_0de90549-6c86-4ce1-9d45-1f3b136fa6e7/limitador/0.log" Nov 02 18:58:31.625522 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:31.625493 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cf9f7b655-p25w5_bfa718f4-ef10-4e8d-b342-e4759b10735a/kube-auth-proxy/0.log" Nov 02 18:58:32.014966 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:32.014924 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-fl8ch_b9b1010f-8c24-44d2-8495-f11e545f2187/istio-proxy/0.log" Nov 02 18:58:32.400107 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:32.400021 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-kgx7c_c208756b-40ff-4691-aad8-9008058e4291/istio-proxy/0.log" Nov 02 18:58:32.591901 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:32.591867 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-r2rbg_6a7bea79-f2bb-489d-b861-28fa66d0ec4d/istio-proxy/0.log" Nov 02 18:58:33.379030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:33.378993 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-xsg9n_4740514e-6d04-468b-930f-8be299ef0f4e/istio-proxy/0.log" Nov 02 18:58:33.957832 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:33.957800 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7d54786449-dvg2t_1200f090-634e-446d-b876-b37375067c84/payload-pre-processing/0.log" Nov 02 18:58:34.133202 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:34.133167 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-fd758c546-br2db_395c02e5-ad21-4b1b-a7ec-9bbd7dbfe204/payload-processing/0.log" Nov 02 18:58:35.554035 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:35.554003 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-pd4sg_310f7705-ba7a-44d3-9c98-6842c0c64c24/main/0.log" Nov 02 18:58:35.754682 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:35.754639 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-r7qcl_e4c4743d-7414-4aa5-b190-fecec685ae2c/main/0.log" Nov 02 18:58:36.176625 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:36.176590 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-flgk9_c1bb0ca2-6b43-4068-be3b-f44f74fe31fa/main/0.log" Nov 02 18:58:36.743426 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:36.743396 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-xp98l_f171b02f-9c9a-489c-a7cd-75cfdff945e5/main/0.log" Nov 02 18:58:47.101686 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.101648 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d27q6/must-gather-scrxq"] Nov 02 18:58:47.102210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.102077 5549 cpu_manager.go:414] "containerMap: removing container" podUID="1ae546ff-a649-4895-9aaf-53395a39e5a5" containerName="maas-api" Nov 02 18:58:47.102210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.102089 5549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae546ff-a649-4895-9aaf-53395a39e5a5" containerName="maas-api" Nov 02 18:58:47.102210 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.102154 5549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae546ff-a649-4895-9aaf-53395a39e5a5" containerName="maas-api" Nov 02 18:58:47.105509 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.105487 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d27q6/must-gather-scrxq" Nov 02 18:58:47.113250 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.113220 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-d27q6\"/\"default-dockercfg-cxsxq\"" Nov 02 18:58:47.113641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.113614 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d27q6\"/\"openshift-service-ca.crt\"" Nov 02 18:58:47.113641 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.113614 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d27q6\"/\"kube-root-ca.crt\"" Nov 02 18:58:47.113817 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.113618 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d27q6/must-gather-scrxq"] Nov 02 18:58:47.159890 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.159852 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:58:47.199728 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.199689 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmxwf\" (UniqueName: \"kubernetes.io/projected/de0ba228-8902-4887-b4e1-698791b50005-kube-api-access-bmxwf\") pod \"must-gather-scrxq\" (UID: \"de0ba228-8902-4887-b4e1-698791b50005\") " pod="openshift-must-gather-d27q6/must-gather-scrxq" Nov 02 18:58:47.200085 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.200050 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/de0ba228-8902-4887-b4e1-698791b50005-must-gather-output\") pod \"must-gather-scrxq\" (UID: \"de0ba228-8902-4887-b4e1-698791b50005\") " pod="openshift-must-gather-d27q6/must-gather-scrxq" Nov 02 18:58:47.201384 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.201362 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nc6cx/must-gather-cts98"] Nov 02 18:58:47.204985 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.204944 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nc6cx/must-gather-cts98" Nov 02 18:58:47.212036 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.212015 5549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nc6cx\"/\"default-dockercfg-qrpjr\"" Nov 02 18:58:47.212330 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.212307 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nc6cx\"/\"kube-root-ca.crt\"" Nov 02 18:58:47.212422 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.212364 5549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nc6cx\"/\"openshift-service-ca.crt\"" Nov 02 18:58:47.213202 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.213179 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nc6cx/must-gather-cts98"] Nov 02 18:58:47.300935 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.300899 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn52v\" (UniqueName: \"kubernetes.io/projected/ec6d4de9-4d76-41a3-9de2-02d944a908b2-kube-api-access-jn52v\") pod \"must-gather-cts98\" (UID: \"ec6d4de9-4d76-41a3-9de2-02d944a908b2\") " pod="openshift-must-gather-nc6cx/must-gather-cts98" Nov 02 18:58:47.301141 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.300988 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmxwf\" (UniqueName: \"kubernetes.io/projected/de0ba228-8902-4887-b4e1-698791b50005-kube-api-access-bmxwf\") pod \"must-gather-scrxq\" (UID: \"de0ba228-8902-4887-b4e1-698791b50005\") " pod="openshift-must-gather-d27q6/must-gather-scrxq" Nov 02 18:58:47.301141 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.301050 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/de0ba228-8902-4887-b4e1-698791b50005-must-gather-output\") pod \"must-gather-scrxq\" (UID: \"de0ba228-8902-4887-b4e1-698791b50005\") " pod="openshift-must-gather-d27q6/must-gather-scrxq" Nov 02 18:58:47.301141 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.301118 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ec6d4de9-4d76-41a3-9de2-02d944a908b2-must-gather-output\") pod \"must-gather-cts98\" (UID: \"ec6d4de9-4d76-41a3-9de2-02d944a908b2\") " pod="openshift-must-gather-nc6cx/must-gather-cts98" Nov 02 18:58:47.301447 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.301428 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/de0ba228-8902-4887-b4e1-698791b50005-must-gather-output\") pod \"must-gather-scrxq\" (UID: \"de0ba228-8902-4887-b4e1-698791b50005\") " pod="openshift-must-gather-d27q6/must-gather-scrxq" Nov 02 18:58:47.305898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.305871 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:58:47.309432 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.309408 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmxwf\" (UniqueName: \"kubernetes.io/projected/de0ba228-8902-4887-b4e1-698791b50005-kube-api-access-bmxwf\") pod \"must-gather-scrxq\" (UID: \"de0ba228-8902-4887-b4e1-698791b50005\") " pod="openshift-must-gather-d27q6/must-gather-scrxq" Nov 02 18:58:47.402007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.401904 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ec6d4de9-4d76-41a3-9de2-02d944a908b2-must-gather-output\") pod \"must-gather-cts98\" (UID: \"ec6d4de9-4d76-41a3-9de2-02d944a908b2\") " pod="openshift-must-gather-nc6cx/must-gather-cts98" Nov 02 18:58:47.402007 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.401998 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn52v\" (UniqueName: \"kubernetes.io/projected/ec6d4de9-4d76-41a3-9de2-02d944a908b2-kube-api-access-jn52v\") pod \"must-gather-cts98\" (UID: \"ec6d4de9-4d76-41a3-9de2-02d944a908b2\") " pod="openshift-must-gather-nc6cx/must-gather-cts98" Nov 02 18:58:47.402318 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.402296 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ec6d4de9-4d76-41a3-9de2-02d944a908b2-must-gather-output\") pod \"must-gather-cts98\" (UID: \"ec6d4de9-4d76-41a3-9de2-02d944a908b2\") " pod="openshift-must-gather-nc6cx/must-gather-cts98" Nov 02 18:58:47.412917 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.412892 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn52v\" (UniqueName: \"kubernetes.io/projected/ec6d4de9-4d76-41a3-9de2-02d944a908b2-kube-api-access-jn52v\") pod \"must-gather-cts98\" (UID: \"ec6d4de9-4d76-41a3-9de2-02d944a908b2\") " pod="openshift-must-gather-nc6cx/must-gather-cts98" Nov 02 18:58:47.421816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.421794 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d27q6/must-gather-scrxq" Nov 02 18:58:47.520416 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.520380 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nc6cx/must-gather-cts98" Nov 02 18:58:47.593965 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:58:47.593911 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:58:47.692550 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.692317 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d27q6/must-gather-scrxq"] Nov 02 18:58:47.698128 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:58:47.698080 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde0ba228_8902_4887_b4e1_698791b50005.slice/crio-8a231958f31f533e214b633c6859b4cbac801ce1b9089b8a17cf1303707c6d18 WatchSource:0}: Error finding container 8a231958f31f533e214b633c6859b4cbac801ce1b9089b8a17cf1303707c6d18: Status 404 returned error can't find the container with id 8a231958f31f533e214b633c6859b4cbac801ce1b9089b8a17cf1303707c6d18 Nov 02 18:58:47.699563 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.699543 5549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:58:47.795320 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:47.795293 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nc6cx/must-gather-cts98"] Nov 02 18:58:47.797766 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:58:47.797729 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec6d4de9_4d76_41a3_9de2_02d944a908b2.slice/crio-9a50bdb5d0877834aa37a3c44b7116615d489d6dcc59e38fa05656729f7350bc WatchSource:0}: Error finding container 9a50bdb5d0877834aa37a3c44b7116615d489d6dcc59e38fa05656729f7350bc: Status 404 returned error can't find the container with id 9a50bdb5d0877834aa37a3c44b7116615d489d6dcc59e38fa05656729f7350bc Nov 02 18:58:48.584635 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:48.584594 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d27q6/must-gather-scrxq" event={"ID":"de0ba228-8902-4887-b4e1-698791b50005","Type":"ContainerStarted","Data":"8a231958f31f533e214b633c6859b4cbac801ce1b9089b8a17cf1303707c6d18"} Nov 02 18:58:48.586030 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:48.586004 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nc6cx/must-gather-cts98" event={"ID":"ec6d4de9-4d76-41a3-9de2-02d944a908b2","Type":"ContainerStarted","Data":"9a50bdb5d0877834aa37a3c44b7116615d489d6dcc59e38fa05656729f7350bc"} Nov 02 18:58:49.593921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:49.593879 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nc6cx/must-gather-cts98" event={"ID":"ec6d4de9-4d76-41a3-9de2-02d944a908b2","Type":"ContainerStarted","Data":"725c6457d8a2e805d29b58d8f46dd7b5aff605d11662cbccf4a3f5be67ce221e"} Nov 02 18:58:49.593921 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:49.593928 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nc6cx/must-gather-cts98" event={"ID":"ec6d4de9-4d76-41a3-9de2-02d944a908b2","Type":"ContainerStarted","Data":"28d14c5318b8026a8e236d07b3e84974310e4060e64b6db23cec4c65ff8ee260"} Nov 02 18:58:49.611778 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:49.611717 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-nc6cx/must-gather-cts98" podStartSLOduration=1.844063003 podStartE2EDuration="2.611703159s" totalImagesPullingTime="767.640156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:58:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:58:49.609730387 +0000 UTC m=+5101.398235915" watchObservedRunningTime="2026-11-02 18:58:49.611703159 +0000 UTC m=+5101.400208730" Nov 02 18:58:50.250170 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:50.250130 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:58:50.461308 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:50.461278 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-956tn_2b6949d0-0006-4488-89ad-296077fcf045/ovn-acl-logging/0.log" Nov 02 18:58:50.697390 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:50.697336 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-drvjl_62013dcf-af15-4041-bdc2-f5437c000082/global-pull-secret-syncer/0.log" Nov 02 18:58:50.960306 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:50.960228 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" Nov 02 18:58:51.172147 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:51.172074 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-133.us-east-2.compute.internal_ebe0e6ace491eef2c5b9302a33c08d4d/haproxy/0.log" Nov 02 18:58:58.626724 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:58.626630 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-f74t5_db3845da-3e81-491c-9e2d-09f220ea1b9d/kuadrant-console-plugin/0.log" Nov 02 18:58:58.735373 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:58:58.735267 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-99bp7_0de90549-6c86-4ce1-9d45-1f3b136fa6e7/limitador/0.log" Nov 02 18:59:00.342803 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:00.342769 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cad2499d-dbf4-443c-ae65-4da5b6bba6f1/alertmanager/0.log" Nov 02 18:59:00.366575 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:00.366537 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cad2499d-dbf4-443c-ae65-4da5b6bba6f1/config-reloader/0.log" Nov 02 18:59:00.390673 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:00.390639 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cad2499d-dbf4-443c-ae65-4da5b6bba6f1/kube-rbac-proxy-web/0.log" Nov 02 18:59:00.414058 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:00.413182 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cad2499d-dbf4-443c-ae65-4da5b6bba6f1/kube-rbac-proxy/0.log" Nov 02 18:59:00.448898 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:00.448727 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cad2499d-dbf4-443c-ae65-4da5b6bba6f1/kube-rbac-proxy-metric/0.log" Nov 02 18:59:00.473737 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:00.473627 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cad2499d-dbf4-443c-ae65-4da5b6bba6f1/prom-label-proxy/0.log" Nov 02 18:59:00.500454 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:00.500415 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cad2499d-dbf4-443c-ae65-4da5b6bba6f1/init-config-reloader/0.log" Nov 02 18:59:00.729139 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:00.729051 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-47799_21691b74-73cc-41de-9c08-0f5e27528dba/node-exporter/0.log" Nov 02 18:59:00.788581 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:00.787787 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-47799_21691b74-73cc-41de-9c08-0f5e27528dba/kube-rbac-proxy/0.log" Nov 02 18:59:00.810822 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:00.810786 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-47799_21691b74-73cc-41de-9c08-0f5e27528dba/init-textfile/0.log" Nov 02 18:59:01.285339 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:01.285304 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-5994g_c893dfcd-6f72-45bd-96ab-521e97584a40/prometheus-operator/0.log" Nov 02 18:59:01.319445 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:01.319412 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-5994g_c893dfcd-6f72-45bd-96ab-521e97584a40/kube-rbac-proxy/0.log" Nov 02 18:59:01.387695 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:01.387659 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fd64789dc-hcqt6_290c0bd1-b749-4ec9-88e1-384ac37be16f/telemeter-client/0.log" Nov 02 18:59:01.412165 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:01.412117 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fd64789dc-hcqt6_290c0bd1-b749-4ec9-88e1-384ac37be16f/reload/0.log" Nov 02 18:59:01.443008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:01.442975 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fd64789dc-hcqt6_290c0bd1-b749-4ec9-88e1-384ac37be16f/kube-rbac-proxy/0.log" Nov 02 18:59:01.879919 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:01.879880 5549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x"] Nov 02 18:59:01.885774 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:01.885741 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:01.892994 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:01.892592 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x"] Nov 02 18:59:01.996162 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:01.996108 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrhsh\" (UniqueName: \"kubernetes.io/projected/6bc347ac-02dc-4cb7-8122-64998b541f7a-kube-api-access-qrhsh\") pod \"perf-node-gather-daemonset-gbq4x\" (UID: \"6bc347ac-02dc-4cb7-8122-64998b541f7a\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:01.996367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:01.996223 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6bc347ac-02dc-4cb7-8122-64998b541f7a-lib-modules\") pod \"perf-node-gather-daemonset-gbq4x\" (UID: \"6bc347ac-02dc-4cb7-8122-64998b541f7a\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:01.996367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:01.996261 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6bc347ac-02dc-4cb7-8122-64998b541f7a-proc\") pod \"perf-node-gather-daemonset-gbq4x\" (UID: \"6bc347ac-02dc-4cb7-8122-64998b541f7a\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:01.996367 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:01.996351 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bc347ac-02dc-4cb7-8122-64998b541f7a-sys\") pod \"perf-node-gather-daemonset-gbq4x\" (UID: \"6bc347ac-02dc-4cb7-8122-64998b541f7a\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:01.996571 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:01.996375 5549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6bc347ac-02dc-4cb7-8122-64998b541f7a-podres\") pod \"perf-node-gather-daemonset-gbq4x\" (UID: \"6bc347ac-02dc-4cb7-8122-64998b541f7a\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:02.098658 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:02.098617 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6bc347ac-02dc-4cb7-8122-64998b541f7a-lib-modules\") pod \"perf-node-gather-daemonset-gbq4x\" (UID: \"6bc347ac-02dc-4cb7-8122-64998b541f7a\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:02.098974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:02.098925 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6bc347ac-02dc-4cb7-8122-64998b541f7a-lib-modules\") pod \"perf-node-gather-daemonset-gbq4x\" (UID: \"6bc347ac-02dc-4cb7-8122-64998b541f7a\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:02.099140 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:02.099124 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6bc347ac-02dc-4cb7-8122-64998b541f7a-proc\") pod \"perf-node-gather-daemonset-gbq4x\" (UID: \"6bc347ac-02dc-4cb7-8122-64998b541f7a\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:02.099365 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:02.099349 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bc347ac-02dc-4cb7-8122-64998b541f7a-sys\") pod \"perf-node-gather-daemonset-gbq4x\" (UID: \"6bc347ac-02dc-4cb7-8122-64998b541f7a\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:02.099498 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:02.099483 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6bc347ac-02dc-4cb7-8122-64998b541f7a-podres\") pod \"perf-node-gather-daemonset-gbq4x\" (UID: \"6bc347ac-02dc-4cb7-8122-64998b541f7a\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:02.099670 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:02.099648 5549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrhsh\" (UniqueName: \"kubernetes.io/projected/6bc347ac-02dc-4cb7-8122-64998b541f7a-kube-api-access-qrhsh\") pod \"perf-node-gather-daemonset-gbq4x\" (UID: \"6bc347ac-02dc-4cb7-8122-64998b541f7a\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:02.099993 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:02.099972 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bc347ac-02dc-4cb7-8122-64998b541f7a-sys\") pod \"perf-node-gather-daemonset-gbq4x\" (UID: \"6bc347ac-02dc-4cb7-8122-64998b541f7a\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:02.100205 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:02.100189 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6bc347ac-02dc-4cb7-8122-64998b541f7a-proc\") pod \"perf-node-gather-daemonset-gbq4x\" (UID: \"6bc347ac-02dc-4cb7-8122-64998b541f7a\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:02.100324 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:02.100248 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6bc347ac-02dc-4cb7-8122-64998b541f7a-podres\") pod \"perf-node-gather-daemonset-gbq4x\" (UID: \"6bc347ac-02dc-4cb7-8122-64998b541f7a\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:02.111847 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:02.111812 5549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrhsh\" (UniqueName: \"kubernetes.io/projected/6bc347ac-02dc-4cb7-8122-64998b541f7a-kube-api-access-qrhsh\") pod \"perf-node-gather-daemonset-gbq4x\" (UID: \"6bc347ac-02dc-4cb7-8122-64998b541f7a\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:02.205893 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:02.205852 5549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:02.515840 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:02.515747 5549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x"] Nov 02 18:59:02.520792 ip-10-0-142-133 kubenswrapper[5549]: W1102 18:59:02.520754 5549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6bc347ac_02dc_4cb7_8122_64998b541f7a.slice/crio-b5760e5869cfa7a332ae6195635251fdef9ca678c7ef866ed901f4b54a42dc05 WatchSource:0}: Error finding container b5760e5869cfa7a332ae6195635251fdef9ca678c7ef866ed901f4b54a42dc05: Status 404 returned error can't find the container with id b5760e5869cfa7a332ae6195635251fdef9ca678c7ef866ed901f4b54a42dc05 Nov 02 18:59:02.833051 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:02.832524 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" event={"ID":"6bc347ac-02dc-4cb7-8122-64998b541f7a","Type":"ContainerStarted","Data":"35819e34fd6ee12a40e785439660831b3cb5083664a384d8b8ae9c867de3d7ab"} Nov 02 18:59:02.833051 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:02.832584 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" event={"ID":"6bc347ac-02dc-4cb7-8122-64998b541f7a","Type":"ContainerStarted","Data":"b5760e5869cfa7a332ae6195635251fdef9ca678c7ef866ed901f4b54a42dc05"} Nov 02 18:59:02.833051 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:02.832848 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:02.834480 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:02.834447 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d27q6/must-gather-scrxq" event={"ID":"de0ba228-8902-4887-b4e1-698791b50005","Type":"ContainerStarted","Data":"83a635c7b6b57048d3805f55d1efc69d63441633a528895b0ddd03fb51a6cbfa"} Nov 02 18:59:02.858974 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:02.855782 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" podStartSLOduration=1.8557660390000001 podStartE2EDuration="1.855766039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 18:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:59:02.853609714 +0000 UTC m=+5114.642115248" watchObservedRunningTime="2026-11-02 18:59:02.855766039 +0000 UTC m=+5114.644271567" Nov 02 18:59:03.852973 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:03.847036 5549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d27q6/must-gather-scrxq" event={"ID":"de0ba228-8902-4887-b4e1-698791b50005","Type":"ContainerStarted","Data":"667a67d4063a78bb042cbecbbc15edc3674ccf48b867b4fd0bdab6158cdbd3f6"} Nov 02 18:59:04.521461 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:04.521422 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9456bdf8d-wxx6s_48cf59b2-8d8a-45f7-a81c-71bdf3ba1e47/console/0.log" Nov 02 18:59:04.527722 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:04.527693 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-56bcd69c95-g7ngq_b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54/manager/0.log" Nov 02 18:59:04.689436 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:04.689396 5549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8s6rw_e4a3f746-8b39-4e1e-8f18-8694b16929e1/konnectivity-agent/0.log" limit=1024 Nov 02 18:59:05.532362 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:05.532322 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56bc685b8f-tzvbj_248b5aef-8f56-4496-9bc2-521bc36176ba/manager/0.log" Nov 02 18:59:05.545149 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:05.545120 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4q597_585a619b-ddf1-497a-9fa0-40d105881005/server/0.log" Nov 02 18:59:07.027133 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:07.027098 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-48wvf_3df7744d-a04e-42aa-8938-31b27451b921/dns/0.log" Nov 02 18:59:07.052183 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:07.052144 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-48wvf_3df7744d-a04e-42aa-8938-31b27451b921/kube-rbac-proxy/0.log" Nov 02 18:59:07.196037 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:07.195996 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-278vv_e354a328-13db-422e-b0e7-0cc0a5af5a90/dns-node-resolver/0.log" Nov 02 18:59:07.311027 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:07.310680 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2wms8_13885bf3-7cdc-40aa-8e58-01eced7a6249/manager/0.log" Nov 02 18:59:07.337154 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:07.334548 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-xswfx_aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4/proxy/0.log" Nov 02 18:59:08.068685 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:08.068647 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dxmdj_ffdc716d-d67b-41da-aa12-c3ede5dd6b1f/node-ca/0.log" Nov 02 18:59:08.538812 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:08.538771 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-56bcd69c95-g7ngq_b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54/manager/0.log" Nov 02 18:59:08.859860 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:08.859785 5549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-gbq4x" Nov 02 18:59:08.878640 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:08.878577 5549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-d27q6/must-gather-scrxq" podStartSLOduration=7.006462178 podStartE2EDuration="21.878559316s" totalImagesPullingTime="14.872097138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:58:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:59:03.87317919 +0000 UTC m=+5115.661684696" watchObservedRunningTime="2026-11-02 18:59:08.878559316 +0000 UTC m=+5120.667064844" Nov 02 18:59:09.641564 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:09.641517 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56bc685b8f-tzvbj_248b5aef-8f56-4496-9bc2-521bc36176ba/manager/0.log" Nov 02 18:59:09.657586 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:09.657554 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4q597_585a619b-ddf1-497a-9fa0-40d105881005/server/0.log" Nov 02 18:59:10.079627 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:10.079559 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cf9f7b655-p25w5_bfa718f4-ef10-4e8d-b342-e4759b10735a/kube-auth-proxy/0.log" Nov 02 18:59:10.152372 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:10.152329 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-fl8ch_b9b1010f-8c24-44d2-8495-f11e545f2187/istio-proxy/0.log" Nov 02 18:59:10.225010 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:10.222206 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-kgx7c_c208756b-40ff-4691-aad8-9008058e4291/istio-proxy/0.log" Nov 02 18:59:10.261044 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:10.261006 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-r2rbg_6a7bea79-f2bb-489d-b861-28fa66d0ec4d/istio-proxy/0.log" Nov 02 18:59:10.399989 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:10.399888 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-xsg9n_4740514e-6d04-468b-930f-8be299ef0f4e/istio-proxy/0.log" Nov 02 18:59:10.497473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:10.497431 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7d54786449-dvg2t_1200f090-634e-446d-b876-b37375067c84/payload-pre-processing/0.log" Nov 02 18:59:10.527986 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:10.527393 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-fd758c546-br2db_395c02e5-ad21-4b1b-a7ec-9bbd7dbfe204/payload-processing/0.log" Nov 02 18:59:11.498598 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:11.498562 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2wms8_13885bf3-7cdc-40aa-8e58-01eced7a6249/manager/0.log" Nov 02 18:59:11.622129 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:11.622082 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-xswfx_aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4/proxy/0.log" Nov 02 18:59:12.965076 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:12.964065 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-56bcd69c95-g7ngq_b3b0f604-ac8f-4ea4-8fbd-28f4789c5c54/manager/0.log" Nov 02 18:59:14.061198 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:14.061142 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56bc685b8f-tzvbj_248b5aef-8f56-4496-9bc2-521bc36176ba/manager/0.log" Nov 02 18:59:14.094092 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:14.094054 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4q597_585a619b-ddf1-497a-9fa0-40d105881005/server/0.log" Nov 02 18:59:14.548628 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:14.547345 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8h628_67a04d53-e707-4b0b-9e32-d84c31e413e5/serve-healthcheck-canary/0.log" Nov 02 18:59:15.407840 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:15.407804 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cf9f7b655-p25w5_bfa718f4-ef10-4e8d-b342-e4759b10735a/kube-auth-proxy/0.log" Nov 02 18:59:15.418057 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:15.418023 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vr49c_babf1523-fa08-4397-b0ff-946af7e07278/kube-rbac-proxy/0.log" Nov 02 18:59:15.442816 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:15.442781 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vr49c_babf1523-fa08-4397-b0ff-946af7e07278/exporter/0.log" Nov 02 18:59:15.452473 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:15.452447 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-fl8ch_b9b1010f-8c24-44d2-8495-f11e545f2187/istio-proxy/0.log" Nov 02 18:59:15.468413 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:15.468385 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vr49c_babf1523-fa08-4397-b0ff-946af7e07278/extractor/0.log" Nov 02 18:59:15.495157 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:15.495126 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-kgx7c_c208756b-40ff-4691-aad8-9008058e4291/istio-proxy/0.log" Nov 02 18:59:15.514262 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:15.514234 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-r2rbg_6a7bea79-f2bb-489d-b861-28fa66d0ec4d/istio-proxy/0.log" Nov 02 18:59:15.597008 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:15.596928 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-xsg9n_4740514e-6d04-468b-930f-8be299ef0f4e/istio-proxy/0.log" Nov 02 18:59:15.651453 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:15.651385 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7d54786449-dvg2t_1200f090-634e-446d-b876-b37375067c84/payload-pre-processing/0.log" Nov 02 18:59:15.667380 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:15.667293 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-fd758c546-br2db_395c02e5-ad21-4b1b-a7ec-9bbd7dbfe204/payload-processing/0.log" Nov 02 18:59:15.874688 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:15.874656 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2wms8_13885bf3-7cdc-40aa-8e58-01eced7a6249/manager/0.log" Nov 02 18:59:15.978783 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:15.978744 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-xswfx_aa514a60-d4ba-4c81-8aca-c1f8f0ae6ef4/proxy/0.log" Nov 02 18:59:17.523359 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:17.523323 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cf9f7b655-p25w5_bfa718f4-ef10-4e8d-b342-e4759b10735a/kube-auth-proxy/0.log" Nov 02 18:59:17.564976 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:17.564588 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-fl8ch_b9b1010f-8c24-44d2-8495-f11e545f2187/istio-proxy/0.log" Nov 02 18:59:17.611703 ip-10-0-142-133 kubenswrapper[5549]: E1102 18:59:17.611665 5549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vr49c" containerName="extractor" Nov 02 18:59:17.622545 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:17.622510 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-kgx7c_c208756b-40ff-4691-aad8-9008058e4291/istio-proxy/0.log" Nov 02 18:59:17.646304 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:17.646265 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-r2rbg_6a7bea79-f2bb-489d-b861-28fa66d0ec4d/istio-proxy/0.log" Nov 02 18:59:17.727071 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:17.727006 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-xsg9n_4740514e-6d04-468b-930f-8be299ef0f4e/istio-proxy/0.log" Nov 02 18:59:17.779141 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:17.779052 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7d54786449-dvg2t_1200f090-634e-446d-b876-b37375067c84/payload-pre-processing/0.log" Nov 02 18:59:17.808223 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:17.807004 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-fd758c546-br2db_395c02e5-ad21-4b1b-a7ec-9bbd7dbfe204/payload-processing/0.log" Nov 02 18:59:18.262391 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:18.262360 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-789486f7c8-qmfz9_edf532ca-42d5-45d5-a384-3c3633ad50b2/manager/0.log" Nov 02 18:59:18.284503 ip-10-0-142-133 kubenswrapper[5549]: I1102 18:59:18.284475 5549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-789486f7c8-qmfz9_edf532ca-42d5-45d5-a384-3c3633ad50b2/copy-manifests/0.log"